Predicting bank failure with machine learning: what the rare-event problem does to your model

Bank failure is uncommon, and that single fact undoes most of what makes machine learning attractive elsewhere. Why headline accuracy flatters a useless model, what class imbalance really costs, and what a supervisor should require before relying on an output.

In short

Machine learning genuinely adds non-linearity, interaction effects and variable selection across a wider input set than a supervisor can hold in mind. Those gains are real and they are not the difficulty.
The difficulty is that bank failure is rare. A model that predicts survival for every institution will look highly accurate in a population where almost every bank survives, which makes headline accuracy the wrong measure entirely.
Rarity also means few examples to learn from, so models tend to fit the particular crises inside the estimation window rather than the general pattern of distress.
Every remedy for class imbalance has a cost, because resampling a supervisory population changes what the model is estimating and therefore what its output means.
The defensible position is that machine learning prioritises supervisory attention and surfaces candidates for closer examination. It does not replace supervisory judgement, and it does not by itself justify an intervention.

On this page

What the newer methods genuinely bring
The rare-event problem, stated plainly
Too few examples to learn from
What is done about imbalance, and what each remedy costs
What a supervisor should require before relying on an output
What to do next

Machine learning has an obvious appeal for supervisors: more inputs than anyone can weigh by hand, relationships that are not straight lines, and no need to specify the functional form in advance. All of that is real. The problem lies elsewhere.

What the newer methods genuinely bring#

Three gains are worth stating plainly, because dismissing them is as unhelpful as overselling them.

**Non-linearity.** Distress rarely arrives in proportion to a ratio; it accelerates. Methods that do not assume a straight line capture that acceleration without being told to expect it.
**Interaction.** A funding profile that is unremarkable alone becomes serious alongside a concentration. Classical specifications must be told which interactions to look for; these methods find them.
**Breadth.** Variable selection across hundreds of reported fields, including ones no supervisor would have nominated, occasionally surfaces a genuinely informative series that nobody was watching.

The rare-event problem, stated plainly#

Bank failure is uncommon. In most supervised populations, across most periods, the overwhelming majority of institutions do not fail. That single fact undoes much of what makes these methods attractive in domains where the event of interest is common.

Consider a model that predicts survival for every institution without exception. In a population where almost every bank survives, that model is correct almost every time. Reported as an accuracy figure it looks excellent. It has learned to say “no”, it will never identify a failing bank, and it is worthless.

A model that has learned to say “no” will be right almost every time, and will never once be useful.

This is why headline accuracy is the wrong measure here, and why a supervisor should treat any single summary figure offered for a distress model with suspicion until they know what share of the population experienced the event.

Too few examples to learn from#

Rarity has a second consequence, less discussed and more damaging. These methods learn from examples, and a rare event provides few. What the model largely learns is the specific crises that happened to fall inside the estimation window.

If those were driven by a property boom, the model becomes an excellent detector of property booms. Presented with distress arriving through a different channel it may see nothing at all, while continuing to report the same confident figures.

What is done about imbalance, and what each remedy costs#

Standard remedies exist. Each changes something, and a supervisor should know what.

**Resampling** — reweighting the population so failures are less outnumbered. The model then estimates a world with a different failure rate from the real one, so its outputs are no longer probabilities of failure in your population.
**Cost weighting** — telling the model that missing a failure is worse than a false alarm. This is honest and useful, and it requires the authority to state that relative cost explicitly, which is a governance decision rather than a technical one.
**Broadening the target** — predicting distress or supervisory downgrade rather than outright failure. This gives far more examples and changes the question being answered, which must then be reflected in how the output is used.

The third is often the most practical, and it is frequently done without saying so. A model marketed as predicting failure that was in fact trained on supervisory downgrades is answering a different and easier question, and the distinction belongs in the documentation.

What a supervisor should require before relying on an output#

Five requirements are reasonable, and none of them is technically demanding to ask for.

State what the model was trained to predict, and what share of the training population experienced it.
Report performance out of time — on a period after the one used for estimation — rather than on a random split of the same years.
Show stability across the cycle, including a period without a crisis in it, since a model tested only on turbulent years has not been tested on the years you will mostly use it in.
Report both error types separately at the operating threshold actually used, since the summary figure hides the trade-off the authority is making.
Disclose which inputs drive the output, and monitor whether that changes over time — a model whose drivers drift has quietly become a different model.

What to do next#

Ask one question of any distress model presented to you, whether built in-house or bought: what proportion of the institutions in the training data actually experienced the event being predicted?

The answer determines how to read every other figure in the document. If it cannot be answered quickly, that is itself the finding, and it matters more than any performance statistic on the page.

Frequently asked

Can machine learning predict bank failure?

It can improve the ranking of institutions by risk, and it handles non-linearity and interaction between variables better than classical approaches. But bank failure is a rare event, and that changes what these methods can honestly deliver. A model predicting survival for every institution will appear highly accurate in a population where almost every bank survives, so headline accuracy is a misleading measure. Rarity also means few examples to learn from, so models tend to fit the particular crises inside the estimation window rather than the general pattern. The defensible position is that machine learning prioritises supervisory attention and surfaces candidates for closer examination — it does not replace supervisory judgement and does not by itself justify intervention.

Why is accuracy a poor measure for a bank distress model?

Because the event is rare. Imagine a model that predicts survival for every institution without exception. In a population where almost every bank survives, that model is correct almost every time, and reported as an accuracy figure it looks excellent — yet it has simply learned to say “no”, will never identify a failing bank, and is worthless. Any single summary figure offered for a distress model should therefore be treated with suspicion until you know what share of the population experienced the event. The useful reporting shows both error types separately at the operating threshold actually in use, because that is where the authority’s real trade-off between missed failures and false alarms becomes visible.

What is class imbalance and how is it handled in supervisory models?

Class imbalance is the condition where the outcome being predicted is far less common than its opposite — in supervision, failures against a large majority of surviving institutions. Three remedies are standard and each carries a cost. Resampling reweights the population so failures are less outnumbered, but the model then estimates a world with a different failure rate from the real one, so its outputs are no longer probabilities of failure in your population. Cost weighting tells the model that missing a failure is worse than a false alarm, which is honest but requires the authority to state that relative cost explicitly — a governance decision, not a technical one. Broadening the target to distress or supervisory downgrade gives far more examples and changes the question being answered.

What should a supervisor require before relying on a distress model?

Five things, none of them technically demanding to ask for. First, a statement of what the model was trained to predict and what share of the training population experienced it — a model marketed as predicting failure but trained on supervisory downgrades is answering an easier question. Second, performance measured out of time, on a period after the one used for estimation, rather than on a random split of the same years. Third, stability across the cycle including a period without a crisis in it. Fourth, both error types reported separately at the operating threshold actually used. Fifth, disclosure of which inputs drive the output, monitored over time, since a model whose drivers drift has quietly become a different model.

The programme behind this article

Work through this material with the practitioners who wrote it.