Sometimes the absurdity of our analytical tools is so obvious that it cannot be spotted without stepping back and having a look at the whole picture. Stress tests as analytical tool are a case in point. On the surface, they seem to complement the standard risk management tools and introduce a degree of “real world” control over the outcome. In practice due to the three major problems in the design of stress tests they end up feeling like one of those primary school math exercises gone wrong: after a long set of calculations you conclude that x = x!
Problem 1: How big a stress?
If we knew beforehand how large the risks are, why would we need stress testing in the first place? It is always possible to come up with a magnitude of stress that can imply an unacceptable level of loss – our limit is the imagination. Since we cannot accurately estimate the likelihood of these scenarios, there is no way of telling which magnitude of stress is the appropriate one. There was an old game: two players think of a number each, the first one tells his number to the second one. The second one then reveals his number, and if it is larger, he wins. Guess who’s the winner in every turn?
Problem 2: What to stress?
Interestingly, this issue pops up in the process of sorting out the Lehman mess. This is a passage from the examiner report in the Chapter 11 proceedings of Lehman: “One of Lehman’s major risk controls was stress testing. …One stress test posited maximum potential losses of $9.4 billion, including $7.4 billion in losses on the previously excluded real estate and private equity positions, and only $2 billion on the previously included trading positions. …But these stress tests were conducted long after these assets had been acquired, and they were never shared with Lehman’s senior management”. Since the scenarios and magnitudes are arbitrary, it is hard to imagine that there were no occasional catastrophic outcomes of analysis. In retrospect, it is always possible to find the scenario which was catastrophic and was not shared with the management.
Often existing bubbles in the global economy can be spotted by the number of IPOs, size of the bonus pool and the opening of dedicated desks by the banks. In many instances, however, this is not the case. While we all know the well publicized imbalances in the economy (China-commodities-trade –Fed etc.), which asset class will be hurt first (or at all) is hard to know. Every crisis in the past hundred years has had unique triggers, mechanisms and consequences. They all had one thing in common – people lost money.
Problem 3: What is the impact?
Last but not least, the most severe problem of all: what is the impact of a stress? A combination of the network effect in action and a complex set of non-linear relationships and tipping points render the whole exercise superficial. Correlations and linear relationships at best describe a world of marginal changes. A stress test, on the other hand, assumes an unusually large change, which by definition triggers unusual reactions.