For decades, a principle known as Occam's razor has guided financial analysts and economists. It suggests that the simplest explanation for a phenomenon is usually the best. In investing, this has translated into a fear of "overfitting"—creating models so complex they perfectly explain past data but fail to predict the future. However, a seismic shift is now underway, challenging this foundational belief with the power of artificial intelligence.
The Trial of a Timeless Principle
The debate ignited in 2021 with a pivotal paper titled "The Virtue of Complexity in Return Prediction." The authors—Bryan Kelly and Kangying Zhou of Yale University, and Semyon Malamud of the Swiss Federal Institute of Technology in Lausanne—presented a startling finding. They analyzed just 12 months of market data using a model with a staggering 12,000 separate parameters. Traditionally, such complexity would be dismissed as a sure path to overfitting. Yet, their results indicated the complex model actually improved forecasts of future returns. This led them to a provocative question: Could Occam's razor, in the context of modern machine learning, be Occam's blunder?
This is not merely an academic exercise. Bryan Kelly is also a portfolio manager at AQR, a major quantitative hedge fund. Known for its traditional, parsimonious methods, AQR is now actively exploring these complex approaches. Kelly argues that researchers have been so preoccupied with the risk of overfitting that they have neglected the opposite danger: underfitting data. In finance, where data on extreme events like market crashes is scarce, the ability to make better predictions from small samples could be enormously profitable.
The Backlash and the Billion-Dollar Stakes
Unsurprisingly, this challenge to orthodoxy has met fierce scepticism. The debate has reached a fever pitch in financial and academic circles. Critics like Álvaro Cartea, Qi Jin, and Yuantao Shi of the University of Oxford warn that the benefits of complexity may vanish if the underlying data is noisy or flawed. Meanwhile, Stefan Nagel of the University of Chicago suggests that for very small data sets, these complex models might simply be mimicking a basic momentum-trading strategy, and their apparent success could be a "lucky coincidence."
Kelly and Malamud have defended their work with further detailed research. While it's too early to write a eulogy for William of Ockham's 14th-century maxim, even sceptics concede that large, complex models can sometimes outperform simpler ones. The implications, however, extend far beyond theory. The global hunt for a trading edge already sees hedge funds spending billions on alternative data, from satellite imagery to social media sentiment analysis. If complexity truly reigns, this arms race will intensify dramatically.
A New Landscape for Investors
The potential victory of complex models would upend the investment industry. Hiring top machine-learning engineers would become a critical priority, potentially pushing investment firms to compete with Silicon Valley's tech giants for talent and their sky-high pay packets. Furthermore, data acquisition and cleaning would become even more crucial competitive advantages.
The shift would also favour scale. The immense computational power required to train and run these AI models is expensive, creating a significant "moat" protecting large hedge funds from smaller rivals. Larger firms could afford to experiment across a wider range of asset classes, while smaller competitors might struggle to keep pace, potentially reducing market competition.
This new era is not without risks. The most advanced machine-learning models often operate as "black boxes," making it difficult for humans to understand their decision-making process. While few will complain when profits are rolling in, a major failure—from chronic underperformance to a catastrophic blow-up of a strategy—could leave investors desperately wishing for a simpler tool to cut through the complexity they once embraced.
