Trusting Backtests Over Provisional Scores • danzells blog

Streaks aren’t meant to be broken. Greetings to Duolingo. So here’s another short update from my Numerai journey.

I’m still finding the rhythm and style for this blog. I don’t want it to be purely technical. I want it to capture snippets of my life and still explain something technical. Posting more is probably the only way to figure that out.

Right now, Numerai gives me fresh provisional scores every day. I know better than to trust them, yet a long streak of red numbers can still get into my head.

Why Numerai gets under my skin

Competitive ML is very much my thing. I started with Kaggle and eventually got hooked on Numerai. For anyone unfamiliar with it, here’s the short version:

Numerai is a hedge fund that incorporates machine learning models developed by a global community of data scientists. Participants receive anonymized tabular financial data and use it to train models that rank stocks relative to one another. They submit predictions on a regular basis, which are later evaluated against actual market outcomes. Numerai then combines selected submissions into a meta-model that supports its investment strategy. Participants may also stake NMR tokens on the performance of their models. Accurate, robust, and ideally uncorrelated predictions are rewarded, while underperforming models may lose part of their stake. In essence, Numerai is a decentralized machine learning competition whose predictions are used by a real hedge fund.

To me, the Numerai tournament is the pinnacle of ML tournaments. What could be more intense than putting my own money behind models I built myself? That requires a healthy amount of confidence in my skills. Haha.

What I still don’t understand is why Numerai doesn’t generate more buzz. We’re in the middle of both a blockchain and an AI bubble, and Numerai connects the two.

The provisional-score trap

Putting money behind my own models can be exhausting, especially after accumulating a few $$$ over the years. This is why:

Numerai publishes results every day, showing how the current round performed. The predictions behind those results were made roughly 20 days earlier. It also gives us a glimpse of the coming month through provisional scores, shown slightly transparent in the animation.

I’ve been doing this long enough to know that I shouldn’t trust those scores. Green can quickly turn red, and red can turn green. I still catch myself letting unresolved rounds affect me, both positively and negatively.

So when a seemingly endless wave of red, negative scores rolls in, remember your hopefully robust, leakage-free validation setup and the backtest. Cough.

Backtesting for stability

My latest experiments focus on reducing the large variance in those scores. The backtest covers roughly the entire season so far. For older rounds, Numerai’s meta-model information is unfortunately too sparse.

I’d rather have a model that performs steadily at a slightly lower level than one that looks good across a full year but repeatedly goes through huge swings.

I compared several model variants under the same time-based conditions. They mainly differ in how much historical information they use, how complex they are, and whether they try to adapt to changing market regimes. I wasn’t looking only for the highest average score. I wanted a robust balance between signal strength and stability.

Reading the figure

(a) Performance over time: The lines show the selected models’ scores in each test period. The larger variants with a longer historical view develop relatively smoothly. A smaller, more reactive model can keep up at times, but it fluctuates much more. A good average alone isn’t enough. I also care about how reliably a model moves through different market regimes.

(b) Mean and uncertainty: Each point summarizes one experiment, while the horizontal lines show the uncertainty of the estimate. The strongest approaches form a tight group with overlapping intervals, so there is no unambiguous winner. Several approaches reach a similar performance level but differ in stability and complexity.

(c) Cumulative performance: This panel shows how small weekly contributions build over the full test window. The steadier models gain their advantage step by step. The more reactive model stays close for a long time, then loses ground during a few weak periods. In this experiment, consistency is worth more than short performance spikes.

(d) Protection against future information: The diagram shows the time split between training and testing. An embargo prevents target windows from overlapping, while the live cutoff includes only data that would have been available at the actual training time. This makes the backtest reflect the real decision as closely as possible.

My reference for the next iteration

The experiments favor models that combine enough historical context with sufficient capacity. Smaller, shorter-term variants react faster but are more vulnerable to individual market regimes. Additional regime adaptation brought no clear benefit within this test window. It still needs to prove itself under meaningfully different conditions.

A narrow lead matters less to me than consistent performance over time. I want the approach that builds performance steadily and stays controlled during weaker periods. For my next iteration, the most stable long-term model becomes the reference. I’ll only adopt new mechanisms if they show repeatable value across further market regimes.

If you’re interested in the models mentioned above:

May the next candle be green.

danzell