The leak is structural, not a typo.
Evaluation ranks each user's impression list as a set. A time-ordered history feature lets a validation row peek at the labels of its own list-mates, which is exactly what the metric asks the model to predict. Test rows can't do that, because their labels stop at the horizon. So the feature looks like genius on validation and is dead weight on test.
The fix: freeze every aggregate at the start of its evaluation window, not at each row's own timestamp. That collapses the validation→test gap from +0.1409 to +0.0070.
rank correlation. It's negatively correlated, so optimising it harder makes your submission worse.
rank correlation. So KAIROS selects on transfer across three temporal backtest folds instead.
Two gates between an idea and "accepted".
Plan a hypothesis, its mechanism, and one falsifiable prediction. Then write the code. A critic checks that the logic holds.
Allowlisted sandbox and blocked outcome columns. A leakage probe requires exactly zero within-user variance.
Isolated subprocess, bounded in time and memory, with an RSS watchdog.
Vectorised GAUC and nDCG@5. Matches the organisers' evaluator to 4.4e-16 and runs 13× faster.
It beat the incumbent on validation? Fine. Does it hold up on the backtest folds?
Log the hypothesis, diff, metrics, and prediction, then check the stop rule. The hidden test is scored once, at the very end.
The audit's strongest check is a proof, not a heuristic. Within-user ranking can't see anything that's constant across a user's list, so a user-level statistic has to have exactly zero within-user variance. Under the naive construction it was 1.24e-01. With the frozen-window fix it's 0.000e+00.
| Valid | Hidden test | vs baseline | |
|---|---|---|---|
| Official FM baseline (reproduced exactly) | 0.6016 | 0.5946 | — |
| Greedy validation-following agent | 0.7339 | 0.5790 | −0.0156 |
| Hand-built ensemble (reference ceiling) | 0.6045 | 0.5976 | +0.0030 |
| KAIROS (the submission) | 0.6030 | 0.5983 | +0.0037 |
0 manual interventions, 0 GPU-hours, 164,789 tokens. Both accepted candidates were independently backtest-confirmed. Bonus: on KuaiRand-1k, the same agent with one environment variable changed went from 0.5856 to 0.6536.