For months our best number was a 99.3% detection rate across 143 validated flood positives, drawn from the INDOFLOODS labelled event record. It led every deck we had and it sat at the top of this site.
Then we went back and asked a duller question about the labels themselves. Our event labeller had been snapping each INDOFLOODS gauge to the nearest pilot city within 120 kilometres. That sounds reasonable until you look at a map of Odisha, where 120 kilometres crosses whole basins.
Where the events actually were
When each event was traced back to the river it actually happened on, the distribution was not what the headline implied:
- Baitarani, 79 events. Anandapur is 86.8 km from Cuttack.
- Rushikulya, 34 events. Purushottampur is 104.6 km away, in Ganjam.
- Brahmani, 29 events. Jenapur, 49.4 km.
- Mahanadi, 1 event. Naraj.
One of 143 validated positives was on the river we were claiming to watch.
It also explained a puzzle we had shelved. Bhubaneswar and Sambalpur had been showing zero scored events, and we had assumed a bug. There were never real ones there to score.
What we did with it
The number was mislabelled, not fabricated. We could have quietly corrected the labeller, rerun the backtest and moved on with a slightly different figure. Instead we retired the headline, published the audit, and put the replacement on the public site: 95.1% across all 143 events, and 94.4% on north Odisha onsets, rain-only, scored against each gauge’s own ERA5 p95 with negatives sampled at the same gauge.
The harder finding was underneath it. Our old “0% false positive rate” had been measured on dry days, which is the one case a rainfall rule separates by construction. On heavy monsoon days at or above a gauge’s own p95 with no recorded flood, the engine fires on every single one: 647 of 647. Sweeping thresholds from 0.5 to 8 times p95, precision never exceeds 24.2% against a 20% base rate.
North Odisha flood days have a median of 1.94 times p95. Non-flood heavy days have 2.03. The days without floods had more rain. Either rainfall genuinely cannot discriminate and we need river level as a feature, or the flood record is not exhaustive and we need river level as a label. Both answers point the same way, which is why real river telemetry is now in production.
Backtest. Every figure here appears on the research page with the same label, and in docs/ModelEarth_Backtest_Relabel_Audit.md.