How it was checked
Seemorgh operator console · Overview · 1 The crossing · 2 Sensors · 3 Fusion · 4 AI layer · 5 Checks · 6 Beyond the scenario
A picture that looks right on a map can still hold the wrong reports in the wrong tracks. What makes this engine checkable is that the scenario knows the answer and the engine doesn’t. The tests compare what the engine produced against what really happened, using information the engine never sees.
Part of the Seemorgh operator console project.
The answer key in the report IDs
Every report the scenario generates carries an ID that encodes which object produced it:
| Feed | ID pattern | What it reveals |
|---|---|---|
| Plot radar | P<t>-<object>-<n>, or P<t>-clutter-<n> | The object, or that the plot is clutter |
| Long-range radar | 10 + object number | The object |
| Precision tracker | PS- and 100 + object number | The object |
| C2 network | c2-<object name>-<t> | The object by name |
| RF sensor | RF-<t>-<object> | The object |
The engine never reads these. It associates by geometry alone, which a phase 2 test checks by giving two distant tracks the same ID. The tests and the timeline audit decode the IDs to check every track. The system under test and the check use different information; that separation is the method.
What failure would have looked like
| Check | What failure would have looked like | Recorded result |
|---|---|---|
| The crossing, t=67 to t=81 | A track holding reports from both aircraft, or one aircraft split into two tracks or misclassified | None in the crossing or anywhere else in the 91 frames; at t=73 to t=75 every report, plots and RF included, traces to one object |
| Stray plots | A stray plot joining a track, or making one of its own | None in any frame; at t=74 all three are left out |
| Timeline audit | An anomaly in any of the 91 frames | None; the engine’s 33 tests pass |
| Repeat export | The same timestep producing a different picture | Two exports from separate processes are byte-identical |
| AI layer | Model text on screen, or a choice shown without being checked | 14 of 15 choices passed validation; the other fell back to a labelled deterministic answer |
These are checks against the one synthetic scenario, not field data.
The 33 fusion tests
Every test builds sensor reports in their native formats, so each also exercises normalisation.
| Phase | Tests | What they assert |
|---|---|---|
| 1. Normalisation | 5 | Knots and feet are converted; IFF calls map to classifications; an RF bearing and range resolve to the right place; polar and local-metre feeds agree |
| 2. Association | 3 | One object seen by two sensors becomes one track; distant objects stay separate; IDs are ignored |
| 3. Fusion | 4 | Position is precision-weighted; velocity comes from a track, not a plot; hostile is never downgraded; an unclassified group is unknown |
| 4. Clutter and priority | 5 | A lone plot is dropped; two nearby plots corroborate; a lone track is kept; a hostile near the asset outranks a distant friend; more sources raise confidence |
| 5. Interface | 3 | All five feeds are exposed; the fused GeoJSON is valid with dense priorities; no plot-only track has fewer than two plots |
| 6. Full timeline | 7 | Across 91 frames: identity-bearing reports are object-pure; three separate, correctly classified tracks; no clutter in or as a track; report timestamps. At the crossing: one hostile and one friend; every report pure at t=73–75 |
| 7. Vision bridge | 6 | A synthetic camera report isn’t copied from the fused answer, joins without a threat label, tightens σ and error, adds one source, keeps altitude and timestamps |
Phases 1 to 5 use small hand-built inputs. Phases 6 and 7 run the real scenario. A separate audit script sweeps all 91 timesteps with the phase 6 checks plus one more, that the hostile stays priority 1. It passes only with the line “91 timesteps, 0 anomalies”.
Recorded runs
- 24 October 2026, console acceptance. The Python and JavaScript suites, desktop playback from t=0 to t=90, and a 390×844 viewport. 152 Python and 5 JavaScript tests passed, after six defects found in the pass were repaired.
- Before the 27 October 2026 recording. All 91 frames were exported twice, from two separate processes. The exports were byte-identical.
- 27 October 2026, the full public test gate. It passed, including the container build, the 33 fusion tests, the 91-step audit and the console’s regression tests.
- 30 October 2026, a read-only rerun of t=74 with the association steps instrumented. It reproduced the recorded tracks exactly.
This page reports these results; it doesn’t rerun them.
Does the uncertainty mean what it says?
Every fused track carries a modelled σ, and a σ is only useful if it is honest. No test checks this across the timeline, so it was checked for this page, from the recorded frames and the scenario’s truth paths. For a circular Gaussian error with standard deviation σ in each axis, the truth should lie within 1σ of the estimate 39% of the time, within √2σ (about 1.41σ) 63% of the time, and within 2σ 86% of the time.
| Track | Mean error to truth | Within 1σ | Within √2σ | Within 2σ |
|---|---|---|---|---|
| Hostile drone (FT-001) | 30.9 m | 40% (36 of 91) | 65% | 88% |
| Friendly interceptor (FT-003) | 4.0 m | 37% (34 of 91) | 64% | 91% |
| Unknown contact (FT-002) | 68.8 m | 57% (52 of 91) | 80% | 95% |
The method reproduces the one published error figure: it gives the hostile track’s error at t=0 as 36.4 m, the value in the bridge experiment’s record.
The hostile and friendly tracks are close to calibrated, and the unknown contact is conservative. That is a good result, with a large caveat: it shows the engine’s error models agree with the scenario generator’s noise, not with any real sensor. The unknown contact’s σ is too large because the engine models the long-range radar’s error from its beamwidth, at a few hundred metres, while the generator adds 120 m. On real data, every one of those models would need measuring again.
The replay’s own checks
The public replay is 204 data files, each listed with its SHA-256. On every push, its CI checks:
- that every file matches its hash
- that the hash list names every file and nothing else
- that all 91 frames and briefs are present
- that both pages run in replay mode
- that every recorded answer is labelled, indexed and dated 27 October 2026
- that no published file contains a local path
Anyone can confirm that the frames shown are the frames exported, without trusting the page.
What isn’t checked
- Plot purity outside t=73 to t=75. In other frames the tests check only the identity-bearing reports. Decoding the recorded frames for this page found all 1,089 contributing reports in all 273 recorded tracks, plots and RF fixes included, tracing to one object each. That holds for this recording, but no test requires it.
- Wrong labels. Every C2 and RF identity call is correct in this scenario. No test feeds a mislabelled report.
- Bunched clutter. No test places two clutter plots close enough to corroborate each other.
- Altitude. Only the precision tracker adds altitude noise, so altitude agreement says nothing.
- Other scenarios. Everything above is one deterministic scenario, and the thresholds and error models were chosen for it.
- Devices and deployment. The console was checked in desktop browsers and phone-sized viewports, not on physical devices, and never against real sensors, live feeds or operators.
Basis: the phase 1 to 7 tests, the timeline audit and the scenario’s ID scheme in the FUSE package at the export commit named in the replay README (private source); the console acceptance record of 24 October 2026; and the replay’s hash list, CI workflow and all 91 recorded frames, in the public replay data. The calibration figures were computed on 30 October 2026 from those frames and the scenario’s stated truth paths; nothing was rerun for them.
Further reading: Yaakov Bar-Shalom, X. Rong Li and Thiagalingam Kirubarajan, Estimation with Applications to Tracking and Navigation (Wiley, 2001), on testing whether an estimator’s stated covariance is consistent with its errors.
Previous: An AI layer with almost no room · Next: Beyond the scenario · Back to the overview