Letting operation decide

Edge Evidence · Overview · 1 One record · 2 Copy and verify · 3 Lab runs · 4 The model · 5 Checks and gaps

Edge Evidence didn’t start as a set of services. The first decision was deliberately modest: a modular monolith with explicit seams. It had one deterministic source, one edge-agent process, SQLite, a local spool and administrative checks. More workers, network storage, a message broker and a cloud runtime belonged to later gates, to be justified by a failure, a state boundary or a scaling need when one actually appeared.

What follows is how those gates were passed. Each run was set up to answer one question, and several of them went wrong in instructive ways.

First, survive an interruption

The first specifications came in narrow slices:

the repository and contracts, with no captured media yet
a deterministic FFmpeg source
finalising media
registering content and capture occurrences
reconciliation after a crash and restart

One early synthetic acceptance report lists 10 of 10 scenarios passed. They include normal completion, exact retry, reconciliation, gap recovery, SIGKILL at chosen points, lock contention and stopping on corruption. A later record added a controlled loopback camera stream. Both are synthetic, and both say so.

The build log around them is more revealing than the green result. One step’s implementation and scenarios were judged technically acceptable, but the evidence package for it was rejected, because a host-generated .DS_Store file had got into its inventory. The fix was a guard on the evidence layout. The next day the cleaned evidence was accepted and the next scope was opened. Nothing was wrong with the recovery mechanism; the package describing it was wrong. A passing test suite and a publishable record were treated as two different things.

OPBD-1: two real sources, one real boundary

The first real-input baseline used two physical sources: an iPad feed, and a Surface laptop publishing through MediaMTX. The accepted report records:

35 paired intervals, 70 intended capture opportunities, all ending COMPLETE, with none missing and none left PREPARED
70 events committed to the local outbox and deliberately not sent, with zero publication attempts
214 of 214 eligible objects VERIFIED on a QNAP network-storage target, with full readback at named checkpoints

The interesting event was a lock conflict. Capture held shared access to the spool while processing needed exclusive access. Processing exited with its documented lock-contention code while the captures carried on. Run again after capture, it produced a safe result, and later exact reruns changed nothing. Nothing was corrupted. The conflict was a repeatable sign that capture and processing wanted different lifecycles.

That is one accepted lab baseline, not a supported rate. The run predates the timestamped ledger format, and the record explicitly forbids inventing one after the fact. It doesn’t split the shared Wi-Fi traffic by source, certify camera profiles, or treat the replica as the original authority.

OPBD-2: a long run that ended as a limitation

OPBD-2 was a controlled 7,200-second run with two sources, an append-only diagnostic ledger and a terminal receipt. It went wrong in two different ways, and each got a different response.

Problem one: the harness was wrong. The first attempt stopped before its 90-minute checkpoint. The reconcile command hit the harness’s 240-second control timeout, and the timeout’s cleanup code then raised NameError, because signal had never been imported. An undrained output pipe was a separate defect, though the record doesn’t show it caused the stop. The attempt was classified as a run-level breach. Its partial database, spool, ledger and replica evidence still passed their integrity checks, and it was neither resumed nor relabelled.

A benchmark on a scratch copy then measured canonical reconciliation at about 270 seconds:

Bar chart: the harness allowed 240 seconds, reconciliation was measured at about 270 seconds, and the corrected budget is 600 seconds

The correction:

raised the budget to 600 seconds
imported signal
replaced the monitored pipes with temporary files
added regression tests for the cleanup path

The corrected run started on fresh state, not on top of the breached one.

Problem two: the source was lossy. The corrected run finished as OPBD2_2H_LIMITATION, not OPBD2_2H_PASS:

all 222 intended occurrences reached a terminal state, with none missing and none left PREPARED
445 of 445 replicas were verified by reading them back from the target
there were zero integrity-breach findings
there were 75 iPad source_gap_observed records

The accounting stayed truthful about the loss; it didn’t make the stream lossless. The harness got a correction. The gaps stayed in the result, because changing the accounting rules to remove them would have been the real failure. The limitation also didn’t authorise the planned eight-hour tier. A two-hour run that ended as a limitation tells this story in full.

OPBD-3: a missing worker

OPBD-3 tested independence. In one drill, Replication was left absent while capture continued, and then had to catch up and verify. The first wave didn’t close cleanly, because its evidence for the replication lag wasn’t satisfactorily sanitised. The rerun was scoped to that drill alone. Its closeout records:

lag past 60 seconds, a breach
then one catch-up invocation
12 of 12 VERIFIED, with nothing pending or missing
a passing final lag check and full readback

The raw diagnostic command had included the private target’s identity, so the retained evidence keeps only sanitised status, values and counts. The improvement here was in what was observed and redacted, not a new repair algorithm. A missing worker didn’t erase local capture, and catch-up needed no new queue. That doesn’t prove unattended replication through arbitrary outages.

OPBD-4: the decision

OPBD-4 used the three runs to answer the extraction question:

Part	Decision	Why
Processing	Extract as its own bounded worker	Repeated lock contention with capture; its jobs stayed durable and caught up later: a separate failure and resource lifecycle
Replication	Keep as a separate worker	It already was one, and OPBD-3 showed it could be absent and catch up
Source capture	Keep as bounded invocations	No observed need for more
Event distribution	Leave optional	The outbox held its events without a broker
Source supervisor, operator API	Defer	Not justified by anything observed

The decision explicitly doesn’t claim that horizontal scale, a cloud worker fleet or a broker had been measured and found necessary. Split out a worker after operation shows why follows the reasoning.

Then packaging taught its own lessons

Exporting the Processor and Replication as standalone public packages, plus a public Showcase that hands a Processor report to Replication, failed first in CI. The logic worked; the packaging didn’t:

FFmpeg and optional test dependencies were missing
a bind-mounted output directory broke atomic replacement
the shell and path context were wrong
--network none blocked even local test traffic
a virtual environment resolved to the wrong interpreter
an old URI form was used where the worker required a digest-addressed one

Each got a narrow fix, and the record forbade changing the workers to get past CI. One “nondeterministic” failure turned out to be GNU Make echoing different commands into the captured output while the fingerprints themselves matched. The fix was make --silent.

Basis: the project’s retained exercise records, as summarised in Development and operations. The raw run ledgers aren’t public, so every count on this page is the retained summary, not a re-derived figure. These are named lab exercises, not current service telemetry. No new runs were made for this page.

Previous: Copy, adopt, verify · Next: A model that explains, never decides