Skip to content
Mehridian

One record, end to end

Edge Evidence · Overview · 1 One record · 2 Copy and verify · 3 Lab runs · 4 The model · 5 Checks and gaps

A processing result is more than an output file. You also need to know which input it came from, whether the work finished, and what happens if the process stops halfway. The public Processor makes that concrete with a deterministic metadata-report operation over immutable raw media. It is not a general AI processing engine. Its database and spool carry the authority; the summary a command prints is a view of that authority, not a replacement for it.

Two stores, two jobs

  • SQLite records the artifact metadata, the processing job’s state, the derived artifact and its one-parent derived_from lineage.
  • The spool, a POSIX directory, holds the raw and derived bytes and manifests, the staging files and the publication lock.

Raw evidence is never replaced, but the spool still has to be writable, because processing adds derived files and uses staging and lock files. “Immutable input” doesn’t mean a read-only filesystem.

Before processing, the code checks the database, the raw artifact’s identity and digest, the file, the manifest and the media. If the input is ambiguous, it refuses instead of picking an occurrence for itself.

A checkpoint is not a result

The job’s identity is derived from the input digest plus the processor, its version, its output contract and its parameters. The same input always gives the same job. The sequence is:

validate input + acquire the spool lock
→ derive the same job identity
→ commit PREPARED
→ stage and publish report, then manifest
→ verify published bytes
→ one database transaction registers artifact + lineage + COMPLETE

There is a deliberate gap between committing PREPARED and the final transaction. The database and the filesystem are not one atomic transaction, so the recovery logic has to handle that gap explicitly instead of hoping it never happens.

Publishing is careful in a specific order:

  • The report goes out before its manifest, using no-replace links, and files and directories are flushed to disk.
  • A report without its manifest is a permitted interrupted state. The next run completes it.
  • A manifest without its report is treated as corruption.
  • Different bytes at a content-addressed path fail closed. They are never overwritten or silently repaired.

Four runs, one job

The checked-in demonstration runs the same synthetic input four ways. Each run is summarised by counters the worker prints from its authority:

Four Processor runs and their counters. First run: one job newly completed. Replay: one already complete, nothing written. Resume after an interruption at PREPARED: one prepared job resumed, under the same job ID. Busy lock: one deferred and one remaining, nothing written. All four exit 0.

  • First run. One eligible input, one job newly completed.
  • Replay. The exact job already exists, so the worker verifies it and reuses it. Nothing is rewritten, and the authoritative state is unchanged.
  • Resume. The demonstration stops a child process at the after_prepared_commit barrier, right after PREPARED is committed. At that moment there is exactly one PREPARED job, zero derived artifacts and zero lineage rows: the half-finished handoff isn’t promoted to a success. The retry resumes that job and finishes it under the same job ID, c9eb642c-10dd-558f-8897-90ac750fc126, and a replay afterwards finds it already complete.
  • Busy lock. Another holder has the spool. The worker defers, creates no processing job and changes nothing authoritative. The work stays counted as remaining.

All four exit 0, because deferring isn’t an error. What distinguishes them is the counters, and whether anything authoritative changed.

These are the demonstration’s frozen expected values, from its fixture, not a newly collected run. They show bounded reliability on one machine. They don’t prove exactly-once execution across arbitrary storage providers or concurrent distributed writers.

What it costs

The design chooses a single-writer SQLite and POSIX authority on purpose, and that has costs:

  • Lock contention reduces throughput.
  • Validation and readback add I/O.
  • Recovery is more explicit than “write a file and mark it done”.

Lock contention isn’t hypothetical. In the first lab run with two real sources, capture held shared access to the spool while processing needed exclusive access, and processing exited with its lock-contention code (page 3).

What a reviewer should look at

Look at the input and derived manifests, the digest bindings, the job state and lineage in SQLite, and then the retained bytes, not only a friendly receipt. Ask two questions: did replay reuse the same identity, and did a refusal leave the authoritative state unchanged?

The hosted demo doesn’t run this path. Its Proof view replays one retained result in which a Processor report was handed to Replication; the next page follows it.

Basis: the public Processor’s architecture notes, pipeline code and synthetic demonstration (examples/synthetic_processor.py, fixtures/cloud_run/expected.json), as traced in the Processor recovery chapter. No new runs were made for this page.

Next: Copy, adopt, verify