Checksums, WORM Audit Trails, and the Tamper-Evidence Discipline
The prior two Trust lessons closed two rings. Runtime confinement last week shut down what a compromised process can do to the host. Zero-trust networking three weeks ago shut down what a compromised workload can reach on the wire. Both disciplines operate on the process and the connection. Neither touches the data.
Data is the third ring, and in a multi-agent system it is where the attack surface is easiest to miss. When one agent writes a payload and another reads it, the reader assumes the payload arrived intact and unaltered. That assumption goes unstated in most architectures. When it breaks — through a storage error, a bug that corrupts a record in transit, or an attacker who can write to a shared store — the downstream agent acts on a premise that never held.
Two distinct failure modes hide under the word "integrity." Corruption happens without intent: a cosmic ray flips a bit, a disk sector returns silent garbage, a packet arrives mangled. Tampering happens with intent: an actor who can reach the data replaces it with a value of their choosing, and the replacement passes every subsequent check because there is no keyed proof of origin to verify. A plain SHA-256 hash catches corruption. It does not catch tampering by someone who can also replace the hash. An HMAC catches both, because the tag depends on a key only the legitimate writer holds.
Three named techniques, together, close the data ring:
SHA-256 or SHA-3-256 over the canonicalized payload bytes. Detects corruption at rest and in transit. Fails if the payload changes by one bit.
HMAC-SHA256 keyed with a secret held only by the legitimate writer. Detects tampering by any actor without the signing key. Cannot be forged without the key.
WORM-committed audit store (S3 Object Lock Compliance mode, CloudTrail Lake). The record of what was written is irrevocable. The reconstruction holds even if the application layer is compromised.
A cryptographic hash function takes an input of any length and returns a fixed-length digest. Two properties matter. The same input always produces the same digest. Any change to the input, even one bit, produces a digest that bears no predictable relationship to the original. SHA-256 returns a 256-bit digest. SHA-3 (Keccak) returns the same size under a different internal construction that shares no design lineage with SHA-2, so a vulnerability in one does not carry to the other.
The hash alone does not distinguish corruption from tampering. If an attacker controls the store where both the payload and its hash live, the attacker replaces the payload, computes a new hash, and replaces the hash. The reader verifies the new hash against the new payload and concludes everything is fine. The hash function is correct. The check was futile.
HMAC — Hash-based Message Authentication Code — closes that gap. The tag is computed as a hash over the payload concatenated with a secret key, in a construction specified by RFC 2104 that prevents length-extension attacks against naive hash(key || message) constructions. An attacker who can modify the payload cannot forge a valid tag without the key. The legitimate writer holds the key. No one else does. The tag is proof of origin.
There is one more failure the tag does not prevent: replay. An attacker who cannot modify a payload can still capture a valid tagged payload and re-deliver it later, in a different context. A nonce, a timestamp, or a sequence number in the signed payload closes the replay window. The reader rejects any payload whose sequence number it has already processed.
WORM storage — Write Once, Read Many — takes a different axis. A WORM-committed record cannot be deleted or overwritten before a configurable retention period expires, regardless of the credentials of the principal asking for deletion. S3 Object Lock in Compliance mode is the canonical cloud implementation: not even the AWS account root can delete a locked object before its retention date. The audit trail written to a WORM store is irrevocable.
The minimal data-integrity discipline for any inter-agent payload: compute a SHA-256 hash of the payload bytes at the point of writing; transmit or store the hash alongside the payload; verify the hash at the point of reading before trusting the content. This catches corruption at rest, corruption in transit, and any accidental truncation or encoding error.
Hash the canonicalized representation of the payload, not the serialization format. JSON field ordering is implementation-dependent; canonicalize — sort keys deterministically, strip insignificant whitespace, fix the encoding to UTF-8 — before hashing, so the hash is stable across serialization contexts.
Add HMAC-SHA256 when the threat model includes an actor who can write to the store or the channel. The signing key must be kept out of the store. Store it in a secret manager with per-agent access grants, so only the intended writer can retrieve the signing key and only the intended reader can retrieve the verification key.
The payload envelope should carry: the payload body, the HMAC tag, a sequence number, a timestamp (ISO-8601, UTC), and the writer's identity. The reader verifies the HMAC before deserializing the body. If verification fails, the reader rejects the payload and logs a tamper-detected event. The failure mode of a bad HMAC is a hard stop — never a soft warning that passes the content forward.
hmac.compare_digest() for the comparison, not a direct equality check. Direct equality returns early on the first byte mismatch, leaking timing information an attacker can use to learn the expected tag byte by byte. compare_digest() runs in constant time regardless of where the mismatch occurs.
Separate the data path from the audit path. Application agents read and write working data through the normal store. Every write event also produces an audit record that goes to the WORM-committed log. The audit record carries the writer identity, the payload hash, the HMAC tag, the sequence number, and the UTC timestamp.
The WORM log is write-only from the perspective of the application agents. No agent has delete permission on the log store. Retention policy is set by operations, not by the agents. Once a record lands in the log, it stays for the retention period regardless of what happens to the application-layer data.
Three agents: an orchestrator that fetches external market data, a normalizer that re-formats it, and a signal-generator that produces trade signals. Two data seams: orchestrator-to-normalizer and normalizer-to-signal-generator.
At each seam, the writer produces an IntegrityEnvelope:
{
"seq": 10042,
"ts": "2026-06-23T10:31:00Z",
"writer": "orchestrator-v2",
"payload_sha256": "a3f9...c8d1",
"hmac_sha256": "7b2e...44fa",
"payload": { ... }
}
The orchestrator retrieves its signing key from Secrets Manager at startup, keeps it in memory, and uses it to compute the tag for every envelope it writes. The normalizer retrieves the verification key and checks the tag before deserializing the payload. If the check passes, the normalizer processes the data, writes its own signed envelope to the second store, and appends both input and output envelope hashes to the WORM audit log.
The sequence number prevents replay. The normalizer tracks the highest sequence number it has processed. An envelope at or below that number is rejected as a duplicate or replay attempt, regardless of whether the HMAC is valid.
The audit pipeline lesson on 28 May built the action-log infrastructure. This lesson specifies what each record in that channel must contain and where it must land: WORM storage, not a mutable log the compromised application layer can overwrite.
The reorg-safe ingestion lesson on 11 June addressed the upstream seam: when an external source retracts finality, the pipeline detects and backfills. The integrity discipline addresses the internal seam: when an agent writes data, the receiving agent must prove it arrived unaltered. Both govern data correctness at different seam types.
Runtime confinement on 17 June removed the kernel surface a compromised process could use against the host. A compromised agent that can write to a shared store can corrupt the data, but it cannot forge a valid HMAC without the signing key, and any corruption it introduces fails the receiver's hash check.
Python's hashlib module implements SHA-256 and SHA-3-256 directly in the standard library. Its hmac module implements HMAC-SHA256 with constant-time comparison via compare_digest(). Today's Python lesson implements the IntegrityEnvelope as a typed dataclass, writes a sign() function the orchestrator calls at write time, and a verify() function the consumer calls before deserializing any payload.
Write the hash at the point of production, not at the point of storage. Verify at the point of consumption, not at the point of display. Use HMAC when the threat model includes a writer who can reach the store. Write every event to an append-only WORM log that no application agent can delete.
The rule states what was written, who wrote it, when, and what the content was. An audit trail built on this rule is a fact record. An audit trail built without it is a story anyone with write access can edit.
Examine this against every data seam in your pipeline. Where the rule is not in place, the seam is open.