Held together by one integer
An agent working on real infrastructure does not finish in one request. In Groundwork, a single activity is allowed to run for four hours, inside a sandbox, streaming tool calls and token deltas the whole way. Somewhere there is a browser tab that wants to watch. The tab will be refreshed. The laptop will sleep. The WiFi will drop mid-word. And when the tab comes back, it must show exactly what happened, in order, with nothing missing and nothing twice, while the agent keeps typing at the far end.
The standard answers to this problem are heavy. A durable message broker with acknowledgments. An outbox table with a relay process. Exactly-once delivery, purchased with operational complexity. Groundwork’s answer is an integer column on the session row, and I want to walk through why that is enough, because the reasoning generalizes far beyond this codebase.
Stamp first, under a lock
Every event in a session, a token delta, a tool call, an approval request,
gets appended through one transaction. It locks the session row with
select ... for update, reads last_sequence, stamps each event in the
batch with the next numbers, inserts them, and bumps the counter. The lock
serializes every writer in that session, so sequences are contiguous by
construction: after event 41 there is always a 42, never a jump to 44. A
unique index on the session id and sequence turns any violation of that
promise into a loud database error rather than a quiet reordering. An
integration test fires ten appends in parallel and asserts the sequences come
out gapless.
One transaction stamps the batch, inserts the rows, and bumps the counter. Contiguity is the whole point.
The cost is a hot row: every append in a session queues on the same lock. Groundwork accepts this because sessions run one turn at a time by design, and a turn’s events arrive from one producer. The lock is not contention, it is the ordering mechanism. Contiguity matters more than write parallelism here, and the next two sections are about why.
Two channels, one source of truth
After the transaction commits, the same batch is published to NATS on the subject for that session, and this is where the design gets opinionated: the publish is fire and forget. Core NATS, no persistence, no acknowledgment, no retry. The entire dual-write is five lines:
export async function appendAndPublishEvents(db, bus, sessionId, events) {
const appended = await appendSessionEvents(db, sessionId, events);
await bus.publish(sessionId, appended);
return appended;
}If the process dies between the commit and the publish, the events are durable but nobody was told. If NATS hiccups, a batch simply never arrives. Textbooks call this the dual-write problem and prescribe an outbox: write the intent to a table in the same transaction, have a relay deliver it, track acknowledgments. Groundwork does none of that, on purpose.
Durable first, broadcast second. The dashed channel is allowed to lose, because every loss is detectable.
The bet is this: a lossy broadcast channel is fine if every consumer can cheaply detect what it missed and fetch it from the source of truth. The sequence number is what makes detection cheap. You do not need the channel to be reliable. You need the gaps to be visible, and a contiguous integer makes a gap as visible as arithmetic.
The read path: subscribe first, then repair forever
When a browser opens a session’s event stream, the server does four things in a deliberately strange order. It subscribes to NATS first, but holds everything that arrives in a buffer. Then it replays history from Postgres, paging by sequence, a thousand rows at a time. Then it drains the buffer, and only then announces the stream is connected. Subscribing before replaying closes the classic race where an event lands between “read the history” and “start listening.”
From then on, every live event passes one check before it reaches the client:
if (event.sequence <= lastSent) return;
if (event.sequence > lastSent + 1) {
const missing = await listSessionEvents(db, sessionId, lastSent, gap);
// ...send the missing rows first
}One comparison handles duplicates, they are simply below the water line, and one comparison detects losses, at which point the server fetches exactly the missing rows from Postgres and sends them before the event that revealed the gap. The client cannot tell repaired events from lucky ones. Try it:
Events stream once a second. Drop a broadcast, then watch the next arrival repair the gap from Postgres.
postgres ledger · last_sequence = 0
the browser · lastSent = 0
The repair loop, live. Note that a dropped broadcast is invisible until the next event arrives; that blind spot is real and discussed below.
Reconnection is the same mechanism wearing a different hat. Each SSE frame
carries id: {sequence}, so a standard EventSource would resume via
Last-Event-ID for free. Groundwork’s own web client uses fetch instead, to
send an auth header, parses the frames by hand, tracks the highest sequence
it has seen, and reconnects with that cursor under exponential backoff, half
a second doubling to five. History pagination uses the same cursor. One
integer is simultaneously the ordering key, the SSE event id, the resume
token, and the pagination cursor. When one value does four jobs, the four
jobs cannot disagree with each other.
The producer side has tripwires, not safety nets
The events are produced inside one enormous Temporal activity, the entire agent loop, with a four-hour ceiling and a heartbeat every ten seconds. Every append doubles as a heartbeat, so liveness reporting is a side effect of doing work rather than a separate obligation. Token deltas are batched before they touch the locked row: a batch flushes at fifty events, or after 33 milliseconds, roughly a 30 fps UI cadence, or immediately when something structural happens, a tool call created, a turn completed, an approval required. Structure must never render late; prose can wait a frame.
Duplicates are handled with a philosophy I have come to respect. Each event carries a producer id and a producer-local counter, covered by a unique index, and there is no on-conflict clause anywhere. Temporal retries for the agent activity are set to one attempt, maximum. So a duplicate append is not a normal thing to be quietly absorbed; it is impossible by construction, and if it happens anyway the insert throws. The unique index is a tripwire, not a safety net. Silent deduplication would have masked exactly the class of bug the design promises cannot exist.
There is a deeper consequence hiding in that one-attempt policy. You cannot replay half of a conversation with a language model and expect the same tokens; the activity is not idempotent, so Temporal’s usual retry-until-success contract would be a lie. Groundwork chooses at-most-once and makes failure explicit: if the worker crashes mid-turn, the session fails, visibly. The event log means you can see exactly how far it got.
The log is the machine, not a diary of it
The part I find most elegant only appears once you notice what else reads
this table. A user message is not a request that happens to get logged; it
is appended as an event, and its row id becomes the trigger of a queued
turn. The workflow claims turns with for update skip locked. An approval
decision is appended as an event, and its id is signaled into the workflow
to become the next segment’s trigger. The event log is not observability
bolted onto the system. It is the coordination substrate, and the UI is
just one more consumer doing exactly what the worker does: reading the
ledger in order.
Scar tissue
The honest ledger, then. Nothing prunes these tables: events and serialized run states accumulate for the life of the deployment, and a busy session’s workflow loops forever without Temporal’s continue-as-new, which means unbounded history on both sides of the fence. The SSE path has no backpressure; a slow consumer buffers on the server. And the repair mechanism has a genuine blind spot you can reproduce in the widget above: gap detection needs a later event to arrive, so if the very last broadcast of a burst is the one that is lost, including a terminal event like a turn completing, the client stays stale until something else moves or the tab reconnects. Each of these is a coherent trade rather than an accident, the code is consistent about all of them, but a second deployment of this system at scale would need answers where today there are none.
What stays with me is the shape of the design. Reliability is usually described as a property of channels, something you buy with brokers and acknowledgments and idempotent consumers. Groundwork treats it as a property of arithmetic: make the source of truth cheap to query, make every message carry its position, and the most unreliable channel in the building becomes good enough.
Number your events, and you can afford to lose them.