Backprop Bulletin practical guide
Build an AI workflow that can resume after an interrupted run
A batch job that calls a model and then acts on the result will sooner or later be interrupted halfway. This guide builds a small workflow that picks up where it stopped, keeps the work it already finished, and checks before repeating an action whose outcome it cannot know.
What you will build
The workflow handles five invented support tickets. For each ticket it takes two steps:
- Draft a reply with a model. Here the model is simulated: a few lines of code that pick a category and a reply template, and that can be told to fail.
- Deliver the reply to a helpdesk. The helpdesk is also simulated, by a file in the working folder. It behaves like many real external systems, such as email, chat or a ticketing service: it does not recognise a repeated request, so sending the same reply twice creates two messages. It does let you look up a message by a reference you sent with it.
The workflow records its progress in a small SQLite database, workflow.db. The helpdesk keeps its own file, helpdesk.json. They are deliberately separate: no transaction in your database can include a change made by another system. That gap is the subject of this guide.
You will run three cases: a normal run; a run that is killed partway and then restarted; and a delivery whose result never comes back, so the workflow cannot tell whether it happened.
Prerequisites and tested runtime
- Node.js 22.13.0 or later. The example uses Node's built-in
node:sqlitemodule, which needs no flag from 22.13.0 and 23.4.0. Nothing else is installed: there are no packages and no build step. - Tested with Node.js 24.22.1 and 22.23.5 on Linux, in Bash. On Node.js 22 the module is marked experimental, so it prints an
ExperimentalWarningto the error stream; everything else, including the output shown here, is the same. - A terminal running Bash, where the commands were tested. They use
echo "exit status: $?"to show each program's exit status. - An empty folder with about 100 kB of free space. Everything the example writes stays in the folder you name with
--dir.
Create an empty folder, open a terminal in it and download the example into it:
Download resumable-workflow.mjs JavaScript, 242 lines, 13.4 kB
Before running a downloaded file, read it. You can also check that you have the same file this guide was tested with:
Run
sha256sum resumable-workflow.mjs
Expected output
9dd42ae05a041cb6fcaf2839b8e75f13234f7e47f9f5ffa8a70a4bc0cb3f8d04 resumable-workflow.mjs
On macOS, use shasum -a 256 resumable-workflow.mjs instead.
The four ideas
1. Durable checkpoints
Each change to a task's state is one committed SQLite transaction. SQLite's atomic commit means that "either all database changes within a single transaction occur or none of them occur", even if the process is killed or the machine loses power during the commit. After a crash, every task is therefore wholly in its old state or wholly in its new one.
function commit(db, fn) {
db.exec("BEGIN IMMEDIATE");
try {
fn();
db.exec("COMMIT");
} catch (error) {
db.exec("ROLLBACK");
throw error;
}
}
The database is opened in write-ahead-log mode with full synchronisation. SQLite documents this combination as atomic, consistent, isolated and durable; with its default journal, full synchronisation is not necessarily durable across a power loss.
db.exec("PRAGMA journal_mode = WAL; PRAGMA synchronous = FULL;");
2. Stable task identity
A restarted run has to recognise the work it already did. Each task's id is therefore derived from its input, not chosen at random: the same ticket text always gives the same id. Tasks are inserted with INSERT OR IGNORE, so running again adds nothing that already exists. The id is also the reference sent with each reply, which is what makes the later lookup possible.
const taskIdOf = (t) => `${t.id}-${createHash("sha256").update(`${WORKFLOW_VERSION}\n${t.id}\n${t.text}`).digest("hex").slice(0, 8)}`;
Changing a ticket's text, or the workflow version, deliberately creates a new task.
3. Retry limits
A model call that fails with a temporary error is retried, up to a limit. Three attempts total: one initial attempt and up to two retries. The workflow waits 0.2 s before the first retry and 0.4 s before the second. The attempt counter is committed before each call, so the limit also holds across crashes: a workflow that keeps crashing on the same ticket cannot retry it forever. A task that reaches the limit is marked failed and is not retried until you say so.
const attempt = task.model_attempts + 1;
commit(db, () => update(db, id, { model_attempts: attempt }));
4. Recovery and reconciliation
Retrying is safe only when repeating the action does no harm. RFC 9110, the HTTP specification, says a client "SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied." Delivering a reply to this helpdesk is not idempotent, so the workflow records its intent (sending) before the call and the result (sent) after it:
commit(db, () => update(db, id, { state: "sending" }));
const message = desk.send({ reference: id, ticket: ticket.id, body: task.reply }, fault);
if (opts["crash-while-sending"] === ticket.id) await crash(`after the helpdesk received ${ticket.id}, before the result was recorded`);
commit(db, () => update(db, id, { state: "sent", helpdesk_message: message }));
(The middle line exists only to simulate a crash for this guide.) If the call times out, or the process dies between those two records, the task's outcome is unknown. The affected ticket is held without resending it. Other tickets may finish during the current run. A subsequent run pauses until unknown delivery outcomes are reconciled. The reconcile command asks the helpdesk whether a message with that reference exists:
const found = desk.findByReference(row.task_id);
if (found) {
commit(db, () => update(db, row.task_id, { state: "sent", helpdesk_message: found.id, note: "confirmed by reconciliation" }));
If the message is found, the task is recorded as sent. If it is not, the task is marked ready to send again.
Case 1: a normal run
Run the workflow on all five tickets, using a working folder called demo-1:
Run
node resumable-workflow.mjs run --dir demo-1; echo "exit status: $?"
Expected output
run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 5
exit status: 0
The status command shows each task's id, state and model attempts, and how many copies of its reply the helpdesk holds:
Run
node resumable-workflow.mjs status --dir demo-1
Expected output
T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 sent, model attempts 1, helpdesk message 3; helpdesk copies: 1
T-1004-16d41b58 sent, model attempts 1, helpdesk message 4; helpdesk copies: 1
T-1005-68d0c897 sent, model attempts 1, helpdesk message 5; helpdesk copies: 1
Running it again changes nothing: every task is already sent, and no model is called.
Case 2: interrupted, then restarted
The --crash-after 2 option makes the process kill itself with SIGKILL after two replies are sent. Node.js documents that SIGKILL "cannot have a listener installed, it will unconditionally terminate Node.js", so no cleanup code runs, much as when a machine loses power or an operator kills a stuck job.
Run
node resumable-workflow.mjs run --dir demo-2 --crash-after 2; echo "exit status: $?"
Expected output
run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
simulated interruption: the process is killed (SIGKILL) after 2 tickets were sent
exit status: 137
Your shell may also print a line such as Killed. Exit status 137 means the process was ended by signal 9, SIGKILL. The two finished tickets were committed before the crash:
Run
node resumable-workflow.mjs status --dir demo-2
Expected output
T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 pending, model attempts 0; helpdesk copies: 0
T-1004-16d41b58 pending, model attempts 0; helpdesk copies: 0
T-1005-68d0c897 pending, model attempts 0; helpdesk copies: 0
Restart with the same command, without the crash option:
Run
node resumable-workflow.mjs run --dir demo-2; echo "exit status: $?"
Expected output
run: 5 tickets, 2 already sent (kept, not repeated)
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 3
exit status: 0
Only the three unfinished tickets were processed: three model calls, not five, and each ticket has exactly one reply at the helpdesk.
The worst moment to be interrupted
A crash at a checkpoint is the easy case. The hard one is a crash after the helpdesk received a reply but before the workflow recorded it. --crash-while-sending T-1003 simulates exactly that:
Run
node resumable-workflow.mjs run --dir demo-2b --crash-while-sending T-1003; echo "exit status: $?"
Expected output
run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
simulated interruption: the process is killed (SIGKILL) after the helpdesk received T-1003, before the result was recorded
exit status: 137
T-1003 was left in the sending state. On restart, the workflow treats it as an unknown outcome and stops rather than send it a second time:
Run
node resumable-workflow.mjs run --dir demo-2b; echo "exit status: $?"
Expected output
run: stopped: T-1003 has an unknown delivery outcome. Run "reconcile" first. Nothing was sent.
exit status: 3
Run
node resumable-workflow.mjs reconcile --dir demo-2b
Expected output
reconcile: 1 task with an unknown delivery outcome
T-1003 found at the helpdesk as message 3: marked sent, not sent again
Run
node resumable-workflow.mjs run --dir demo-2b; echo "exit status: $?"
Expected output
run: 5 tickets, 3 already sent (kept, not repeated)
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 2
exit status: 0
Case 3: an outcome the workflow cannot know
Two failures look identical from the workflow's side. With --lose-response T-1003, the helpdesk stores T-1003's reply but its answer never arrives. With --lose-request T-1004, T-1004's request never reaches the helpdesk. In both cases the workflow sees only a timeout.
Run
node resumable-workflow.mjs run --dir demo-3 --lose-response T-1003 --lose-request T-1004; echo "exit status: $?"
Expected output
run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 send timed out: delivery outcome unknown, not retried
T-1004 drafted: privacy (model attempt 1)
T-1004 send timed out: delivery outcome unknown, not retried
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 4
summary: 3 sent, 0 failed, 2 unknown; simulated model calls in this run: 5
next step: run "reconcile" before anything is sent again
exit status: 3
Neither common shortcut is safe here. Sending both again would give the T-1003 customer two replies. Treating both as sent would leave the T-1004 customer with none. A plain rerun refuses to guess:
Run
node resumable-workflow.mjs run --dir demo-3; echo "exit status: $?"
Expected output
run: stopped: T-1003, T-1004 have an unknown delivery outcome. Run "reconcile" first. Nothing was sent.
exit status: 3
Reconciliation asks the helpdesk what actually happened:
Run
node resumable-workflow.mjs reconcile --dir demo-3
Expected output
reconcile: 2 tasks with an unknown delivery outcome
T-1003 found at the helpdesk as message 3: marked sent, not sent again
T-1004 not found at the helpdesk: marked ready to send again
Now the next run sends only what is missing, reusing T-1004's saved draft without calling the model again:
Run
node resumable-workflow.mjs run --dir demo-3; echo "exit status: $?"
Expected output
run: 5 tickets, 4 already sent (kept, not repeated)
T-1004 draft kept from an earlier run
T-1004 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 0
exit status: 0
Run
node resumable-workflow.mjs status --dir demo-3
Expected output
T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 sent, model attempts 1, helpdesk message 3; helpdesk copies: 1
T-1004-16d41b58 sent, model attempts 1, helpdesk message 5; helpdesk copies: 1
T-1005-68d0c897 sent, model attempts 1, helpdesk message 4; helpdesk copies: 1
Every ticket has exactly one reply.
Retry limits in action
--flaky-model T-1002 makes the model fail once on T-1002. --model-down T-1005 makes it fail every time on T-1005. This run takes about a second longer because of the waits between attempts.
Run
node resumable-workflow.mjs run --dir demo-4 --flaky-model T-1002 --model-down T-1005; echo "exit status: $?"
Expected output
run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 model attempt 1 of 3 failed: model temporarily unavailable
T-1002 drafted: bug (model attempt 2)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 model attempt 1 of 3 failed: model temporarily unavailable
T-1005 model attempt 2 of 3 failed: model temporarily unavailable
T-1005 model attempt 3 of 3 failed: model temporarily unavailable
T-1005 giving up after 3 model attempts: marked failed
summary: 4 sent, 1 failed, 0 unknown; simulated model calls in this run: 8
exit status: 2
A plain rerun leaves the failed task alone:
Run
node resumable-workflow.mjs run --dir demo-4; echo "exit status: $?"
Expected output
run: 5 tickets, 4 already sent (kept, not repeated)
T-1005 failed earlier after 3 model attempts: not retried (use --retry-failed)
summary: 4 sent, 1 failed, 0 unknown; simulated model calls in this run: 0
exit status: 2
Once the cause is fixed (here, simply by dropping --model-down), ask for the failed task to be tried again:
Run
node resumable-workflow.mjs run --dir demo-4 --retry-failed; echo "exit status: $?"
Expected output
run: 5 tickets, 4 already sent (kept, not repeated)
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 1
exit status: 0
Recovery steps
The exit status tells you what to do next: 0 means everything is sent, 2 means some tasks failed, 3 means some outcomes are unknown and 64 means the command was wrong.
- After any interruption, run the same command again. Finished tasks are kept, and drafts already made are reused.
- If it stops with an unknown outcome (exit status 3), run
reconcile, read what it found, then run again. Do not editworkflow.dbby hand to force a resend. - If the external system cannot answer the lookup, leave the task unknown and check that system yourself. Sending again on a guess is how duplicates happen. This example's simulated helpdesk always answers, but a real one may not.
- If tasks failed (exit status 2), find the cause first, then rerun with
--retry-failed. - Do not delete
workflow.dbto "start fresh" while the external system holds earlier deliveries. The workflow would forget what it sent and send it again.
Clean up
The example wrote only the demo-* folders, each holding workflow.db (plus workflow.db-wal and workflow.db-shm after a crash) and helpdesk.json. Delete them, or the whole folder you created:
rm -rf demo-1 demo-2 demo-2b demo-3 demo-4
What this does not do
- It is not exactly-once delivery. The database guarantees only its own state. The helpdesk is outside every local transaction, so the workflow avoids duplicates by recording intent, refusing to guess and reconciling. That depends on the external system offering a reliable lookup or some other way to detect a repeat, such as an idempotency key it honours.
- Model calls can repeat. If the process dies during a model call, the next run calls the model again. With a real provider, that can cost money and may produce a different draft. The retry limit bounds the repeats; it does not prevent them.
- A lookup can be wrong. Some systems show a new record only after a delay. If "not found" might mean "not yet visible", wait or check another way before sending again.
- Durability depends on your storage. SQLite "assumes that the operating system that it is running on works as advertised". Its documentation notes that flush and fsync are broken on some versions of Windows and Linux, which can corrupt a database after a power loss.
- One run at a time. The example has no lock. Two copies running against the same folder could both send the same reply.
- It does not fix connection errors. It does not stop timeouts or connection failures from any model provider, Claude included. It limits what such failures can do to your workflow: lost work, endless retries and duplicate actions.
- It is a teaching example. The model and the helpdesk are simulated, the inputs are invented, and there is no authentication, permission check or monitoring. Before adapting it, replace the simulated parts with your real ones and test the same three cases against them.
Sources
Technical statements on this page were checked against these primary sources on 16 November 2026:
- SQLite: Atomic Commit In SQLite: atomic commit through a crash or power failure, and the assumption that flush and fsync work as advertised.
- SQLite: PRAGMA synchronous: FULL is atomic, consistent, isolated and durable in WAL mode, and not necessarily durable across a power loss with a rollback journal.
- SQLite: Transaction:
BEGIN IMMEDIATEstarts a write transaction at once. - Node.js: SQLite (node:sqlite): added in 22.5.0, no longer behind a flag from 22.13.0 and 23.4.0, and a release candidate from 25.7.0.
- Node.js: process signal events:
SIGKILLcannot have a listener and unconditionally terminates Node.js. - RFC 9110, HTTP Semantics, section 9.2.2: Idempotent Methods: when a request may be repeated automatically, and when it should not be.