← Back to the homepage

Backprop Bulletin practical guide

Build an AI workflow that can resume after an interrupted run

A batch job that calls a model and then acts on the result will sooner or later be interrupted halfway. This guide builds a small workflow that picks up where it stopped, keeps the work it already finished, and checks before repeating an action whose outcome it cannot know.

By Backprop Bulletin. Tested on 16 November 2026 with Node.js 24.22.1 and 22.23.5 on Linux.

About this guide. It was written for this site by Backprop Bulletin with AI assistance. Every command below was run as shown, and each expected output is copied from that run; an automated check runs them again. No outside expert has reviewed it. The example uses invented tickets, a simulated model and a simulated helpdesk. It needs no account, key, network connection or paid service, and it sends nothing anywhere.

What you will build

The workflow handles five invented support tickets. For each ticket it takes two steps:

  1. Draft a reply with a model. Here the model is simulated: a few lines of code that pick a category and a reply template, and that can be told to fail.
  2. Deliver the reply to a helpdesk. The helpdesk is also simulated, by a file in the working folder. It behaves like many real external systems, such as email, chat or a ticketing service: it does not recognise a repeated request, so sending the same reply twice creates two messages. It does let you look up a message by a reference you sent with it.

The workflow records its progress in a small SQLite database, workflow.db. The helpdesk keeps its own file, helpdesk.json. They are deliberately separate: no transaction in your database can include a change made by another system. That gap is the subject of this guide.

You will run three cases: a normal run; a run that is killed partway and then restarted; and a delivery whose result never comes back, so the workflow cannot tell whether it happened.

Prerequisites and tested runtime

Create an empty folder, open a terminal in it and download the example into it:

Download resumable-workflow.mjs JavaScript, 242 lines, 13.4 kB

Before running a downloaded file, read it. You can also check that you have the same file this guide was tested with:

Run

sha256sum resumable-workflow.mjs

Expected output

9dd42ae05a041cb6fcaf2839b8e75f13234f7e47f9f5ffa8a70a4bc0cb3f8d04  resumable-workflow.mjs

On macOS, use shasum -a 256 resumable-workflow.mjs instead.

The four ideas

1. Durable checkpoints

Each change to a task's state is one committed SQLite transaction. SQLite's atomic commit means that "either all database changes within a single transaction occur or none of them occur", even if the process is killed or the machine loses power during the commit. After a crash, every task is therefore wholly in its old state or wholly in its new one.

function commit(db, fn) {
  db.exec("BEGIN IMMEDIATE");
  try {
    fn();
    db.exec("COMMIT");
  } catch (error) {
    db.exec("ROLLBACK");
    throw error;
  }
}

The database is opened in write-ahead-log mode with full synchronisation. SQLite documents this combination as atomic, consistent, isolated and durable; with its default journal, full synchronisation is not necessarily durable across a power loss.

  db.exec("PRAGMA journal_mode = WAL; PRAGMA synchronous = FULL;");

2. Stable task identity

A restarted run has to recognise the work it already did. Each task's id is therefore derived from its input, not chosen at random: the same ticket text always gives the same id. Tasks are inserted with INSERT OR IGNORE, so running again adds nothing that already exists. The id is also the reference sent with each reply, which is what makes the later lookup possible.

const taskIdOf = (t) => `${t.id}-${createHash("sha256").update(`${WORKFLOW_VERSION}\n${t.id}\n${t.text}`).digest("hex").slice(0, 8)}`;

Changing a ticket's text, or the workflow version, deliberately creates a new task.

3. Retry limits

A model call that fails with a temporary error is retried, up to a limit. Three attempts total: one initial attempt and up to two retries. The workflow waits 0.2 s before the first retry and 0.4 s before the second. The attempt counter is committed before each call, so the limit also holds across crashes: a workflow that keeps crashing on the same ticket cannot retry it forever. A task that reaches the limit is marked failed and is not retried until you say so.

      const attempt = task.model_attempts + 1;
      commit(db, () => update(db, id, { model_attempts: attempt }));

4. Recovery and reconciliation

Retrying is safe only when repeating the action does no harm. RFC 9110, the HTTP specification, says a client "SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied." Delivering a reply to this helpdesk is not idempotent, so the workflow records its intent (sending) before the call and the result (sent) after it:

    commit(db, () => update(db, id, { state: "sending" }));
      const message = desk.send({ reference: id, ticket: ticket.id, body: task.reply }, fault);
      if (opts["crash-while-sending"] === ticket.id) await crash(`after the helpdesk received ${ticket.id}, before the result was recorded`);
      commit(db, () => update(db, id, { state: "sent", helpdesk_message: message }));

(The middle line exists only to simulate a crash for this guide.) If the call times out, or the process dies between those two records, the task's outcome is unknown. The affected ticket is held without resending it. Other tickets may finish during the current run. A subsequent run pauses until unknown delivery outcomes are reconciled. The reconcile command asks the helpdesk whether a message with that reference exists:

    const found = desk.findByReference(row.task_id);
    if (found) {
      commit(db, () => update(db, row.task_id, { state: "sent", helpdesk_message: found.id, note: "confirmed by reconciliation" }));

If the message is found, the task is recorded as sent. If it is not, the task is marked ready to send again.

Case 1: a normal run

Run the workflow on all five tickets, using a working folder called demo-1:

Run

node resumable-workflow.mjs run --dir demo-1; echo "exit status: $?"

Expected output

run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 5
exit status: 0

The status command shows each task's id, state and model attempts, and how many copies of its reply the helpdesk holds:

Run

node resumable-workflow.mjs status --dir demo-1

Expected output

T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 sent, model attempts 1, helpdesk message 3; helpdesk copies: 1
T-1004-16d41b58 sent, model attempts 1, helpdesk message 4; helpdesk copies: 1
T-1005-68d0c897 sent, model attempts 1, helpdesk message 5; helpdesk copies: 1

Running it again changes nothing: every task is already sent, and no model is called.

Case 2: interrupted, then restarted

The --crash-after 2 option makes the process kill itself with SIGKILL after two replies are sent. Node.js documents that SIGKILL "cannot have a listener installed, it will unconditionally terminate Node.js", so no cleanup code runs, much as when a machine loses power or an operator kills a stuck job.

Run

node resumable-workflow.mjs run --dir demo-2 --crash-after 2; echo "exit status: $?"

Expected output

run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
simulated interruption: the process is killed (SIGKILL) after 2 tickets were sent
exit status: 137

Your shell may also print a line such as Killed. Exit status 137 means the process was ended by signal 9, SIGKILL. The two finished tickets were committed before the crash:

Run

node resumable-workflow.mjs status --dir demo-2

Expected output

T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 pending, model attempts 0; helpdesk copies: 0
T-1004-16d41b58 pending, model attempts 0; helpdesk copies: 0
T-1005-68d0c897 pending, model attempts 0; helpdesk copies: 0

Restart with the same command, without the crash option:

Run

node resumable-workflow.mjs run --dir demo-2; echo "exit status: $?"

Expected output

run: 5 tickets, 2 already sent (kept, not repeated)
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 3
exit status: 0

Only the three unfinished tickets were processed: three model calls, not five, and each ticket has exactly one reply at the helpdesk.

The worst moment to be interrupted

A crash at a checkpoint is the easy case. The hard one is a crash after the helpdesk received a reply but before the workflow recorded it. --crash-while-sending T-1003 simulates exactly that:

Run

node resumable-workflow.mjs run --dir demo-2b --crash-while-sending T-1003; echo "exit status: $?"

Expected output

run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
simulated interruption: the process is killed (SIGKILL) after the helpdesk received T-1003, before the result was recorded
exit status: 137

T-1003 was left in the sending state. On restart, the workflow treats it as an unknown outcome and stops rather than send it a second time:

Run

node resumable-workflow.mjs run --dir demo-2b; echo "exit status: $?"

Expected output

run: stopped: T-1003 has an unknown delivery outcome. Run "reconcile" first. Nothing was sent.
exit status: 3

Run

node resumable-workflow.mjs reconcile --dir demo-2b

Expected output

reconcile: 1 task with an unknown delivery outcome
T-1003 found at the helpdesk as message 3: marked sent, not sent again

Run

node resumable-workflow.mjs run --dir demo-2b; echo "exit status: $?"

Expected output

run: 5 tickets, 3 already sent (kept, not repeated)
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 2
exit status: 0

Case 3: an outcome the workflow cannot know

Two failures look identical from the workflow's side. With --lose-response T-1003, the helpdesk stores T-1003's reply but its answer never arrives. With --lose-request T-1004, T-1004's request never reaches the helpdesk. In both cases the workflow sees only a timeout.

Run

node resumable-workflow.mjs run --dir demo-3 --lose-response T-1003 --lose-request T-1004; echo "exit status: $?"

Expected output

run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 drafted: bug (model attempt 1)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 send timed out: delivery outcome unknown, not retried
T-1004 drafted: privacy (model attempt 1)
T-1004 send timed out: delivery outcome unknown, not retried
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 4
summary: 3 sent, 0 failed, 2 unknown; simulated model calls in this run: 5
next step: run "reconcile" before anything is sent again
exit status: 3

Neither common shortcut is safe here. Sending both again would give the T-1003 customer two replies. Treating both as sent would leave the T-1004 customer with none. A plain rerun refuses to guess:

Run

node resumable-workflow.mjs run --dir demo-3; echo "exit status: $?"

Expected output

run: stopped: T-1003, T-1004 have an unknown delivery outcome. Run "reconcile" first. Nothing was sent.
exit status: 3

Reconciliation asks the helpdesk what actually happened:

Run

node resumable-workflow.mjs reconcile --dir demo-3

Expected output

reconcile: 2 tasks with an unknown delivery outcome
T-1003 found at the helpdesk as message 3: marked sent, not sent again
T-1004 not found at the helpdesk: marked ready to send again

Now the next run sends only what is missing, reusing T-1004's saved draft without calling the model again:

Run

node resumable-workflow.mjs run --dir demo-3; echo "exit status: $?"

Expected output

run: 5 tickets, 4 already sent (kept, not repeated)
T-1004 draft kept from an earlier run
T-1004 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 0
exit status: 0

Run

node resumable-workflow.mjs status --dir demo-3

Expected output

T-1001-e2ebe6c3 sent, model attempts 1, helpdesk message 1; helpdesk copies: 1
T-1002-89e8a7ed sent, model attempts 1, helpdesk message 2; helpdesk copies: 1
T-1003-6bc5f001 sent, model attempts 1, helpdesk message 3; helpdesk copies: 1
T-1004-16d41b58 sent, model attempts 1, helpdesk message 5; helpdesk copies: 1
T-1005-68d0c897 sent, model attempts 1, helpdesk message 4; helpdesk copies: 1

Every ticket has exactly one reply.

Retry limits in action

--flaky-model T-1002 makes the model fail once on T-1002. --model-down T-1005 makes it fail every time on T-1005. This run takes about a second longer because of the waits between attempts.

Run

node resumable-workflow.mjs run --dir demo-4 --flaky-model T-1002 --model-down T-1005; echo "exit status: $?"

Expected output

run: 5 tickets, 0 already sent (kept, not repeated)
T-1001 drafted: billing (model attempt 1)
T-1001 sent: helpdesk message 1
T-1002 model attempt 1 of 3 failed: model temporarily unavailable
T-1002 drafted: bug (model attempt 2)
T-1002 sent: helpdesk message 2
T-1003 drafted: how-to (model attempt 1)
T-1003 sent: helpdesk message 3
T-1004 drafted: privacy (model attempt 1)
T-1004 sent: helpdesk message 4
T-1005 model attempt 1 of 3 failed: model temporarily unavailable
T-1005 model attempt 2 of 3 failed: model temporarily unavailable
T-1005 model attempt 3 of 3 failed: model temporarily unavailable
T-1005 giving up after 3 model attempts: marked failed
summary: 4 sent, 1 failed, 0 unknown; simulated model calls in this run: 8
exit status: 2

A plain rerun leaves the failed task alone:

Run

node resumable-workflow.mjs run --dir demo-4; echo "exit status: $?"

Expected output

run: 5 tickets, 4 already sent (kept, not repeated)
T-1005 failed earlier after 3 model attempts: not retried (use --retry-failed)
summary: 4 sent, 1 failed, 0 unknown; simulated model calls in this run: 0
exit status: 2

Once the cause is fixed (here, simply by dropping --model-down), ask for the failed task to be tried again:

Run

node resumable-workflow.mjs run --dir demo-4 --retry-failed; echo "exit status: $?"

Expected output

run: 5 tickets, 4 already sent (kept, not repeated)
T-1005 drafted: outage (model attempt 1)
T-1005 sent: helpdesk message 5
summary: 5 sent, 0 failed, 0 unknown; simulated model calls in this run: 1
exit status: 0

Recovery steps

The exit status tells you what to do next: 0 means everything is sent, 2 means some tasks failed, 3 means some outcomes are unknown and 64 means the command was wrong.

  1. After any interruption, run the same command again. Finished tasks are kept, and drafts already made are reused.
  2. If it stops with an unknown outcome (exit status 3), run reconcile, read what it found, then run again. Do not edit workflow.db by hand to force a resend.
  3. If the external system cannot answer the lookup, leave the task unknown and check that system yourself. Sending again on a guess is how duplicates happen. This example's simulated helpdesk always answers, but a real one may not.
  4. If tasks failed (exit status 2), find the cause first, then rerun with --retry-failed.
  5. Do not delete workflow.db to "start fresh" while the external system holds earlier deliveries. The workflow would forget what it sent and send it again.

Clean up

The example wrote only the demo-* folders, each holding workflow.db (plus workflow.db-wal and workflow.db-shm after a crash) and helpdesk.json. Delete them, or the whole folder you created:

rm -rf demo-1 demo-2 demo-2b demo-3 demo-4

What this does not do

Sources

Technical statements on this page were checked against these primary sources on 16 November 2026: