LangGraph Crash Recovery Replays Tasks: A Hands-On SQLite Atomic Commit Test

LangGraph's durable execution lets workflows recover after a crash, but unfinished tasks are re-executed upon recovery—is this by design, or a hidden trap? A developer shared a runnable crash test on Reddit that uses SQLite as a lightweight sink to precisely reproduce the phenomenon of a task being replayed after its database write has already committed when the process is killed, and compares the different outcomes across three transaction modes.

Test Design: Make the Process Die Immediately After Commit

The entire experiment involves no models or external services. A counter task receives an operation ID (operation_id) and executes the logic in SQLite: "first check the dedup table, if unseen then increment the counter, then record the dedup key." The tester deliberately calls os._exit(91) to terminate the process within the time window where "the counter increment has committed but the task has not yet returned," then resumes the same thread with invoke(None, config).

The test environment is precisely pinned: Python 3.14.7, LangGraph 1.2.20, langgraph-checkpoint-sqlite 3.1.3, langgraph-checkpoint 4.3.1, langchain-core 1.7.6, and SQLite 3.53.4. Each mode starts with the counter at zero and reads the final value after recovery. The difference among the three modes (split, atomic, fresh) in the code lies solely in the transaction boundaries and the timing of operation ID generation.

os._exit(91) is fundamentally different from the common sys.exit(): the former directly triggers an OS-level process termination without executing any Python cleanup logic—including finally blocks, atexit callbacks, buffer flushing, and context managers' __exit__ methods. This is exactly why the test chose it: to simulate the crash window where "the task has produced side effects but has not yet returned," one must ensure the process dies before LangGraph has a chance to mark the node as complete. If sys.exit() were used, LangGraph's exception handling path might catch the exit signal and write a failure state, thereby masking the replay behavior. In addition, exit code 91 is a deliberately chosen non-standard value to make it easy to distinguish this artificial termination from real runtime errors in logs or shells. Only by understanding this can one grasp that the test reproduces a "hard crash" rather than a "graceful failure," which follows a different recovery semantics in LangGraph.

Three Modes, Three Outcomes

split mode: The counter increment and dedup record are committed in two separate transactions. The process exits after the first commit; upon recovery the dedup table has no record yet, so the task increments again. The final counter is 2.

atomic mode: The counter increment and dedup record are committed in the same transaction. Upon recovery the dedup record already exists, but the task itself is still re-executed—only this time the execution finds the operation ID already processed and skips the increment. The final counter is 1.

fresh mode: Both writes share a transaction, but upon recovery the task internally generates a brand-new operation ID using uuid4().hex. The dedup mechanism fails completely, the counter increments again, and the final value is 2.

You may not have noticed that the call logs (calls.log) for all three modes record two entries. Dedup changes the effect on the sink, not whether the task runs twice.

"At-least-once" and "exactly-once" are two typical message/task delivery semantics in distributed systems. The former guarantees a task is executed at least once but may duplicate during failure recovery; the latter guarantees exactly one execution but usually requires heavyweight mechanisms such as distributed transactions or two-phase commit. LangGraph's durable execution belongs to the former: the checkpoint records "the task has started" rather than "the task has committed," so a task not marked complete after a crash will inevitably be replayed. To approach "exactly-once" on top of this, the common practice is to make the sink idempotent—that is, repeated execution of the same operation ID produces no additional side effects. This is precisely the role played by the SQLite dedup table in the test. It must be emphasized that idempotency is not a guarantee provided by the execution framework but a property the developer must construct on the side-effect side; the framework is only responsible for "at least once," and the responsibility for converging to "once" lies in the business code.

Key Insight: A Single Operation ID Cannot Support Multi-Write Workflows

For this workflow with only one action, the persisted input carries the operation ID, enabling it to be recognized upon recovery. But the tester points out that a workflow containing multiple write actions needs a separate ID for each logical action, rather than using one dedup key for the entire thread. Otherwise, upon recovery it will either re-execute or incorrectly skip subsequent actions.

Another easily overlooked boundary is that this test verifies local process termination and restart, not power loss or HTTP sinks. External APIs need their own idempotency contracts—putting HTTP calls inside a SQLite transaction does not grant database rollback behavior. LangGraph's idempotency guide explains why unfinished tasks run again, while SQLite's transaction documentation defines the boundaries of local writes.

The granularity design of idempotency keys is key to whether an idempotent sink works correctly. Too coarse—for example, one key shared across the entire workflow thread—causes the first action to be processed and subsequent actions to be incorrectly skipped upon recovery; too fine—for example, generating a new key on every retry—makes dedup completely ineffective, as demonstrated by fresh mode. A reasonable design assigns a stable and unique ID to each "logical action," and that ID must remain unchanged when the task is replayed, usually deterministically derived from the workflow's input parameters or upstream nodes rather than randomly generated inside the node. This is consistent with idempotency key practices in payment APIs such as Stripe: the client generates a key for each business intent and reuses the same key on retries, and the server uses it to identify duplicate requests. In multi-write workflows, this means the node needs to maintain an action list and carry an independent idempotency key for each item in the list.

Practical Implications for Developers

This reproducible test reminds us that LangGraph's "durable execution" does not equal "exactly-once execution." Its semantics are closer to "at least once," and only with an idempotent sink can it converge to the desired effect. When designing nodes with side effects, one must explicitly consider: whether the side effects are re-entrant, whether the granularity of dedup keys aligns with logical actions, and whether transaction boundaries cover all relevant writes.

The test code has actually been executed locally, and the author notes that this is an AI-assisted article. For readers who want to reproduce it, all that is needed is a clean Python environment, installation of the pinned dependencies above, saving the code as demo.py, and running it with the start and resume arguments respectively to observe the behavioral differences among the three modes.

LangGraph's checkpoint mechanism is the foundation of its durable execution: before and after each node execution, the framework serializes the thread state and writes it to the checkpointer (SQLite in this test). Upon recovery, the framework rebuilds state from the last checkpoint and reschedules unfinished nodes. The key point is that the timing of checkpoint writes and the commit timing of side effects inside the node are decoupled—the node may have completed its database write, but the checkpoint has not yet recorded the node as complete, so the node is rescheduled upon recovery. This decoupling is a trade-off between performance and reliability: requiring atomic commits of checkpoints and side effects would necessitate distributed transactions, which are costly. Therefore LangGraph chose "at-least-once" semantics and left idempotency responsibility to the developer. Understanding this timing relationship between checkpoints and side effects is a prerequisite for designing reliable workflows.