Part 6 — Handle Failures with Retries, Progress Events, and an Idempotent Outbox

Part 6 of 8 · Basic Python is enough · Tested with Python 3.12.4, LangGraph 1.2.16 and SQLite checkpointer 3.1.2.

Recover from a simulated transient lookup failure, show the nodes that complete, and prevent a repeated queue operation from creating duplicate rows.

Project files: lessons/06_retries/main.py. Run commands from the extracted repository root. The matching article ZIP includes this article, its diagrams, and the complete tested repository.

A workflow needs a failure story

We can pause and resume a review. Now ask what should happen if an operation fails temporarily, or if a queue write succeeds just before the process stops.

This article adds three concrete behaviors: limited retries for a simulated connection failure, visible node progress, and a local queue that tolerates an identical repeated write. These solve different problems. A retry does not validate an answer, and a saved checkpoint does not make every external action happen exactly once.

The runnable demonstration assembles the final workflow from src/support_desk/workflow.py. The functions still perform the responsibilities introduced in earlier lessons: classify, look up evidence, draft, check, and review. Read that file alongside the smaller demonstration.

python lessons/06_retries/main.py

The demonstration uses a temporary directory and an in-memory checkpointer. Its queue disappears when the demonstration finishes. Use the final CLI in article 8 for a queue and checkpoint that persist across commands.

Retry only the error you intend to retry

The example lookup function deliberately raises ConnectionError on its first call and delegates to the normal lookup on its second call:

def make_flaky_lookup():
    calls = 0

    def flaky_lookup(text, category):
        nonlocal calls
        calls += 1
        if calls == 1:
            raise ConnectionError("Simulated temporary lookup failure")
        return retrieve(text, category)

    return flaky_lookup

This is a test fixture, not a claim that reading our tiny JSON file normally requires a network connection. It makes a transient failure repeatable without depending on a flaky external service.

The graph registers this policy on the lookup node:

builder.add_node("lookup", lookup, retry_policy=RetryPolicy(
    max_attempts=3,
    initial_interval=0.01,
    jitter=False,
    retry_on=ConnectionError,
))

Generate and explain the graph built by this code

The diagram below is generated from this lesson's compiled get_graph() nodes and edges. Node positions are chosen for readability; the connections come from the Python graph, rather than being copied from a generic illustration.

The following excerpt is inside the complete builder function. Node functions are defined earlier in the linked source file; run the lesson or final package rather than pasting this excerpt by itself.

builder = StateGraph(TicketState)
builder.add_node("classify", classify)
builder.add_node("lookup", lookup, retry_policy=RetryPolicy(
    max_attempts=3, initial_interval=0.01, jitter=False, retry_on=ConnectionError))
builder.add_node("draft", draft)
builder.add_node("check", check)
builder.add_node("revise", revise)
builder.add_node("approval", approval)
builder.add_node("queue", queue)
builder.add_node("escalate", escalate)
builder.add_edge(START, "classify")
builder.add_conditional_edges("classify", after_classify, ["lookup", "escalate"])
builder.add_conditional_edges("lookup", after_lookup, ["draft", "escalate"])
builder.add_edge("draft", "check")
builder.add_conditional_edges("check", after_check, ["approval", "revise", "escalate"])
builder.add_edge("revise", "draft")
builder.add_conditional_edges("approval", after_approval, ["queue", END])
builder.add_edge("queue", END)
builder.add_edge("escalate", END)
return builder.compile(checkpointer=checkpointer)

After installing the project, run this command from its root:

python tools/render_graphs.py --part 6

It writes assets/generated/part-06.svg (the image), .mmd (Mermaid source), and .json (the captured nodes and edges). Rendering is local and does not call a model or a remote image service. If you add nodes, update the layout positions in the renderer as well.

Open the full-size graph.

Read the arrows with the code

• lookup: RetryPolicy handles selected exceptions inside node execution. Its internal retry does not create a separate graph edge.
• check → revise → draft: This visible loop is the separate bounded draft-revision rule, not the lookup retry policy.
• approval → queue or END: Approval leads to the local queue; rejection ends without queueing. The lesson script simulates the reviewer decision, while the final CLI uses an explicit command.
• escalate → END: Unsupported requests and exhausted draft checks finish with needs_human.

Solid lines are fixed transitions. Orange dashed lines mark possible conditional destinations; the routing function chooses the destination for a particular run. A node can contain several Python operations that are not separate graph nodes.

The short interval keeps the teaching demonstration quick. Real services need retry settings appropriate to their behavior. We do not include ValueError here: malformed input generally does not become valid because the program waits and repeats it. The graph API documents retry policies attached to nodes. Node retries

In the demonstration, lookup_calls should be 2. If every attempt raises the selected exception, the policy stops after three attempts and propagates the error. A test checks that no outbox row appears in this failure case.

Separate operation retries from draft revisions

The first two mechanisms must not silently approve a reply. Approval remains a separate step regardless of how often the earlier code ran.

Print useful progress without inventing model reasoning

The demonstration consumes graph updates:

for update in graph.stream(state, config, stream_mode="updates"):
    print("STEP:", ", ".join(update))

It prints node-update names. A successful draft path includes classify, lookup, draft, and check, then an interrupt marker when review pauses the run. The failed first lookup does not produce a successful state update; the separate call counter demonstrates that the retry occurred.

This is not token streaming. The demo drafter has no tokens to stream, and the local-model adapter later requests a complete response. It is also not a display of hidden reasoning. The terminal shows the graph events the application makes observable.

Put the side effect after approval

The final graph connects approval to the queue only when approved is true. The queue writes a local SQLite row. There is deliberately no SMTP client or external delivery tool in this repository.

The outbox uses the ticket ID as its primary key. The important operation is:

INSERT OR IGNORE INTO outbox VALUES (?, ?, ?)

The values are passed as parameters. After the insert, the implementation reads the row and compares its reply and citations with the requested values. An existing identical row is acceptable. An existing different payload raises an error, rather than pretending a conflicting write succeeded.

This is idempotency for a particular local operation: repeating put with the same ticket and payload has the same stored result. It is not a claim that all graph steps execute once.

Why the queue needs its own protection

The checkpoint database and outbox database are separate. A queue transaction could commit before a later checkpoint records node completion. If that node is executed again, the queue must handle a repeated write itself. Conversely, a checkpoint cannot recall an email that an unrelated delivery service already sent.

Our demonstration calls outbox.put again with the exact row just queued. The printed outbox_rows_after_repeat should still be 1. A test separately attempts a different reply under the same ID and confirms that the original row remains unchanged.

SQLite connection handling matters here too. A connection's transaction context commits or rolls back, but does not itself guarantee the connection is closed. The implementation uses contextlib.closing as well as the transaction context. This avoids a Windows file-lock error when the temporary demonstration directory is cleaned up.

Understand the simulated approval

The lesson script supplies Command(resume={"approve": True}) to demonstrate the post-review path automatically. Its source comments label this as a simulated reviewer decision. The final CLI instead requires a separate explicit approve command. Running a demonstration is not a design recommendation to auto-approve real customer replies.

For practice, change the first-call failure to an always-failing function and predict the number of lookup calls. Then restore it and inspect the queue behavior after rejection in the tests. In article 7, we will replace only drafting with a local model while leaving these routing and approval boundaries in place.

This is an educational project using fictional workshop notes and synthetic tickets. It does not access the real CodeTips Academy website or send email. Repository-hosted links can be added after the owner publishes the code; current source links are relative to this repository.