Home » Posts » Building Durable Agents

Building Durable Agents

A practical guide to prevent transient model provider errors, tool call failures, or good old network hiccups from breaking your agents.
July 8, 2026 · 17 min · Suggest Changes
An old watchmaker busy at his workbench.

I know you are going to point Claude Code to this with “make my agents durable, make no mistakes.”

Show QR code Hide QR code
QR code linking to https://navendu.me/posts/durable-agents/

An “agent” is just a loop. It does three things:

  1. Take a prompt, pass it on to the LLM, and get the response.
  2. If the response contains tool calls, execute the tools and return the results to the LLM.
  3. Break out of the loop if there are no more tool calls in the response, and return the response.
flowchart TB p([User prompt]) --> llm[Call the LLM] llm --> d{Tool calls?} d -->|Yes| t[Execute tool calls] t --> llm d -->|No| r([Return final response])

For example, consider this otherwise useless agent that suggests what to pack for your upcoming flight.

flowchart TB u([What should I pack
for my flight QF1?]) --> llm subgraph agentloop [The agent loop] direction TB llm{{LLM}} -->|"1 · get_flight_status('QF1')"| f["Flight API → London"] f --> llm llm -->|"2 · get_weather('London')"| w["Weather API → 6°C, clear"] w --> llm end llm -->|no tool calls| out([Pack a warm coat,
a scarf, and gloves.])
  1. Based on the prompt, the LLM decided to call get_flight_status("QF1") to retrieve the destination.
  2. The loop continues, and the tool result is returned to the LLM, after which it decides to call get_weather("London").
  3. The loop continues again, and the weather in London is returned to the LLM.
  4. The LLM now has all the data to decide on the appropriate apparel and produce the response without any tool calls.
  5. The loop breaks. The user sees the final response.

while the loop is simple, a lot can go wrong and make it unreliable:

  1. The APIs might have a transient issue (rate limits or 5xx errors) that could bring down the entire loop. If it fails during the Nth turn, all the N LLM calls are wasted.
  2. The upstream model provider can have an outage. You know the kind when Claude’s status page goes Christmas red in the middle of June.
  3. The process itself can die in long-running loops (which are fairly common now; my Claude Code sessions regularly run for more than 30 minutes) when the machine restarts (during a new deploy, due to an issue, etc.), and there’ll be no way to recover from the middle of the agent loop.

A better solution is to use a durable execution platform like Temporal, which handles all of these for you. When a tool call fails, it is retried. If your machine restarts, it resumes from the exact iteration without having to spend all that money again on the same tokens.

I’ve been using Temporal for the past year to build all my agents. While Temporal has been around for a while as a building block for durable systems (e.g., payment flows), it naturally fits the kinds of problems we face when building agents. Temporal is also free and open source, and it has SDKs in all major programming languages.

My goal is to share how I’ve been using it to make my agent loops durable and, as a bonus, observable, using the example apparel-suggestion agent from before. I would suggest everyone first go through the quickstart guide of the programming language of their choice before proceeding, as I don’t want to parrot the docs here.

Starting the Temporal Development Server

I have the Temporal CLI installed on my local machine, which helps me spin up a local Temporal server for development. It also helps me connect to and monitor my production Temporal instances. To install the CLI:

brew install temporal

Then start the dev server:

temporal server start-dev

The Temporal server runs on port 7233, and a web UI will be available at http://localhost:8233.

TIP

You can also run Temporal inside Docker if you don’t want to install the CLI.

Unreliable Agent

Let’s start with the unreliable agent from our previous example. I will use OpenAI’s model and SDK, but you could just as well use any other platform.

IMPORTANT

Set the OPENAI_API_KEY environment variable before running the script.

import json

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment
MODEL = "gpt-5.4-nano-2026-03-17"

def get_flight_status(flight):
    return json.dumps({"flight": flight, "destination": "London (LHR)", "status": "on time"})

def get_weather(city):
    return json.dumps({"city": city, "temperature_c": 6, "conditions": "clear"})

TOOL_IMPLS = {"get_flight_status": get_flight_status, "get_weather": get_weather}

SYSTEM_PROMPT = (
    "You are a travel assistant. Given a flight number, use get_flight_status to find "
    "the destination, then get_weather for that city, then tell the user what to pack "
    "in one short sentence."
)

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_flight_status",
            "description": "Look up the destination city and status of a flight by its number.",
            "parameters": {
                "type": "object",
                "properties": {"flight": {"type": "string", "description": "Flight number, e.g. QF1"}},
                "required": ["flight"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string", "description": "City name, e.g. London"}},
                "required": ["city"],
            },
        },
    },
]

def call_llm(messages):
    completion = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        tools=TOOLS,
        parallel_tool_calls=False,
    )
    return completion.choices[0].message

def run_tool(tool_call):
    name = tool_call.function.name
    args = json.loads(tool_call.function.arguments)
    return TOOL_IMPLS[name](**args)

def run_agent(goal):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": goal},
    ]
    while True:
        message = call_llm(messages)
        messages.append(message)

        if not message.tool_calls:
            return message.content

        result = run_tool(message.tool_calls[0])
        messages.append(
            {"role": "tool", "tool_call_id": message.tool_calls[0].id, "content": result}
        )


if __name__ == "__main__":
    print(run_agent("What should I pack for my flight QF1?"))

Try running this, and you’ll see the final response. Now let’s see how Temporal makes it durable.

Durability with Temporal

The agent loop maps to a workflow in Temporal. It orchestrates a sequence of steps deterministically. The actual executions—the LLM call and the tool calls—happen inside a Temporal activity.

Let’s start by moving the LLM and tool call functions into activities.

 1import json
 2from dataclasses import dataclass
 3
 4from openai import OpenAI
 5from temporalio import activity
 6
 7client = OpenAI()  # reads OPENAI_API_KEY from the environment
 8MODEL = "gpt-5.4-nano-2026-03-17"
 9
10def get_flight_status(flight):
11    return json.dumps({"flight": flight, "destination": "London (LHR)", "status": "on time"})
12
13def get_weather(city):
14    return json.dumps({"city": city, "temperature_c": 6, "conditions": "clear"})
15
16TOOL_IMPLS = {"get_flight_status": get_flight_status, "get_weather": get_weather}
17
18SYSTEM_PROMPT = (
19    "You are a travel assistant. Given a flight number, use get_flight_status to find "
20    "the destination, then get_weather for that city, then tell the user what to pack "
21    "in one short sentence."
22)
23
24TOOLS = [
25    {
26        "type": "function",
27        "function": {
28            "name": "get_flight_status",
29            "description": "Look up the destination city and status of a flight by its number.",
30            "parameters": {
31                "type": "object",
32                "properties": {"flight": {"type": "string", "description": "Flight number, e.g. QF1"}},
33                "required": ["flight"],
34            },
35        },
36    },
37    {
38        "type": "function",
39        "function": {
40            "name": "get_weather",
41            "description": "Get the current weather for a city.",
42            "parameters": {
43                "type": "object",
44                "properties": {"city": {"type": "string", "description": "City name, e.g. London"}},
45                "required": ["city"],
46            },
47        },
48    },
49]
50
51@dataclass
52class LLMResponse:
53    content: str | None
54    message: dict
55    tool_calls: list[dict]
56    tool_call: dict | None
57
58@activity.defn
59def call_llm(messages: list[dict]) -> LLMResponse:
60    completion = client.chat.completions.create(
61        model=MODEL,
62        messages=messages,
63        tools=TOOLS,
64        parallel_tool_calls=False,
65    )
66    message = completion.choices[0].message
67    tool_calls = [call.model_dump() for call in message.tool_calls or []]
68    return LLMResponse(
69        content=message.content,
70        message=message.model_dump(exclude_none=True),
71        tool_calls=tool_calls,
72        tool_call=tool_calls[0] if tool_calls else None,
73    )
74
75@activity.defn
76def run_tool(tool_call: dict) -> str:
77    name = tool_call["function"]["name"]
78    args = json.loads(tool_call["function"]["arguments"])
79    return TOOL_IMPLS[name](**args)

Here’s what we changed:

  • 5: Import Temporal’s activity decorator.
  • 51-56: LLMResponse is a dataclass to serialize whatever an activity returns to be stored in Temporal (and used in replays).
  • 58 and 75: The @activity.defn decorator makes Temporal treat them as activities and remember their return values.
  • 67-73: call_llm makes the same API call as before but hands back the dataclass, and model_dump() flattens the SDK object into plain dicts.

Now we can orchestrate these activities through a workflow. i.e., instead of calling the LLM and tool functions directly, we run them as Temporal activities. Temporal stores each activity result in the workflow’s event history, and if the process dies mid-run, these results are used instead of rerunning.

 1from datetime import timedelta
 2
 3from temporalio import workflow
 4from temporalio.common import RetryPolicy
 5
 6with workflow.unsafe.imports_passed_through():
 7    from activities import call_llm, run_tool, SYSTEM_PROMPT
 8
 9@workflow.defn
10class AgentWorkflow:
11    @workflow.run
12    async def run(self, goal: str) -> str:
13        messages = [
14            {"role": "system", "content": SYSTEM_PROMPT},
15            {"role": "user", "content": goal},
16        ]
17        while True:
18            response = await workflow.execute_activity(
19                call_llm,
20                args=[messages],
21                start_to_close_timeout=timedelta(seconds=60),
22                retry_policy=RetryPolicy(maximum_attempts=5),
23            )
24            messages.append(response.message)
25
26            if not response.tool_calls:
27                return response.content
28
29            result = await workflow.execute_activity(
30                run_tool,
31                args=[response.tool_call],
32                start_to_close_timeout=timedelta(minutes=5),
33            )
34            messages.append(
35                {"role": "tool", "tool_call_id": response.tool_call["id"], "content": result}
36            )

The only real change is that we now wrap the loop in a workflow, and the function calls become activity executions:

  • 18-23: The model call runs as an activity with a 60-second timeout and a retry policy that runs the activity up to five times with exponential backoff. Temporal lets you set up better retry policies for retryable and non-retryable errors, which I also use frequently in my workflows.
  • 29-33: Similarly, tools are executed in their own activity with a longer five-minute timeout (and default policies). The result of the tool run is in the event history, which can be replayed instead of running it again, which is quite useful for non-idempotent operations.

With these slight modifications leveraging Temporal, you get retries with backoff and recovery out of the box, without having to reinvent a worse wheel yourself.

Server, Workers, and Client

Temporal has three components:

  1. Server: The dev server we started at the beginning. It stores the event history and distributes work to workers via a task queue.
  2. Workers: This is what runs your code. A worker picks up work from the server, executes the workflow and its activities, and reports the results. Workers are stateless, so you can spin up multiple workers to scale horizontally.
  3. Client: This starts a workflow. It just tells the server to start one and does not execute any of your code.

The client hands a workflow to the server, and any worker polling the same task queue (agent, in the example below) picks it up. For example, the client might sit inside your backend and start a workflow on a REST call, while one or more workers poll the server and execute the agent loop.

flowchart TB req([REST API call]) --> client subgraph backend [Your backend] client[Temporal client] end server[Temporal server] queue[["agent task queue"]] history[(Event history)] server -.- queue server -.- history subgraph worker [Worker process] loop["Agent loop:
LLM + tool activities"] end client -->|"1 · start workflow"| server loop -->|"2 · poll task queue"| server server -->|"3 · deliver task"| loop loop -->|"4 · report result"| server server -.->|"5 · final result"| client

The dev server is already running. Let’s create the worker, which will register the workflow and activities, and then the client, which starts the workflow.

import asyncio
from concurrent.futures import ThreadPoolExecutor

from temporalio.client import Client
from temporalio.worker import Worker

from activities import call_llm, run_tool
from workflow import AgentWorkflow

async def main():
    client = await Client.connect("localhost:7233")
    worker = Worker(
        client,
        task_queue="agent",
        workflows=[AgentWorkflow],
        activities=[call_llm, run_tool],
        activity_executor=ThreadPoolExecutor(max_workers=10),
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

Then start a run from a client:

import asyncio

from temporalio.client import Client

from workflow import AgentWorkflow

async def main():
    client = await Client.connect("localhost:7233")
    result = await client.execute_workflow(
        AgentWorkflow.run,
        "What should I pack for my flight QF1?",
        id="pack-for-qf1",
        task_queue="agent",
    )
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

With the dev server still running, start the worker and kick off a run:

python worker.py

python starter.py

You will see a similar answer from the agent as before. You can open the Temporal web UI at http://localhost:8233, and you will find this workflow run. Each LLM call and tool call shows up in the event history, and you will be able to click through to see exactly what happened in each step. This inherent observability is quite useful when debugging long-running agent workflows.

The agent loop in the Temporal Web UI
The agent loop in the Temporal Web UI

Each call_llm and run_tool is its own activity in the event history.

You can try building more complex, multi-step agents with more tools that could have ephemeral issues to see how Temporal helps with durability.

HIRING

I’m hiring software engineers and researchers to join our team at Eternis. Get in touch if you’d like to work with us.

For my team and me, all of our agent loops are, by default, in Temporal. We self-host the open source version of Temporal, and it seems to meet all our needs.

Temporal also translates well to multi-agent architectures. For example, Temporal has the concept of child workflows, which can correspond to child agents spawned by a main agent. Once you get used to Temporal, you will start to think about your agent architecture in terms of these familiar building blocks.

Webmentions • Last updated at 1:01 PM, 7th July 2026

Have you written a response to this? Send me a webmention by entering the URL.