Agent orchestration and durable execution

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Agent orchestration · also covers Workflow orchestration · Step 19 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

Orchestration decides which agent or step runs next and what context it receives; durable execution makes sure a long run survives crashes, deploys and rate limits without losing or repeating work. Agent runs are long, expensive and stateful, so a failure halfway through cannot simply restart from the top. The lesson covers multi-agent coordination patterns, then the workflow engines and checkpointing frameworks that give those patterns reliable state.

Read more

How it is done in practice

Teams usually pick the durability layer from what the customer already runs: a workflow engine where one exists, a checkpointed graph framework backed by Postgres otherwise. The agent loop is written so every model and tool call is a recorded step with a timeout, a retry policy and an idempotency key, and human approval is a durable wait instead of a blocking request. Before go-live the team kills workers mid-run, replays histories against new code, and uses gradual rollouts so agents already in flight finish on the version they started on.

Technologies

TemporalLangGraphOpenAI Agents SDKPostgresAWS Step FunctionsClaude Agent SDK

Diagram

flowchart TD
  C[Client starts run] --> W[Workflow: agent loop]
  W --> A1[Activity: model call]
  A1 --> H[(Event history)]
  W --> A2[Activity: tool call with idempotency key]
  A2 --> H
  W --> P[Durable wait for human approval]
  P --> H
  X[Worker crash] -.-> R[New worker replays history]
  H --> R
  R --> W
  W --> O[Orchestrator spawns subagents]
  O --> S1[Subagent A own context]
  O --> S2[Subagent B own context]
  S1 --> W
  S2 --> W
flowchart TD
  C[Client starts run] --> W[Workflow: agent loop]
  W --> A1[Activity: model call]
  A1 --> H[(Event history)]
  W --> A2[Activity: tool call with idempotency key]
  A2 --> H
  W --> P[Durable wait for human approval]
  P --> H
  X[Worker crash] -.-> R[New worker replays history]
  H --> R
  R --> W
  W --> O[Orchestrator spawns subagents]
  O --> S1[Subagent A own context]
  O --> S2[Subagent B own context]
  S1 --> W
  S2 --> W

Code

python
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

@workflow.defn
class TriageWorkflow:
    @workflow.run
    async def run(self, ticket_id: str) -> str:
        opts = dict(
            start_to_close_timeout=timedelta(minutes=2),
            retry_policy=RetryPolicy(maximum_attempts=4),
        )
        messages = [ticket_id]
        for step in range(8):
            reply = await workflow.execute_activity("call_model", messages, **opts)
            if not reply.get("tool"):
                return reply["text"]
            key = f"{workflow.info().workflow_id}-{step}"
            messages.append(await workflow.execute_activity("run_tool", [reply, key], **opts))
        return "escalated"

Subtopics

Multi-agent orchestration patterns

Model-driven versus code-driven

There are two ways to coordinate agents. In model-driven orchestration, a lead agent plans, decides which specialists to invoke and synthesizes results; this adapts to open-ended tasks. In code-driven orchestration, the program fixes the flow: chain agents so one output feeds the next, branch on a structured classification, run independent agents in parallel, or loop a worker with an evaluator. Code-driven flows are more predictable in speed, cost and behavior. Most production systems mix them, with code owning the outer flow and a model deciding inside bounded steps.

Orchestrator and subagents

In the orchestrator-worker design used for research systems, a lead agent decomposes a query and spawns subagents that work in parallel, each with its own context window, returning condensed findings. This buys breadth and context isolation at a price: multi-agent runs consume several times more tokens than a single agent, so they fit high-value, parallelizable tasks. Delegation quality decides the outcome. Each subagent needs an objective, an output format, guidance on tools and sources, and clear task boundaries, otherwise workers duplicate effort or leave gaps.

Workflow engines and replay

How durable execution works

In Temporal, a workflow is code whose progress is recorded as an event history. Anything with side effects, such as a model call, a tool call or a database write, runs as an activity whose result is stored. If a worker dies, another worker replays the workflow code against the history, reusing recorded activity results until it reaches the point of failure, then continues. This requires workflow code to be deterministic: no direct network calls, random values or wall-clock reads inside the workflow function. Timers and waits for external signals are durable too, so a run can pause for days.

Mapping agents onto it

For an agent, the loop is the workflow and each model call and tool call is an activity with its own timeout and retry policy. The nondeterministic model output is safe because it is recorded once and replayed from history. Long conversations make histories large, so engines provide a way to continue as a fresh run carrying forward summarized state. Changing workflow code while runs are in flight needs versioning, because replay of an old history against new code must take the same branches.

Idempotency, retries and budget control

Beyond the basics

Engines give at-least-once activity execution, so every side-effecting tool needs an idempotency key derived from the run and step, passed through to the downstream system. Agent-specific additions: classify errors so a malformed tool argument is returned to the model as an observation while a rate limit is retried with backoff by the engine; set heartbeats on long activities so stalls are detected; and treat token spend as a resource checked before each model activity, with the run failing closed when the cap is reached.

Graph state, checkpoints and human approval

Checkpointed graphs

LangGraph models an agent system as a graph of nodes that read and update a shared typed state, with edges, including conditional ones, deciding what runs next. A checkpointer persists state after each step under a thread identifier, backed by a store such as Postgres. That gives resumption after failure, inspection of any earlier state, and the ability to fork a run from a past checkpoint for debugging. A node can interrupt the run to wait for a person; the graph resumes later with the human's input merged into state, which is how approval steps and edits are implemented.

Sources

Related lessons

Practice this
Building Model Context Protocol (MCP) serversEvaluating LLM and agent systems