A payment tool returns a timeout. The agent assumes nothing happened and tries again. The first request actually reached the processor, so the customer gets charged twice. The model did not hallucinate the second charge. The system gave it no durable way to know that the first call had already taken effect.

That distinction matters. Prompting an agent to "only charge once" is not a safety mechanism when the network can lose the response after the server has done the work. The fix belongs at the tool boundary: every side-effecting operation needs a stable identity, a record of its result, and a retry policy that understands the cost of another attempt.

Idempotent actions should not create new effects on repeated calls

The retry boundary

Most agent runtimes behave like at-least-once systems around external work. Temporal states this plainly for Activities: a worker can complete an operation and crash before reporting the result, which causes the Activity to run again. The same shape appears in a home-grown queue, a Python loop around an LLM, or a tool runner that retries after a 504.

There are two different failures hiding inside the word "timeout":

  • The request failed before the external service began. Retrying is normally safe.
  • The service completed the side effect, but the response was lost. Retrying can create a duplicate.

The caller cannot reliably distinguish those cases from a socket exception. That is why the downstream operation must do the deduplication. A read-only search can usually be retried freely. A payment, email, ticket creation, inventory reservation, or database insert needs an operation key.

Stripe's API gives the clean version of this contract. A client sends an idempotency key with a POST request. Stripe stores the first result and returns that same result when the key is reused, rather than creating a second object. Its documentation says keys can be removed after they are at least 24 hours old, and a reused key with different parameters is rejected. The key is not a prompt instruction. It is an external fact that survives model uncertainty.

This also changes how you think about retry budgets. A retry is not free just because the model output is cheap. It may repeat a paid API call, consume a rate limit, or trigger another real-world action. Temporal's fixed retry example counts the initial execution in the limit: maximum_attempts=3 means one initial attempt plus two retries. That is a much more useful number than "retry a few times." The article's practical default is three attempts, with the phrase "3 attempts" reserved for the configured limit in logs and dashboards.

A safe implementation recipe

Start with a key generated before the tool makes its first external call. The key should identify the workflow execution and the logical step, not the model's latest text. Temporal's Python guidance uses the Workflow Run ID and Activity ID for this reason. Those values remain stable when the Activity is retried.

A minimal tool wrapper looks like this:

async def charge_payment(order_id: str, amount_cents: int):
    run = current_workflow_run_id()
    step = current_activity_id()
    key = f"{run}:{step}:charge"

    cached = await idempotency_store.get(key)
    if cached is not None:
        return cached

    result = await payment_api.charge(
        order_id=order_id,
        amount_cents=amount_cents,
        idempotency_key=key,
    )
    await idempotency_store.put(key, result, ttl_hours=24)
    return result

The exact framework does not matter as much as the contract. Generate the key outside the model. Pass it through the tool interface. Forward it to the downstream API when that API supports keys. If the service does not support them, create a database table with a unique constraint on the key and store the completed result before returning it to the caller.

The table needs more than a boolean such as done. Store the request fingerprint, status, response body, and timestamps. A key that is accidentally reused with a different amount should fail loudly, not return the result for the old amount. A request that is still running needs an explicit state so a second worker can wait, take ownership after a lease expires, or report an ambiguous outcome to a human.

A practical record can be as small as:

CREATE TABLE tool_operations (
  operation_key TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  status TEXT NOT NULL,
  response_json TEXT,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL
);

Do not put a fresh random UUID inside the retry handler. That creates a new operation every time the handler runs, which defeats the entire point. The random value belongs to a logical workflow step created once. On a retry, reconstruct the same key from durable identifiers.

Next, separate tool errors into classes. Retry a short network interruption or a provider rate limit with backoff. Stop immediately for invalid input, a rejected payment, a missing permission, or a request that violates a business rule. Temporal's Python documentation recommends marking permanent application errors as non-retryable instead of waiting for a timeout and burning more attempts.

Finally, log the key with every attempt. The log should answer four questions without reconstructing the entire conversation: which workflow step made the call, which key it used, whether the downstream service accepted it, and whether the response came from a fresh execution or a deduplication record. Never log the payment credential or other sensitive payload just to make this trace useful.

Where retries must stop

A stable key solves duplicate execution of one tool. It does not make a multi-step workflow atomic. Consider an order flow with inventory reservation, payment, and confirmation email. Payment may succeed while the email fails. Replaying the entire plan is wrong even if payment is idempotent, because the workflow still needs to decide whether to resend the email, release inventory, refund the payment, or wait for a human.

This is where a saga is more honest than a pretend transaction. Record each completed step and define a compensating operation for steps that can be reversed. If payment succeeds and the order must be abandoned, issue one refund using its own stable key. If inventory is reserved, release that reservation with a key tied to the same workflow step. Compensation must be idempotent too. A retrying refund tool that can refund twice is not a recovery mechanism.

Use a retry matrix before turning on automatic retries:

Tool type Default policy Required protection
Search or fetch Retry transient errors Request timeout and rate limit backoff
Database read Retry transient errors Consistent read policy
Record creation Retry only with a key Unique operation key and stored result
Payment or refund Retry with provider key Request fingerprint and reconciliation
Email or notification Retry with message key Provider deduplication or outbox
Irreversible action Stop or ask for approval Explicit human checkpoint

For paid calls, cap attempts even when the operation is technically safe to repeat. Three attempts is a reasonable starting probe, not a universal answer. If the first call is ambiguous and the provider has no idempotency support, one attempt plus a reconciliation lookup is safer than blindly sending a second request. Temporal's documentation explicitly notes that setting maximum_attempts=1 disables retries for operations where another execution could create a duplicate side effect.

The decision rule is simple. Read tools can often retry. Write tools need a stable key and a durable result. Multi-step writes need both idempotency and compensation. Irreversible actions need a human or a downstream service that can prove exactly what happened.

This is the part agent demos usually skip. They show the model choosing the right tool, then treat the network as a detail. Production systems do the opposite. They assume the model can choose badly, the worker can die at the worst moment, and the response can disappear after the side effect. A key generated before execution will not make the model smarter. It makes failure boring, which is the point.

Sources