A lot of AI work is pretending to be urgent. A nightly document summary is not a chatbot response. A queue of product descriptions is not a voice assistant. A weekly classification job does not need an answer before the user blinks.

That distinction can cut the model bill in half. OpenAI's Batch API advertises 50% lower costs and a 24-hour completion window. Anthropic's Message Batches API charges 50% of standard prices, and its documentation says most batches finish in less than 1 hour. The catch is that you are trading response time for a small job system: input files, stable IDs, polling, result retrieval, expiry handling, and reconciliation.

Provider-documented batch pricing is 50% of standard pricing

The savings are real. So is the engineering work people leave out of the headline.

The batch decision

Use a normal synchronous request when a person is waiting, when the next step depends on the answer immediately, or when a tool call can change external state. A support reply, checkout decision, browser action, and interactive coding assistant belong here. The cost of waiting, retrying a conversation, or confusing a user is greater than the token discount.

Use a batch endpoint when the work can sit in a queue without changing the product experience. Good candidates include overnight report generation, bulk tagging, offline evaluation, transcript classification, embedding refreshes, content moderation backfills, and migrations where you already have a file of inputs.

The useful test is simple: if a job misses its preferred completion time, does a human have to stop working? If the answer is yes, keep it synchronous. If the answer is no, batch it and make the delay visible in your own UI.

The price math is deliberately boring. A workload that costs $100 through standard calls costs about $50 at a documented 50% batch rate, before storage, orchestration, and engineering time. A $240 monthly workload becomes $120 in model spend. That is enough to matter, but only if the work was already asynchronous. Do not turn a live request into a 24-hour queue just to make a dashboard look cheaper.

There is also a throughput reason to batch. OpenAI documents a separate rate-limit pool for Batch API requests, so bulk processing does not consume tokens from standard per-model limits. Anthropic provides separate batch mechanics and supports large request lists, but its docs call out parameters that do not fit asynchronous processing, including streaming and stateful thread fields. These are not interchangeable drop-in modes.

A practical choice looks like this:

  • Need an answer in seconds or need to stream tokens: use the standard API.
  • Can wait until later and already use OpenAI: use Batch API with JSONL.
  • Can wait and already use Claude: use Message Batches with one stable custom ID per request.
  • Need the cheapest provider for a mixed workload: compare the actual model prices, then apply the batch discount. The percentage is similar, but the base rate and supported capabilities differ.

The workflow that survives failure

The fragile version is one script that submits a file, sleeps for a while, and assumes every output arrived. That script works in a demo and loses data in production.

Start with a durable job table or a JSONL manifest. Give each input a stable operation ID that comes from your own record, not from the provider's batch ID. For example, invoice-2026-08-16-00421 should identify the same invoice through submission, polling, output parsing, retries, and final database updates. Provider batch IDs identify a container. Your custom ID identifies the work.

OpenAI's documented flow is explicit. Build a .jsonl file, put one request on each line, include a unique custom_id, upload the file for the batch purpose, create the batch with a completion window of 24h, poll its status, then download the output or error file. Each input file targets one model. That constraint belongs in the manifest builder, not in a late API error.

A minimal input line has the shape below:

{"custom_id":"invoice-2026-08-16-00421","method":"POST","url":"/v1/responses","body":{"model":"your-model","input":"Extract the total and currency from this invoice..."}}

Keep the source document ID in your own database as well. The custom_id is for joining results, not a replacement for data ownership. When the output file arrives, parse every line into one of four states: succeeded, errored, canceled, or expired. Write those states before handing successful content to the next step. A result parser that only stores successful text will make a failed batch look smaller than it really was.

Anthropic's shape is different. A Message Batch contains a list of requests, each with a custom_id and message parameters. You create it, poll processing_status, and retrieve results from the batch when it reaches ended. Anthropic's documentation says a failed request does not stop the other requests, which is useful, but it also means partial completion is normal. Your reconciliation loop must be normal too.

Treat the batch as a state machine:

  1. planned: input exists in your database and has not been submitted.
  2. submitted: provider accepted the batch and returned a provider ID.
  3. processing: poll status without creating a second batch.
  4. completed: retrieve results and join by your stable custom ID.
  5. partial: retain successes and schedule only errored or expired work.
  6. reconciled: every input has a terminal state and an audit record.

Do not retry the whole batch because one request failed. That can duplicate successful work and double your bill. Store a submission fingerprint, reject a second submission for the same unfinished fingerprint, and retry only the rows whose provider result says they need another attempt. The same idempotent retry keys that protect agent tools also protect bulk AI jobs. NestFrontier's guide to idempotent retry keys covers the general pattern.

The discount has a boundary

Batch APIs are a cost lever, not a quality upgrade. They do not make a weak extraction prompt reliable. Run a small synchronous sample first, validate the schema, and record the fields you expect before putting thousands of requests into an asynchronous queue.

A 50% discount also does not mean 50% lower total cost. You still pay for the code that prepares inputs, stores source files, polls status, retrieves results, handles partial failures, and reviews ambiguous outputs. For a small job that runs once, the human setup time can exceed the model savings. For a recurring nightly job with thousands of independent records, the arithmetic changes quickly.

Privacy changes too. Anthropic documents that batch data and results are stored for up to 29 days after batch creation. OpenAI's workflow requires uploading the input file before processing. That may be fine for public text and internal evaluation data. It deserves a separate approval for invoices, support conversations, health data, or anything under a retention policy. Delete provider-side files when the job no longer needs them, and do not copy sensitive source text into logs just to make debugging convenient.

The final rule is less exciting than the discount: batch the waiting, repeatable work; keep the interactive path boring and synchronous. Put the boundary in code, attach a stable ID to every request, and make partial completion visible. The 50% number is useful only after those three things are true.

Sources