A lot of AI spending comes from treating every request like a live conversation. That is a bad default for work nobody reads until tomorrow morning.

OpenAI's Batch API and Anthropic's Message Batches API both cut usage prices by 50% for asynchronous jobs. The catch is the part most pricing summaries skip: you are trading the request-response loop for a result file that may arrive later, out of order, or with individual requests marked as failed. The discount is real. So is the scheduling problem.

Batch APIs trade latency for a 50% price cut

The jobs that qualify

The right question is not "Can this call be batched?" Nearly any repetitive call can. The question is whether a human or downstream system needs the answer before the next action can happen.

Batch the work when the input set is known, each request can be evaluated independently, and the result can wait. Good examples include nightly document classification, a weekly support-ticket digest, an offline eval suite, metadata extraction from a backlog, transcript labeling, and generating draft descriptions for a catalog. If somebody asks for the result at 9am and the job can run at 1am, paying the interactive price is hard to defend.

Keep the request synchronous when the result controls the next screen, blocks a deploy, powers a live chat, or feeds a tool call that must happen inside the same turn. Streaming is also a strong signal to stay synchronous. Anthropic explicitly excludes streaming from batch requests, while OpenAI's batch guide describes a file-based workflow rather than an immediate response.

A simple routing test works better than a vague "real time" label:

  • If a user is waiting, use a normal request.
  • If a queue can absorb the delay, consider a batch.
  • If a human reviews the output the next day, batch it.
  • If the model must call a tool and use that result before answering, keep the loop synchronous.

This is where batching fits beside model routing. First choose an appropriate model for the job. Then decide whether the job needs an immediate answer. A cheap model used synchronously can still cost more than a larger model used in a discounted overnight batch if the workload is large enough.

The scheduling rule also gives you a useful boundary for coding agents. An agent editing a repository while you watch needs a live loop. A nightly run that summarizes merged pull requests, labels stale issues, or generates a test-coverage report does not. The Hacker News discussion around OpenCode described the same problem in plainer terms: a $20-per-change habit can be a workflow problem, not only a pricing problem. People often pay for a string of tiny interactive turns when one well-scoped offline job would have been cheaper and easier to inspect.

The cost math and failure cases

The arithmetic is simple. Let S be the synchronous cost for a fixed set of input and output tokens. A batch using the documented 50% discount has an expected model bill of 0.5S, before any engineering or storage cost. If a nightly job normally costs $40, the model portion becomes about $20. At 30 runs per month, that is $600 instead of $1,200, assuming the request shape and model remain the same.

That last assumption matters. The discount does not rescue wasteful prompts. If every request includes a huge repeated instruction block, you still pay for it, just at half price. Measure the token mix first. Keep stable instructions stable, put per-item data in the request body, and test a small sample through the ordinary endpoint before submitting thousands of requests. Anthropic's documentation notes that prompt caching can work with message batches on a best-effort basis, but asynchronous processing can reduce cache-hit certainty. Treat caching as a possible extra, not as money already saved in your spreadsheet.

OpenAI's flow is a JSONL file, one request per line. Each line needs a unique custom_id. You upload the file, create a batch with a 24h completion window, poll its status, then download the output file. Anthropic uses a list of requests with a custom_id and a params object, then exposes results through a results URL. The surface syntax differs, but the application design is the same: create, wait, reconcile.

A minimal OpenAI input line looks like this:

{"custom_id":"ticket-0042","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Classify this ticket: ..."}],"max_tokens":120}}

Do not match output lines by position. Match them by custom_id. Anthropic says batch results may not match input order, and both APIs expose per-request status information. Your collector should create a map keyed by the ID, count every expected item, and put anything missing into a retry queue. This is the same discipline that prevents retry storms: retry failed items, not the entire batch.

There are four failure cases worth designing before the first production run.

First, validation errors. A malformed request can sit in your queue until the batch finishes, depending on the provider and the error. Run a representative sample synchronously and validate your JSONL before upload. Second, partial failure. Some requests can succeed while others error. Persist successful results before retrying anything. Third, expiration. Both providers document a 24-hour boundary, and unfinished work can expire. Mark expired IDs explicitly so the next run does not silently treat them as successful. Fourth, duplicate submission. Give every job a stable run ID and make your result writer idempotent. If a worker restarts after upload, it should look up the existing batch instead of creating a second one.

Privacy needs a place in the decision too. Batch processing stores inputs and outputs on the provider side while the job runs. Anthropic documents retention of batch data for up to 29 days after creation, with deletion available after processing. That may be acceptable for public transcripts or synthetic eval data and unacceptable for customer records. A 50% price cut is not a reason to bypass a data-retention policy.

The practical implementation is a small queue, not a new agent framework. Store the source record, provider, batch ID, custom ID, submission time, and processing status. Poll with a backoff. Fetch results in a streaming or chunked way when the file is large. Update each source record by ID. Keep the original prompt and model version next to the result so a rerun is explainable six weeks later.

Use the discount as a scheduling tool

The batch APIs are not a universal cheaper mode. They are a second lane for work that already has a natural waiting period. That distinction keeps the migration honest.

Start with one job that is already reviewed the next morning. Export a few hundred requests, run a synchronous sample to confirm the shape, then compare the batch bill with the old run. Track completion time, error rate, missing IDs, and the number of manual retries, not only the advertised discount. If the job cannot tolerate an occasional 24-hour delay or a provider-side retention window, leave it on the normal endpoint.

For everything else, the 50% number is a useful floor for the conversation. The real saving comes when you change the workflow so the model does not have to pretend every piece of work is urgent.

Sources