OpenAI Batch is an easy discount to misread. The token line can shrink by half, but the bargain is for work that can sit in a queue for as long as 24 hours. Send a customer-facing response there and you have bought a cheaper way to miss your latency target. Send a nightly catalog cleanup there and the trade can be excellent.

Illustrative token spend index: synchronous processing at 100, Batch at 50

The useful question is not whether your app makes many requests. It is whether each result can arrive later, and whether your system can safely account for every result that does arrive. OpenAI describes Batch as asynchronous processing with a 50% discount against synchronous API pricing, a separate rate-limit pool, and a 24-hour completion window. That is a real price reduction, not a free speed-up.

When the 24-hour wait pays

Start by sorting work by deadline, not volume. If a person is waiting on the answer, keep the request on a synchronous path. Chat replies, interactive search, live moderation decisions, and an agent's next tool call all have a user or program waiting. A delayed answer can cost more than the token discount saves.

Batch is a better fit for work that already runs away from the request path: nightly document labeling, product metadata refreshes, offline evaluations, embedding a back catalog, or generating drafts for review. These jobs have a natural queue and can surface a completed artifact later. If a job has a fixed deadline, leave a margin. The published window is up to 24 hours, not a promise that every run will finish in minutes.

A quick screen before changing code:

  • Does anything downstream block until this exact result returns? If yes, keep it synchronous.
  • Can each record be processed independently and identified after completion? If no, add stable IDs or keep the current path.
  • Can you tolerate a result arriving late, failing, or needing a retry? If no, do not move it yet.
  • Is the endpoint and model supported by Batch? Check the current model and endpoint list before preparing files.

The docs currently allow up to 50,000 requests in one batch input file, capped at 200 MB. Those are ceilings, not a reason to create one enormous job. Smaller jobs are easier to inspect and rerun. Each JSONL line must contain one request and a unique custom_id; requests in one input file must target a single model.

For a rough cost check, take a workload that currently spends $100 on eligible input and output tokens. The documented Batch discount makes the token portion about $50, before retries, engineering time, storage, or other charges. If half of your token bill is only $10, this saves roughly $5. A migration that takes a day to build will not pay for itself quickly. If the same job burns $2,000 a month and is already asynchronous, saving about $1,000 is worth a controlled test. These are simple illustrations, not a quote: use the active model's rates and your actual input/output mix.

That distinction is where generic Batch guides often skip a step. Half off model tokens does not mean half off total system cost. You still own the queue worker, output parsing, validation, alerting, and retry logic. Count setup and failure rework in your automation break-even before calling the change a win.

The retry trap and break-even

Treat a batch as a collection of per-record jobs, not one all-or-nothing response. The official guide says a batch that expires cancels unfinished requests. Completed responses remain available in the output file, and errors, including expired requests, are written separately. That means a top-level expired status does not tell you to discard every result or resubmit everything.

Give each input row a stable custom_id that maps to your own database record. When the output arrives, validate the ID, response status, and expected schema before applying anything. Join results by that ID, never by line position. Then compare requested, succeeded, and failed counts. A request can fail even when its batch completed, so inspect per-item outcomes either way.

The safe recovery loop is deliberately boring: collect successful results once, mark them complete in your own store, inspect the error file, and create a new input file only for records that are safe to retry. Keep the same business record ID but add an attempt identifier to your job log. Make downstream writes idempotent so a rerun cannot create duplicate invoices, emails, or database rows. For actions with external side effects, send model output through review or validation rather than retrying the action itself.

Before moving a whole workload, run a small slice in parallel with the existing path. Compare output quality, token usage, completion delay, failed-item rate, and the time your team spends handling exceptions. Do not judge success by the batch status alone. Preserve the synchronous route as a fallback while you learn how the actual workload behaves.

Batch makes economic sense when the job is already asynchronous, quality checks are automated, and the monthly token savings exceed the cost of operating the extra path. It is a poor fit when the result gates an interactive step, failures require manual repair, or the saved dollars disappear into queue maintenance.

The API is not asking you to batch everything. It is giving you a different service tier for work that does not need to happen now. Start with one overnight job, account for every custom_id, and keep the live path live.

Sources