The first question in an AI infrastructure meeting should not be “Which open model can we run?” It should be “How many tokens will this machine actually serve when nobody is watching?” A GPU that sits idle is a subscription with fans.

The current API prices make the default choice surprisingly hard to beat. OpenAI lists GPT-5.6 Luna at $0.20 per million input tokens and $1.20 per million output tokens. That is $4.40 for a workload with 10 million input tokens and 2 million output tokens in a month. If the same workload can wait, Batch processing cuts the input and output rates by 50%, taking the bill to $2.20. That is before counting engineering time, patching, monitoring, storage, failover, and the electricity bill for a self-hosted server.

Input token price after caching and batch optimization

That does not make self-hosting pointless. It changes the order of operations. First make the API path cheap and measurable. Then compare its monthly bill with the fixed cost of a server that will run at a known utilization. If the comparison starts with “I can download the weights,” it has already skipped the expensive part.

The cost model

Use two separate calculations. One describes the API path. The other describes the machine. Do not mix a token price with a GPU rental quote and call the result total cost of ownership.

For the API, record input tokens, output tokens, cache-hit input tokens, batch-eligible tokens, and retries for one representative month. The basic formula is:

api_monthly = (input_millions × input_rate)
            + (output_millions × output_rate)
            + retry_costs
            + tool_or_search_fees

For the Luna example, 10M input plus 2M output is (10 × 0.20) + (2 × 1.20), or $4.40. If all requests are asynchronous, the documented 50% Batch discount makes it $2.20. Real agents rarely get that clean result because interactive turns, tool calls, and retries do not all qualify. Keep those categories separate in your spreadsheet.

Caching can move the answer again. Anthropic’s prompt-caching documentation lists Claude Sonnet 5 cache hits at $0.20 per million tokens (cache hit), against a $2.00 base input price through August 31, 2026. The default cache lifetime is 5 minutes. That means a stable system prompt can be cheap when requests arrive close together, then become a full-price input when the session goes quiet. A cache hit is a traffic pattern, not a permanent discount.

This is why the first comparison should be “optimized API versus self-hosted,” not “API versus self-hosted.” Split the workload into three lanes:

  • Interactive: keep it on the lowest-latency API or a warm local server. Do not force a 24-hour batch queue onto a user-facing request.
  • Repeated context: mark stable system instructions, tool definitions, and reference material for caching. Track cache writes and reads rather than assuming every request hits.
  • Deferred work: move evaluations, document extraction, nightly reports, and bulk classification to Batch when the provider supports it.

The self-hosted side needs a fixed monthly floor:

self_host_monthly = gpu_or_instance_cost
                  + storage
                  + electricity
                  + monitoring
                  + maintenance_time
                  + backup_capacity

If a rented GPU costs $400 for the month and the model is available only 30% of the time, the relevant capacity is not “one GPU.” It is $400 divided by the useful served workload. If you own the hardware, amortize the purchase over its useful life and still include power and the person who will notice when the server has been unhealthy for six hours.

The break-even equation is simple:

break_even_output_millions = fixed_monthly_cost / api_output_rate

That simplified equation is useful only when comparing the same task and output quality. At $1.20 per million output tokens, a $400 monthly serving floor needs about 333 million output tokens just to match the raw Luna output price. At $12 per million output tokens, the same floor needs about 33 million. The model choice changes the answer by an order of magnitude. So does whether the API bill includes a cheap mini model, caching, or batch processing.

There is another trap: input and output are not interchangeable. Agent workloads often read a large repository, call tools, and produce a small patch. A cost model that only counts generated text can make self-hosting look cheaper than it is. Log both sides, plus the tokens created by tool results and retrieval.

For a serving implementation, vLLM is a practical baseline because its documentation exposes 2 API endpoint families plus health checks and Prometheus metrics. That gives a reader a path from “local model” to an observable service. It does not give free capacity. A compatible endpoint still needs a model that fits, a queue policy, authentication, upgrades, and enough headroom for concurrent requests.

The decision matrix

Choose the API when volume is low or spiky, the model changes often, or the team cannot dedicate time to serving operations. An API also wins when the task needs a frontier model that is not available as open weights. At low utilization, paying for idle GPU memory is worse than paying for tokens.

Choose an optimized API path when the work is mostly interactive but contains repeated instructions or deferred jobs. This is the neglected middle ground. Prompt caching and Batch processing can remove enough waste that a GPU purchase becomes a solution to a problem you no longer have. Read our prompt-caching cost model for the cache-safe workflow, then measure your own hit rate.

Choose self-hosting when the workload is steady, the model is stable, privacy or network control is a hard requirement, and the server can stay busy. “We might need it later” is not utilization. Put a minimum in writing. For example, require a forecasted monthly API bill at least twice the fixed serving cost before buying hardware, because the self-hosted estimate will be wrong in at least one direction.

Choose a hybrid setup when the work has two very different shapes. Keep simple classification, embeddings, or routine transformations on a small local model. Route hard reasoning and bursts to an API. A hybrid system is less elegant than one universal model, but elegance does not pay the GPU invoice.

The failure modes are predictable. Teams count model weights but omit KV-cache memory. They benchmark one prompt at batch size one and deploy a concurrent agent swarm. They compare a self-hosted model’s raw token speed with an API model’s billed output price. They forget that a GPU server has to be reachable, patched, and monitored at 3 a.m. They also treat privacy as a free benefit, even though private deployment shifts the burden to access control, audit logs, and incident response.

A safer workflow is a two-week measurement exercise. Export token counts and latency from the current API workload. Tag requests as interactive, repeated-context, or deferred. Apply caching and batch processing where they fit. Run the same representative prompts through the intended self-hosted model with the concurrency you expect, not the concurrency that looks good in a screenshot. Record useful tokens per dollar, p95 latency, error rate, and operator hours. Then plug those numbers into the formulas above.

The conclusion is uncomfortable but useful: self-hosting is usually a capacity decision before it is a cost decision. If the workload does not keep a machine busy, the API is renting flexibility. If it does, or if data cannot leave your network, self-hosting can be the right trade. But buy the GPU after the logs prove the need, not after a model card makes the hardware look cheap.

Sources