The real math of self-hosting models.

Self-hosting an open-weight model is a capital decision dressed up as an engineering decision. The engineering is real, but it is not what decides the outcome. One number decides it: sustained utilization. Not peak throughput, not benchmark tokens per second, not the sticker price of the GPU. The fraction of every hour, every day, that your hardware is actually generating tokens someone asked for.

A GPU cluster running at 15% utilization is an API subscription with extra steps. You still pay per token; you just also pay for an on-call rotation, a procurement cycle, and 85% of a very expensive machine doing nothing. We would rather tell you that in a scoping call than have you discover it on a depreciation schedule.

The only variable that matters

API pricing is linear: every token costs the same, whether you send one million or one hundred billion. Self-hosting is step-fixed: the node costs the same per month whether it is saturated or idle. So the per-token cost of self-hosting is not a number, it is a curve, and utilization is the denominator.

Our reference bench: a 4×H100 node running Qwen3.5-122B-A10B at FP8 with continuous batching. Owned hardware, amortized over 36 months, plus power, colocation, and a share of ops labor comes to about $6,200 per month all-in. At sustained load the node holds roughly 6,000 output tokens per second, which is about 15.5B tokens per month of capacity. Our API baseline for a comparable 70B-class model is $0.75 per million tokens, blended input and output.

Divide fixed cost by tokens actually served and the curve falls out. This is our bench; your numbers will differ, and the table is built so you can rerun it with yours.

Assumptions stated: $6,200/mo node cost, 15.5B tokens/mo capacity, $0.75/M API price. Our bench; your numbers will differ. The break-even sits at roughly 53% sustained utilization, and every input to that figure is one division away from being checked:

# back-of-envelope: is self-hosting worth it for you?
node_cost_month = 6_200          # 4xH100 owned, 36-mo amort + power + colo + ops
peak_tok_s      = 6_000          # Qwen3.5-122B-A10B, FP8, batched (our bench)
api_price_per_m = 0.75           # blended $/M for comparable API

cap_m_tokens = peak_tok_s * 86_400 * 30 / 1e6   # ~15,552 M tokens/mo

your_tokens_per_day_m = 40       # <-- put your real number here
util = your_tokens_per_day_m * 30 / cap_m_tokens

self_host_per_m = node_cost_month / (cap_m_tokens * util)
breakeven_util  = node_cost_month / (cap_m_tokens * api_price_per_m)

print(f"utilization:      {util:.0%}")        # 8%
print(f"self-host $/M:    {self_host_per_m:.2f}")   # $5.17
print(f"break-even util:  {breakeven_util:.0%}")    # 53%

Forty million tokens a day sounds like a lot. It is 8% utilization of one node, and it prices out at $5.17 per million against an API at $0.75. Volume that feels large in a dashboard is often small against hardware capacity. That gap is where most bad self-hosting decisions are made.

The costs people forget to model

The table above is the flattering version, because it prices the hardware and stops. Four costs show up later and they are all real.

Ops on-call

An inference fleet pages. Driver regressions, NVLink flaps, a KV cache leak after a serving-framework upgrade. Someone answers at 3am or your product is down. If that someone does not already exist on your payroll, add a meaningful fraction of an SRE to the monthly number before you compare anything.

Eval regression testing

On an API, the provider upgrades the model and you mostly ride along. Self-hosted, every weight update, quantization change, and serving-engine bump is your migration. You need a frozen eval set and a regression gate, and you need to run it every time. We budget real engineering days per upgrade cycle for this, not hours.

Capacity headroom

You cannot run a latency-sensitive service at 100% utilization; queues form and p99 dies. Practical ceilings sit around 70–80% for interactive traffic. That means your effective capacity is smaller than the spec sheet, which pushes the break-even further out. Batch workloads can run hotter, which is one reason they self-host so well.

Egress, if you go hybrid

Teams that self-host the cheap 80% of traffic and burst the rest to an API often forget that shipping context back and forth across a cloud boundary is metered. On long-context workloads, egress fees can quietly claw back a chunk of the savings. Model it or be surprised by it.

When we tell prospects not to self-host

We sell self-hosting. We still say no to it regularly, because a bad-fit deployment becomes a reference against us within a year. The no cases are consistent:

• Low volume. Under a few hundred million tokens a day on a 70B-class model, the arithmetic above simply does not close. Stay on the API and spend the engineering elsewhere.
• Spiky traffic. A 10:1 peak-to-mean ratio means you either size for the peak and idle, or size for the mean and fail at the peak. APIs absorb spikes for you; that absorption is worth paying for.
• No residency or sovereignty requirement. If no regulator, contract, or customer forbids your data leaving a boundary, then cost is the only argument, and cost alone needs the utilization to back it up.

There is also a softer signal: if a team wants to self-host primarily because the GPUs feel like an asset, we slow the conversation down. Depreciating silicon is a liability with a fan attached unless the tokens flow.

When it is a clear yes

The yes cases are just as consistent. Steady, high-volume traffic, the kind where a nightly batch pipeline or a high-QPS product keeps the fleet above 60% around the clock. Hard sovereignty requirements, where the data cannot leave the building regardless of price. And latency co-location, where the model must sit next to the data or the user and a round-trip to a provider region breaks the budget.

When those conditions hold, the results are not subtle. Our sovereign-infrastructure build ran the full playbook above and landed at −78% per million tokens against the API bill it replaced, with 0 bytes of egress, because nothing ever left the boundary. On client fleets with sustained load we serve Qwen3.5 fleets under $0.40 per million tokensblended, all-in. Those numbers are only available on the right side of the break-even chart. On the wrong side, they are fiction.

The checklist we run at scoping

Every self-hosting engagement starts with the same seven questions. You can run them yourself before you ever talk to us:

1. Tokens per day, from logs. Measured, not projected. Ninety days of history if you have it.
2. Peak-to-mean ratio. Above roughly 3:1, self-hosting alone gets hard; think hybrid or stay put.
3. Residency requirement, in writing. A contract clause or regulation, not a preference.
4. Latency budget. Does co-location buy you something an API region cannot?
5. Who carries the pager. A named team, or a hiring plan with dates.
6. Eval harness. A frozen test set you trust enough to gate a model upgrade on.
7. Honest all-in node cost. Amortization, power, colo, ops labor. Then divide by realistic utilization, not capacity.

Strong answers on volume plus at least one of sovereignty or latency, and self-hosting will probably pay for itself within the first hardware cycle. Weak answers on volume and the rest does not matter.

Pull your last 90 days of token logs, run the ten-line script above with your real numbers, and see which side of 53% you land on. If the math is close or the constraints are unusual, send us the output and we will run the full version against our bench data. We will tell you if the answer is no, in writing, because that answer is the cheapest thing we sell.