Serving Qwen3.5 for under $0.40 per million tokens.

When a team tells us self-hosting is more expensive than calling an API, they are almost always describing an idle GPU, not an expensive model. An 8×H100 node running at 12% utilization costs the same per hour as one running at 85%. The second one produces seven times the tokens. Everything in this post is a technique for closing that gap.

The headline number is a blended fleet cost: most traffic served by the dense workhorse Qwen3.5-27B, the hard minority by the flagship Qwen3.5-397B-A17B, both Apache 2.0. Run that split at high utilization and the fleet lands under $0.40 per million tokens. In one deployment that worked out to 78% below the equivalent API bill, with 0 bytes of prompt data leaving the client's network (see our sovereign infrastructure case study). Here is the whole method, in the order requests experience it.

Why the flagship is cheap: 397B parameters, 17B working

Qwen3.5-397B-A17B is a sparse mixture-of-experts model: 397B total parameters, but only about 17B are active for any given token. You pay VRAM for the whole model and compute for the active slice. That asymmetry is the entire economic story — frontier-class quality at the per-token compute cost of a mid-size model. It is also why the serving math below looks nothing like the dense-70B era: the bottleneck shifts from raw FLOPs to memory capacity and routing.

The request path

Every request enters through a gateway that does auth, rate limiting, and request logging. Behind it sits a router that decides which model pool the request deserves. Behind the router sit vLLM pods: a pool of 2-GPU pods serving Qwen3.5-27B, and a pool of 8-GPU tensor-parallel pods serving the 397B flagship. That is the entire topology.

Nothing here is exotic. The cost savings come from what each box does under load, so the rest of this post walks the path right to left: how the pods stay busy, how the model fits, and how the router keeps expensive silicon reserved for questions that need it.

Continuous batching: utilization is the whole game

A single decode step for one sequence uses a small fraction of an H100's compute. The GPU spends most of its time waiting on memory reads of the model weights. If you serve one request at a time, you pay for the whole weight read and use it to produce one token. If you batch 48 requests, you pay for roughly the same weight read and produce 48 tokens. Decode throughput scales close to linearly with batch size until you saturate memory bandwidth or run out of cache space.

Static batching (wait for 48 requests, run them together, return when the longest finishes) wastes capacity because sequences finish at different times. Continuous batching, which vLLM does by default, admits new sequences into the running batch the moment old ones finish. The batch stays full, per-token latency stays flat, and the GPU stays at high occupancy as long as there is queued work.

This is why we say utilization is the whole game. Every dollar figure later in this post assumes the batch is full. A flagship pod serving batch-size-2 traffic all day produces tokens at roughly 20x the cost of the same pod at batch 48. No quantization trick recovers that.

Our standard launch config for an 8×H100 flagship pod:

# fp8 is native on Hopper (fp4 to fit a 4-GPU node).
# max-model-len caps context; max-num-seqs caps the batch.
# prefix caching means shared system prompts hit cache.
vllm serve Qwen/Qwen3.5-397B-A17B \
  --quantization fp8 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching

Two of these flags do most of the work. max-model-len caps how much cache a single greedy request can claim, and max-num-seqs sets how full the batch is allowed to get. The model's native context is 262k tokens; serving it uncapped because it is possible is how fleets die. Tune both against your real traffic distribution, not the defaults.

Quantization: FP8 by default, FP4 to shrink the node

The 397B checkpoint is ~807GB on disk in BF16. Nobody serves it that way on sane budgets. The choice is between two working configurations:

• FP8 (W8A8) on an 8×H100 node. Hopper has native FP8 tensor cores, the quality loss is close to measurement noise on our evals, and it is our default.
• FP4/INT4 when the budget demands a 4×H100 node. The fleet halves in price and the batch budget tightens. Quality loss is small but nonzero; we gate it with evals per workload.
• No aggressive quantization when the workload punishes it.

That last case is real. We refuse to quantize below FP8 when the task is structured extraction with strict JSON schemas, code generation that feeds a compiler, or anything where our before/after eval shows a regression above the client's tolerance. Low-bit error concentrates in exactly the low-probability tail that structured outputs depend on. When a client's eval suite drops more than a point, we eat the extra GPU cost. Quantization is a cost lever, not a default posture.

The cache problem got smaller — by architecture

Dense transformers pay a growing memory tax: every token in an active request keeps attention keys and values resident in GPU memory, so long contexts strangle batch size. Qwen3.5 changes that arithmetic structurally. Its 60 layers are built as repeating blocks of three linear-attention layers (Gated DeltaNet) followed by one standard gated-attention layer. The linear-attention layers carry a fixed-size recurrent state instead of a per-token cache — only a quarter of the stack accumulates KV at all.

The operational consequence: cache pressure grows several times slower with context length than on the dense models we served in 2025. Long-context requests that used to halve a pod's batch size now shave it. Paged attention in vLLM still manages what cache remains — allocated in fixed blocks on demand, near-zero fragmentation — but the ceiling it manages is meaningfully higher. You still size max-model-len so one greedy request cannot starve the batch; you just stop paying rent on context you never used.

Route cheap questions to the cheap pool

Most production traffic does not need the flagship. In support-style workloads on our bench, the majority of requests are greetings, FAQ-shaped lookups, and short rewrites that Qwen3.5-27B answers as well as the 397B does. The router in fig 1 classifies each prompt before dispatch: a small classifier (a fine-tuned encoder, ~30ms on CPU) scores the prompt for reasoning depth, and low scorers go to the 27B pool.

The economics are blunt. The 27B pool produces tokens at roughly a fifth of the flagship's cost, so shifting 75% of traffic there cuts the blended cost per million tokens by more than half before any other optimization. We keep the router boring on purpose: a confidence threshold with a fallback rule, so anything ambiguous escalates to the flagship. Misrouting hard questions to the cheap model is a quality incident; misrouting easy questions to the expensive model is just a few cents. And the 27B is no toy — on agentic coding benchmarks the current small dense Qwens trade blows with the flagship, which is exactly why the split can be this aggressive.

Reserved vs spot GPU economics

The other half of the price is the GPU-hour itself. On-demand H100s run $3–4/GPU-hr across the providers we deploy on. A one-year reserved commitment lands closer to $2/GPU-hr. Spot pricing goes lower still, with the risk of a two-minute eviction notice.

Our pattern: size the reserved fleet for the traffic floor (the load you serve at 3am), and stack spot capacity on top for peaks. vLLM pods drain gracefully on preemption, the gateway retries in-flight requests against the reserved pool, and the router sheds 27B-eligible traffic first. Batch and offline workloads (evals, backfills, embedding jobs) run entirely on spot, because a preempted backfill just resumes.

The cost math

Here is the arithmetic end to end, from our test bench. One caveat before the table: we price blended tokens, meaning every token the engine processes, prefill and decode, at a representative 3.5:1 input:output mix. APIs price input and output separately, so compare against your own blended API rate.

Check the arithmetic yourself: dollars per hour divided by millions of tokens per hour. The utilization row is the honest part. Below ~80% sustained utilization the blend drifts past $0.40, and the routing plus continuous batching sections above are the entire mechanism for staying under it. This is our bench; your numbers will differ with your traffic mix, context lengths, and negotiated GPU rates. Published third-party benchmarks for the 397B land in the same band — roughly $0.50–1.80 per million depending on how hard you chase latency — which is why the blend, not the flagship alone, is what gets you under $0.40.

What breaks first at scale

It is rarely the model. The failure modes we actually page on, in the order they usually appear: queue collapse when a traffic spike outruns the batch ceiling and time-to-first-token climbs from 400ms to 20s while throughput looks perfectly healthy; router drift as the traffic mix shifts and yesterday's 75/25 split quietly becomes 55/45, blowing the blended cost without any dashboard turning red; and cache pressure from a new feature that doubles average context length — gentler on this architecture than the last one, but still a capacity change.

The fixes are operational, not architectural. Alert on time-to-first-token and queue depth, not GPU utilization. Recompute the router's traffic split weekly against a labeled sample. And treat average context length as a capacity input: any product change that grows prompts is a capacity change, and it should go through the same review.

If you want to sanity-check your own numbers, start with the two that dominate: your blended API rate today, and your realistic sustained utilization. Put them next to the table above. If the gap looks like ours did, run the bench on a single spot node for a week before you commit to anything reserved.