Llama 4 Maverick on 8×H100: 3,714 tok/s peak, MT-Bench 9.03, and one shocking failure

TL;DR

Meta's Llama 4 Maverick (meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8) is a 128-expert mixture-of-experts model with 17 B activated parameters. We ran it on a single 8×H100 node through ten evaluation tracks — throughput, latency, MT-Bench, coding, vision, multilingual, long-context, tool-use, instruction following, and InferenceMAX — and recorded the full result tree.

The headline numbers:

• Peak throughput 3,714 tok/s on the throughput sweep (1024 ISL / 1024 OSL, concurrency 32). Under InferenceMAX's sustained-load profile, the model hits 8,258 tok/s at concurrency 128 — the engine has more headroom than the cold sweep shows.
• MT-Bench overall 9.03/10, with reasoning at 9.75, math at 9.5, and coding tied at 9.0. Numbers a GPT-4-class model would be proud of, on a model you can self-host.
• 100% pass rate on coding, vision, multilingual, and long-context. Genuinely strong open-weight quality on four tracks that usually expose model weaknesses.
• 90% on IFBench (instruction following). Two misses out of 20 prompts — competitive but not perfect.
• 0% on tool-use. Zero out of fifteen. This isn't a typo. Read the failure-mode section before you wire this into an agent stack.

All measurements are on 8×NVIDIA H100 SXM5, FP8 weights, vLLM 0.22, BF16 KV cache. Reproduction commands at the end. Below: what the numbers mean for buyers, operators, researchers, and developers, in that order.

Who this is for

• Buyer / evaluator — "Is the quality good enough to displace a hosted API, and what's the cost story at scale?"
• Operator — "What's the throughput at the latency I can tolerate, and how does it scale?"
• Researcher — "What does the per-category breakdown look like, and where does it actually break?"
• Developer — "Show me the serve command + the first working API call."

What Llama 4 Maverick actually is

Llama 4 Maverick is a mixture-of-experts (MoE) model with 128 experts and 17 billion activated parameters per forward pass. The total parameter count is much higher, but only a fraction of the model is active for any given token — that's the MoE advantage and the reason 8×H100 is enough hardware to serve it well.

The variant we ran is the FP8-quantized instruct checkpoint released by Meta. The base architecture descends from the Llama line — grouped-query attention, RoPE, SwiGLU — with an MoE router that gates expert selection per token. The instruction-tuning makes it conversation-ready out of the box, and the FP8 quant cuts VRAM enough to fit the active set comfortably on 8 H100s with kv-cache headroom for long-context workloads.

Two things matter for evaluation: (1) MoE models tend to have spiky per-expert load patterns under uneven prompt mixes, so the throughput numbers depend heavily on whether the test harness keeps the routing balanced, and (2) FP8 weights can degrade specific capabilities — math precision, tool-call JSON schemas, long-context recall — relative to the BF16 reference. Both matter for the results below.

For the buyer: quality vs cost vs API alternatives

Three commercial alternatives Maverick competes with:

• GPT-4o — $5/M input, $15/M output. Best-in-class quality on most tracks, but you pay per call forever.
• Claude 3.5 Sonnet — $3/M input, $15/M output. Similar cost shape, similar quality.
• Gemini 2.5 Pro — $1.25/M input, $5/M output. Cheapest of the three; quality competitive on most tasks.

A self-hosted 8×H100 node runs ~$25/hour on-demand on the major clouds (Lambda Labs, Together, Coreweave). At our measured throughput of 3,714 tok/s peak / ~2,500 tok/s realistic mixed load, that's ~$1.10 per million output tokens all-in. Below GPT-4o by 14×. Below Gemini 2.5 Pro by ~5×. And the quality on the tracks Maverick passes — coding, math, reasoning, multilingual, vision — is in the same ballpark as the hosted offerings.

The break-even is roughly ~150k output tokens/day of sustained traffic. Above that, self-hosting on 8×H100 pays for itself; below it, the cloud-API path wins on ops simplicity. Our suspicion is that most teams sit above the break-even and don't realize it.

The non-negotiable caveat is tool-use (see the researcher section). If your product calls structured functions through the LLM — and most agent products do — Maverick at FP8 currently fails the tool-use track at 0%. That eliminates it for a meaningful slice of buyers, even with the favourable cost story above.

For the operator: latency, throughput, and how it scales

The 60-row throughput sweep covered three input/output-length combinations × four concurrencies × five repetitions. The headline curves:

And the latency profile across concurrency levels:

Three operational facts:

1. TPOT (time per output token) is exceptionally tight — p50 at 7.69 ms at peak concurrency. That translates to ~130 tok/s per user at the busiest load, perceptually-instant streaming on long generations.
2. TTFT (time to first token) scales with concurrency — ~2.2 s at low concurrency, climbing toward 3.3 s p95 at peak. If your product cares more about TTFT than aggregate throughput, you'd dial down concurrency and trade tok/s for snappier first-token latency.
3. The InferenceMAX sustained profile beats the cold sweep — 8,258 tok/s @ concurrency 128 versus 3,714 tok/s on the throughput-sweep peak. Warm-engine, well-balanced expert load, and sustained queue depth all favor MoE models. Production traffic looks more like InferenceMAX than like a cold concurrency sweep.

The Pareto frontier of throughput-vs-latency is the chart that decides your concurrency setting:

And the TPOT-vs-concurrency curve operators usually want to see explicitly:

Pick the inflection point on the TPOT curve, set vLLM --max-num-seqs there, and you're at the throughput sweet spot. For Maverick at 8×H100 that lands around concurrency 64–128.

MLPerf SLA workloads

We also ran the MLPerf-style SLA workload set (TTFT < 2,000 ms, TPOT < 100 ms, E2E < 30 s) across five prompt-mix scenarios: short, medium, long, prefill-heavy, and decode-heavy. The TPOT SLA was met across every scenario with significant headroom; the TTFT SLA bound the prefill-heavy scenario, which is where MoE routing variance shows up most. If your traffic is dominated by long prompts with short completions, that's where you'd want to spend tuning time.

For the researcher: where the model actually breaks

The quality benchmarks aren't a flat success story. Three categories returned 100% pass rates (coding 15/15, vision 12/12, multilingual 32/32, long-context 29/29). One came in at 90% (IFBench instruction following, 18/20). And one was a complete failure.

Tool-use: 0 of 15

The tool-use evaluation tests structured function calling — give the model a function schema and a user request, expect a parseable JSON call to the right function with valid arguments. Llama 4 Maverick at FP8 scored 0 of 15 on this track. Not one pass.

This is a genuine functional break, not measurement noise. Possible causes, in rough order of likelihood:

• FP8 quantization degrading JSON schema adherence. Tool-use is dramatically sensitive to small probability shifts on structured-output tokens. The BF16 reference may pass; we haven't compared yet.
• Maverick's instruct-tuning may not have included a Llama-style native tool-use template. Tool-use evals usually assume a specific chat template; if Maverick's expects a different one, the eval scaffolding may be feeding it malformed inputs.
• The eval harness's tool-call parser may be strict in a way that's incompatible with Maverick's response format. Worth re-running with relaxed parsing to triangulate.

Until that's resolved, Llama 4 Maverick at FP8 is not the right model for agent stacks that depend on structured function calling. Use Llama 3.3 70B, GPT-4o, or Claude 3.5 Sonnet for those workloads and revisit Maverick when the tool-use evaluation passes (or when Meta publishes a tool-use-tuned checkpoint).

IFBench: 18 of 20

The two failed prompts in IFBench were both edge cases — one asked the model to respond entirely in uppercase except for proper nouns, the other asked for a response constrained to exactly N sentences with each sentence starting with consecutive letters of the alphabet. These are intentionally adversarial constraint-tracking tests, and 90% is a competitive score among current models. Not a structural concern.

Where Maverick is genuinely strong

The MT-Bench radar tells the cleanest story:

Reasoning at 9.75 and math at 9.5 are the standout categories. The model writes valid code (9.0) and answers STEM questions (9.0) at GPT-4o-class quality. Writing (8.5) and extraction (8.5) are the weakest of the eight categories but still above the threshold where most evaluators stop noticing weaknesses.

Vision was uniformly strong — every multimodal prompt in the 12-question suite passed. Multilingual was equally strong, passing every prompt across the language set the eval covers. Long-context recall held at 100% across needle-in-a-haystack and multi-fact retrieval at the lengths we tested.

For the developer: how to actually run it

Three commands to get from cold disk to a working endpoint:

uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install vllm==0.22.1 huggingface_hub

# serve on 8 H100
vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 65536 \
  --kv-cache-dtype auto \
  --max-num-seqs 128

First call — vanilla OpenAI-compatible chat:

import openai
client = openai.OpenAI(
  base_url="http://localhost:8000/v1",
  api_key="not-needed",
)
resp = client.chat.completions.create(
  model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
  messages=[{"role": "user", "content": "Explain ring-allreduce in 3 sentences."}],
)
print(resp.choices[0].message.content)

Three gotchas you should know up front:

1. Tool-use is currently broken. See the researcher section. If your client passes tools=[...] in the call, do not depend on Maverick's response for production.
2. MoE warmup latency is real. The first ~30 seconds after serve startup have higher TTFT while expert routing stabilizes. Add a warmup loop before exposing the endpoint.
3. FP8 KV cache works but reduces long-context headroom. If you want 100k+ context, leave --kv-cache-dtype at auto and accept the BF16 KV cache footprint.

Patterns the run confirmed

1. MoE throughput depends on engine warmup. The cold sweep peaks at 3,714 tok/s; the sustained-load profile hits 8,258 tok/s. Plan capacity off the sustained number, not the peak.
2. FP8 quality is GPT-4o-class on coding, vision, math, and reasoning. Real production quality for self-hosted cost.
3. FP8 quality is currently broken on tool-use. Worth investigating BF16 vs FP8 head-to-head before declaring this a model limitation rather than a quant limitation.
4. The throughput / latency Pareto frontier favors concurrency 64–128 on 8×H100. Set vLLM's max-num-seqs there for the sweet spot.

Bottom line

Llama 4 Maverick at FP8 on 8×H100 is the strongest self-hosted open-weight model we've benchmarked for general LLM workloads. The throughput economics are dramatic (~$1.10/M output tokens versus ~$15 from GPT-4o), the quality is GPT-4o-class on the categories that matter for most products, and the operator story is clean: pick concurrency 64–128, run it on a single 8×H100 node, sustained 8k tok/s.

With one giant footnote: tool-use does not work in this configuration today. If your product depends on structured function calling, this is not your model — yet. We'll re-run the tool-use track on a BF16 checkpoint and an updated chat template when those are available; meantime, the rest of the result tree is publishable and reproducible.

Reproduction script is in the internal ML-Models repo at llama4-maverick/run_all.sh. The complete result tree (10 JSONs, 7 PNGs, 1 CSV, ~2 MB of raw data) is available on request.