Updated minutes ago
ReleasedOctober 29, 2024Verified 3mo ago · docs.anthropic.com
Anthropic

Claude 3.5 Haiku

Anthropic · dense · 20B parameters · 200,000 context

Quality
67.0
Deploy this model →Use via API →Calculate ROI →Compare with othersFine-tune this model

Parameters

20B

Context Window

195K tokens

Architecture

Dense

Best GPU

H100 SXM

Cheapest API

$4.00/M

Quality Score

67/100

Intelligence Brief

Claude 3.5 Haiku is a 20B parameter DENSE model from Anthropic, featuring Grouped Query Attention (GQA) with 40 layers and 5,120 hidden dimensions. With a 200,000 token context window, it supports tools, vision, structured output, code, math, multilingual. On standardized benchmarks, it achieves MMLU 78, HumanEval 55, GSM8K 85. The most cost-effective API deployment is via anthropic at $4.00/M output tokens. For self-hosted inference, H100 SXM delivers optimal throughput at $1794/month.

Provider pricing

2 providers · canonical: anthropic
Provider Input $/M Output $/M ▲Notes
anthropiccanonical$0.800$4.00cheapest output
openrouter$0.800$4.00cheapest input · cheapest output

Prices update via the nightly pricing cron + admin approvals at /admin/ingest-queue. The leaderboard's Input/Output cells show the canonical rate above; this table shows the full spread.

Recent changes

Loading…

Related models

5 suggestions

Picks: same family first, then same vendor within ±2× params, then top tag-overlap matches. Price shown is the cheapest Output $/M across providers — the row's page shows the canonical anchor.

Architecture Details

TypeDENSE
Total Parameters20B
Active Parameters20B
Layers40
Hidden Dimension5,120
Attention Heads40
KV Heads8
Head Dimension128
Vocab Size152,064

Memory Requirements

BF16 Weights

40.0 GB

FP8 Weights

20.0 GB

INT4 Weights

10.0 GB

KV-Cache per Token40960 bytes
Activation Estimate1.50 GB

Fits on (single GPU) — most practical first

BEST FITRTX 4070 Ti12GBINT4RTX 4070 Super12GBINT4RTX 306012GBINT4A400016GBINT4RTX 408016GBINT4T416GBINT4V100 16GB16GBINT4RTX 4060 Ti 16GB16GBINT4A216GBINT4TPU v5e16GBINT4A10G24GBFP8A3024GBFP8

GPU Compatibility Matrix

Claude 3.5 Haiku is compatible with 74% of GPU configurations across 41 GPUs at 3 precision levels.

BF16 (Full)
FP8 (Half)
INT4 (Quarter)
Blackwell(7 GPUs)
B200 NVL (pair)360GB
89%
94%
97%
B300288GB
86%
93%
97%
B100 SXM192GB
79%
90%
95%
GB200 NVL72 (per GPU)192GB
79%
90%
95%
Hopper(7 GPUs)
H100 NVL 94GB (per GPU pair)188GB
79%
89%
95%
H200 SXM141GB
72%
86%
93%
H2096GB
58%
79%
90%
GH20096GB
58%
79%
90%
Ada Lovelace(11 GPUs)
L40S48GB
17%
58%
79%
L4048GB
17%
58%
79%
RTX 6000 Ada48GB
17%
58%
79%
L2048GB
17%
58%
79%
Ampere(16 GPUs)
A100 80GB SXM80GB
50%
75%
88%
A100 80GB PCIe80GB
50%
75%
88%
A1664GB
38%
69%
84%
RTX A600048GB
17%
58%
79%
Legend:No fitVery tightTightModerateGoodExcellent

GPU Recommendations

H100 SXMoptimal

BF16 · 1 GPU · tensorrt-llm

100/100

score

Throughput

722.3 tok/s

Latency (ITL)

1.4ms

Est. TTFT

0ms

Cost/Month

$1794

Cost/M Tokens

$0.94

Use this config →
H100 PCIeoptimal

BF16 · 1 GPU · tensorrt-llm

100/100

score

Throughput

431.2 tok/s

Latency (ITL)

2.3ms

Est. TTFT

0ms

Cost/Month

$1794

Cost/M Tokens

$1.58

Use this config →
H20optimal

BF16 · 1 GPU · tensorrt-llm

95/100

score

Throughput

862.5 tok/s

Latency (ITL)

1.2ms

Est. TTFT

0ms

Cost/Month

$940

Cost/M Tokens

$0.41

Use this config →

Deployment Options

API

API Deployment

anthropic

$4.00/M

output tokens

Self-Hosted

Single GPU

H100 SXM

$1794/mo

Min VRAM: 20 GB

Scale

Multi-GPU

A100 40GB SXM x2

412.1 tok/s

TP· $1613/mo

Open Full Calculator →

API Pricing Comparison

ProviderInput $/MOutput $/MBadges
anthropic$0.80$4.00
Cheapest
openrouter$0.80$4.00
Low Input

Cost Analysis

ProviderInput $/MOutput $/M~Monthly Cost
anthropicBest Value$0.80$4.00$24
openrouter$0.80$4.00$24

Cost per 1,000 Requests

Short (500 tok)

$1.20

via anthropic

Medium (2K tok)

$4.80

via anthropic

Long (8K tok)

$14.40

via anthropic

Performance Estimates

Throughput by GPU

H100 SXM
722.3 tok/s
H100 PCIe
431.2 tok/s
H20
862.5 tok/s

VRAM Breakdown (H100 SXM, BF16)

Weights
Act
Weights 40.0 GBKV-Cache 2.7 GBActivations 12.0 GBOverhead 2.0 GB

Quality Benchmarks

Average
70th percentile across all models
MMLU
78.0
Below Average (47th pctile)
HumanEval
55.0
Average (54th pctile)
GSM8K
85.0
Average (51th pctile)
MT-Bench
83.0
Bottom 25% (0th pctile)

Capabilities

Features

✓ Tool Use✓ Vision✓ Code✓ Math✗ Reasoning✓ Multilingual✓ Structured Output

Supported Frameworks

Supported Precisions

BF16 (default)

Where to Deploy Claude 3.5 Haiku

Similar Models

Claude Haiku 4.5

30B params · moe

Quality: 50

from $5.00/M

Lower qualityCompare →

GPT-3.5 Turbo

20B params · dense

Quality: 67

from $1.50/M

Smaller context, CheaperCompare →

GigaChat 20B

20B params · dense

Quality: 50

Smaller context, Lower qualityCompare →

InternLM 20B

20B params · dense

Quality: 50

Smaller context, Lower qualityCompare →

InternLM 2.5 20B

19.9B params · dense

Quality: 50

from $0.50/M

Lower quality, CheaperCompare →

Frequently Asked Questions

How much VRAM does Claude 3.5 Haiku need for inference?

Claude 3.5 Haiku requires approximately 40.0 GB of VRAM at BF16 precision, 20.0 GB at FP8, or 10.0 GB at INT4 quantization. Additional VRAM is needed for KV-cache (40960 bytes per token) and activations (~1.50 GB).

What is the best GPU for Claude 3.5 Haiku?

The top recommended GPU for Claude 3.5 Haiku is the H100 SXM using BF16 precision. It achieves approximately 722.3 tokens/sec at an estimated cost of $1794/month ($0.94/M tokens). Score: 100/100.

How much does Claude 3.5 Haiku inference cost?

Claude 3.5 Haiku API inference starts from $0.80/M input tokens and $4.00/M output tokens. Self-hosted inference costs depend on your GPU configuration — use our ROI calculator for a detailed breakdown.