Updated minutes ago
Mistral

Mixtral 8x7B Instruct

Mistral AI · moe · 46.7B parameters · 32,768 context

Quality
69.0
Calculate ROI →Compare with othersFine-Tune This Model →

Parameters

46.7B

Context Window

32K tokens

Architecture

MoE

Best GPU

B200 SXM

Cheapest API

$0.60/M

Quality Score

69/100

Intelligence Brief

Mixtral 8x7B Instruct is a 46.7B parameter Mixture-of-Experts (8 experts, 2 active) model from Mistral AI, featuring Grouped Query Attention (GQA) with 32 layers and 4,096 hidden dimensions. With a 32,768 token context window, it supports tools, structured output, code, math, multilingual. On standardized benchmarks, it achieves MMLU 72.6, HumanEval 42, GSM8K 76. The most cost-effective API deployment is via together at $0.60/M output tokens. For self-hosted inference, B200 SXM delivers optimal throughput at $4261/month.

Architecture Details

TypeMOE
Total Parameters46.7B
Active Parameters12.9B
Layers32
Hidden Dimension4,096
Attention Heads32
KV Heads8
Head Dimension128
Vocab Size32,000
Total Experts8
Active Experts2

Memory Requirements

BF16 Weights

93.4 GB

FP8 Weights

46.7 GB

INT4 Weights

23.4 GB

KV-Cache per Token131072 bytes
Activation Estimate1.50 GB

Fits on (single GPU) — most practical first

BEST FITRTX 509032GBINT4V100 32GB32GBINT4Instinct MI10032GBINT4TPU v432GBINT4TPU v6e (Trillium)32GBINT4Cloud AI 10032GBINT4A100 40GB SXM40GBINT4A100 40GB PCIe40GBINT4RTX A600048GBINT4A4048GBINT4L40S48GBINT4L4048GBINT4

GPU Compatibility Matrix

Mixtral 8x7B Instruct is compatible with 52% of GPU configurations across 41 GPUs at 3 precision levels.

BF16 (Full)
FP8 (Half)
INT4 (Quarter)
Blackwell(7 GPUs)
B200 NVL (pair)360GB
74%
87%
94%
B300288GB
68%
84%
92%
B100 SXM192GB
51%
76%
88%
GB200 NVL72 (per GPU)192GB
51%
76%
88%
Hopper(7 GPUs)
H100 NVL 94GB (per GPU pair)188GB
50%
75%
88%
H200 SXM141GB
34%
67%
83%
H2096GB
3%
51%
76%
GH20096GB
3%
51%
76%
Ada Lovelace(11 GPUs)
L40S48GB
✗
3%
51%
L4048GB
✗
3%
51%
RTX 6000 Ada48GB
✗
3%
51%
L2048GB
✗
3%
51%
Ampere(16 GPUs)
A100 80GB SXM80GB
✗
42%
71%
A100 80GB PCIe80GB
✗
42%
71%
A1664GB
✗
27%
64%
RTX A600048GB
✗
3%
51%
Legend:No fitVery tightTightModerateGoodExcellent

GPU Recommendations

B200 SXMoptimal

FP8 · 1 GPU · tensorrt-llm

100/100

score

Throughput

1.1K tok/s

Latency (ITL)

1.0ms

Est. TTFT

0ms

Cost/Month

$4261

Cost/M Tokens

$1.54

Use this config →
B100 SXMoptimal

FP8 · 1 GPU · tensorrt-llm

100/100

score

Throughput

1.1K tok/s

Latency (ITL)

1.0ms

Est. TTFT

0ms

Cost/Month

$4271

Cost/M Tokens

$1.55

Use this config →
GB200 NVL72 (per GPU)optimal

FP8 · 1 GPU · tensorrt-llm

100/100

score

Throughput

1.1K tok/s

Latency (ITL)

1.0ms

Est. TTFT

0ms

Cost/Month

$6169

Cost/M Tokens

$2.24

Use this config →

Deployment Options

API

API Deployment

together

$0.60/M

output tokens

Self-Hosted

Single GPU

B200 SXM

$4261/mo

Min VRAM: 47 GB

Scale

Multi-GPU

A100 80GB SXM x2

834.1 tok/s

TP· $2259/mo

Open Full Calculator →

API Pricing Comparison

ProviderInput $/MOutput $/MBadges
together$0.60$0.60
Cheapest

Cost Analysis

ProviderInput $/MOutput $/M~Monthly Cost
togetherBest Value$0.60$0.60$6

Cost per 1,000 Requests

Short (500 tok)

$0.42

via together

Medium (2K tok)

$1.68

via together

Long (8K tok)

$6.00

via together

Performance Estimates

Throughput by GPU

B200 SXM
1.1K tok/s
B100 SXM
1.1K tok/s
GB200 NVL72 (per GPU)
1.1K tok/s

VRAM Breakdown (B200 SXM, FP8)

Weights
Act
Weights 46.7 GBKV-Cache 1.1 GBActivations 12.0 GBOverhead 2.3 GB

Precision Impact

bf16

93.4 GB

weights/GPU

fp8

46.7 GB

weights/GPU

~1.1K tok/s

int4

23.4 GB

weights/GPU

Quality Benchmarks

Average
73th percentile across all models
MMLU
72.6
Below Average (40th pctile)
HumanEval
42.0
Below Average (30th pctile)
GSM8K
76.0
Below Average (40th pctile)
MT-Bench
78.0
Bottom 25% (0th pctile)

Capabilities

Features

✓ Tool Use✗ Vision✓ Code✓ Math✗ Reasoning✓ Multilingual✓ Structured Output

Supported Frameworks

vllmsglangtgitensorrt-llmollama

Supported Precisions

BF16 (default)FP8INT4

Where to Deploy Mixtral 8x7B Instruct

Similar Models

Mixtral 8x7B

46.7B params · moe

Quality: 67

from $0.50/M

Similar specsCompare →

Amazon Nova Pro

50B params · dense

Quality: 50

from $3.20/M

Larger context, Lower quality, More expensiveCompare →

Gemini 2.0 Flash

50B params · moe

Quality: 80

from $0.40/M

Larger context, Higher quality, CheaperCompare →

Gemini 1.5 Flash

50B params · moe

Quality: 75

from $0.30/M

Larger context, Higher quality, CheaperCompare →

Llama 3.1 Nemotron 51B

51B params · dense

Quality: 78

from $0.40/M

Larger context, Higher quality, CheaperCompare →

Frequently Asked Questions

How much VRAM does Mixtral 8x7B Instruct need for inference?

Mixtral 8x7B Instruct requires approximately 93.4 GB of VRAM at BF16 precision, 46.7 GB at FP8, or 23.4 GB at INT4 quantization. Additional VRAM is needed for KV-cache (131072 bytes per token) and activations (~1.50 GB).

What is the best GPU for Mixtral 8x7B Instruct?

The top recommended GPU for Mixtral 8x7B Instruct is the B200 SXM using FP8 precision. It achieves approximately 1.1K tokens/sec at an estimated cost of $4261/month ($1.54/M tokens). Score: 100/100.

How much does Mixtral 8x7B Instruct inference cost?

Mixtral 8x7B Instruct API inference starts from $0.60/M input tokens and $0.60/M output tokens. Self-hosted inference costs depend on your GPU configuration — use our ROI calculator for a detailed breakdown.