Phi 3.5 MoE
Microsoft · moe · 41.9B parameters · 131,072 context
Parameters
41.9B
Context Window
128K tokens
Architecture
MoE
Best GPU
B200 SXM
Cheapest API
$0.30/M
Quality Score
74/100
Intelligence Brief
Phi 3.5 MoE is a 41.9B parameter Mixture-of-Experts (16 experts, 2 active) model from Microsoft, featuring Grouped Query Attention (GQA) with 32 layers and 4,096 hidden dimensions. With a 131,072 token context window, it supports tools, structured output, code, math, multilingual. On standardized benchmarks, it achieves MMLU 78.9, HumanEval 52, GSM8K 84. The most cost-effective API deployment is via deepinfra at $0.30/M output tokens. For self-hosted inference, B200 SXM delivers optimal throughput at $4261/month.
Provider pricing
1 provider · canonical: deepinfra| Provider | Input $/M | Output $/M ▲ | Notes |
|---|---|---|---|
| deepinfracanonical | $0.100 | $0.300 | cheapest input · cheapest output |
Prices update via the nightly pricing cron + admin approvals at /admin/ingest-queue. The leaderboard's Input/Output cells show the canonical rate above; this table shows the full spread.
Recent changes
Loading…
Related models
5 suggestions- Phi 1.5Phi · 1.3B—
- Phi 1Phi · 1.3B—
- Phi 2Phi · 2.7B—
- Phi 4 MiniPhi · 3.8B$0.350/M out
- Phi-4Phi · 14.7B$0.140/M out
Picks: same family first, then same vendor within ±2× params, then top tag-overlap matches. Price shown is the cheapest Output $/M across providers — the row's page shows the canonical anchor.
Architecture Details
Memory Requirements
BF16 Weights
83.8 GB
FP8 Weights
41.9 GB
INT4 Weights
20.9 GB
GPU Compatibility Matrix
Phi 3.5 MoE is compatible with 52% of GPU configurations across 41 GPUs at 3 precision levels.
GPU Recommendations
FP8 · 1 GPU · tensorrt-llm
100/100
score
Throughput
1.1K tok/s
Latency (ITL)
1.0ms
Est. TTFT
0ms
Cost/Month
$4261
Cost/M Tokens
$1.54
FP8 · 1 GPU · tensorrt-llm
100/100
score
Throughput
1.1K tok/s
Latency (ITL)
1.0ms
Est. TTFT
0ms
Cost/Month
$4271
Cost/M Tokens
$1.55
FP8 · 1 GPU · tensorrt-llm
100/100
score
Throughput
1.1K tok/s
Latency (ITL)
1.0ms
Est. TTFT
0ms
Cost/Month
$6169
Cost/M Tokens
$2.24
Deployment Options
API Deployment
deepinfra
$0.30/M
output tokens
Single GPU
B200 SXM
$4261/mo
Min VRAM: 42 GB
Multi-GPU
A100 80GB SXM x2
1.1K tok/s
TP· $2259/mo
API Pricing Comparison
| Provider | Input $/M | Output $/M | Badges |
|---|---|---|---|
| deepinfra | $0.10 | $0.30 | Cheapest |
Cost Analysis
| Provider | Input $/M | Output $/M | ~Monthly Cost |
|---|---|---|---|
| deepinfraBest Value | $0.10 | $0.30 | $2 |
Cost per 1,000 Requests
Short (500 tok)
$0.11
via deepinfra
Medium (2K tok)
$0.44
via deepinfra
Long (8K tok)
$1.40
via deepinfra
Performance Estimates
Throughput by GPU
VRAM Breakdown (B200 SXM, FP8)
Precision Impact
bf16
83.8 GB
weights/GPU
fp8
41.9 GB
weights/GPU
~1.1K tok/s
int4
20.9 GB
weights/GPU
Quality Benchmarks
Capabilities
Features
Supported Frameworks
Supported Precisions
Where to Deploy Phi 3.5 MoE
Self-Hosted Infrastructure
Similar Models
Phi-4
14.7B params · dense
Quality: 73
from $0.14/M
Falcon 40B
40B params · dense
Quality: 48
from $0.80/M
VILA 1.5 40B
40B params · dense
Quality: 73
from $1.00/M
Mixtral 8x7B Instruct
46.7B params · moe
Quality: 69
from $0.24/M
Mixtral 8x7B
46.7B params · moe
Quality: 67
from $0.50/M
Frequently Asked Questions
How much VRAM does Phi 3.5 MoE need for inference?
Phi 3.5 MoE requires approximately 83.8 GB of VRAM at BF16 precision, 41.9 GB at FP8, or 20.9 GB at INT4 quantization. Additional VRAM is needed for KV-cache (131072 bytes per token) and activations (~1.00 GB).
What is the best GPU for Phi 3.5 MoE?
The top recommended GPU for Phi 3.5 MoE is the B200 SXM using FP8 precision. It achieves approximately 1.1K tokens/sec at an estimated cost of $4261/month ($1.54/M tokens). Score: 100/100.
How much does Phi 3.5 MoE inference cost?
Phi 3.5 MoE API inference starts from $0.10/M input tokens and $0.30/M output tokens. Self-hosted inference costs depend on your GPU configuration — use our ROI calculator for a detailed breakdown.