Updated minutes ago
Salesforce

CodeGen2 16B

Salesforce · dense · 16B parameters · 2,048 context

Quality
50.0
Calculate ROI →Compare with othersFine-Tune This Model →

Parameters

16B

Context Window

2K tokens

Architecture

Dense

Best GPU

H100 SXM

Intelligence Brief

CodeGen2 16B is a 16B parameter DENSE model from Salesforce, featuring Multi-Head Attention (MHA) with 34 layers and 6,144 hidden dimensions. With a 2,048 token context window, it supports code. For self-hosted inference, H100 SXM delivers optimal throughput at $1794/month.

Architecture Details

TypeDENSE
Total Parameters16B
Active Parameters16B
Layers34
Hidden Dimension6,144
Attention Heads48
KV Heads48
Head Dimension128
Vocab Size51,200

Memory Requirements

BF16 Weights

32.0 GB

FP8 Weights

16.0 GB

INT4 Weights

8.0 GB

KV-Cache per Token417792 bytes
Activation Estimate1.20 GB

Fits on (single GPU) — most practical first

BEST FITRTX 308010GBINT4RTX 4070 Ti12GBINT4RTX 4070 Super12GBINT4RTX 306012GBINT4A400016GBINT4RTX 408016GBINT4T416GBINT4V100 16GB16GBINT4RTX 4060 Ti 16GB16GBINT4A216GBINT4TPU v5e16GBINT4A10G24GBFP8

GPU Compatibility Matrix

CodeGen2 16B is compatible with 76% of GPU configurations across 41 GPUs at 3 precision levels.

BF16 (Full)
FP8 (Half)
INT4 (Quarter)
Blackwell(7 GPUs)
B200 NVL (pair)360GB
91%
96%
98%
B300288GB
89%
94%
97%
B100 SXM192GB
83%
92%
96%
GB200 NVL72 (per GPU)192GB
83%
92%
96%
Hopper(7 GPUs)
H100 NVL 94GB (per GPU pair)188GB
83%
91%
96%
H200 SXM141GB
77%
89%
94%
H2096GB
67%
83%
92%
GH20096GB
67%
83%
92%
Ada Lovelace(11 GPUs)
L40S48GB
33%
67%
83%
L4048GB
33%
67%
83%
RTX 6000 Ada48GB
33%
67%
83%
L2048GB
33%
67%
83%
Ampere(16 GPUs)
A100 80GB SXM80GB
60%
80%
90%
A100 80GB PCIe80GB
60%
80%
90%
A1664GB
50%
75%
88%
RTX A600048GB
33%
67%
83%
Legend:No fitVery tightTightModerateGoodExcellent

GPU Recommendations

H100 SXMoptimal

FP8 · 1 GPU · tensorrt-llm

100/100

score

Throughput

1.1K tok/s

Latency (ITL)

1.0ms

Est. TTFT

0ms

Cost/Month

$1794

Cost/M Tokens

$0.65

Use this config →
H100 PCIeoptimal

FP8 · 1 GPU · tensorrt-llm

95/100

score

Throughput

1.0K tok/s

Latency (ITL)

1.0ms

Est. TTFT

0ms

Cost/Month

$1794

Cost/M Tokens

$0.65

Use this config →
RTX A6000optimal

BF16 · 1 GPU · vllm

95/100

score

Throughput

129.6 tok/s

Latency (ITL)

7.7ms

Est. TTFT

1ms

Cost/Month

$465

Cost/M Tokens

$1.37

Use this config →

Deployment Options

API

API Deployment

No API pricing available

Self-Hosted

Single GPU

H100 SXM

$1794/mo

Min VRAM: 16 GB

Scale

Multi-GPU

A10G x2

169.4 tok/s

TP· $569/mo

Open Full Calculator →

API Pricing Comparison

No API pricing data available for this model.

Performance Estimates

Throughput by GPU

H100 SXM
1.1K tok/s
H100 PCIe
1.0K tok/s
RTX A6000
129.6 tok/s

VRAM Breakdown (H100 SXM, FP8)

Weights
KV
Act
Weights 16.0 GBKV-Cache 6.8 GBActivations 9.6 GBOverhead 0.8 GB

Precision Impact

bf16

32.0 GB

weights/GPU

fp8

16.0 GB

weights/GPU

~1.1K tok/s

int4

8.0 GB

weights/GPU

Capabilities

Features

✗ Tool Use✗ Vision✓ Code✗ Math✗ Reasoning✗ Multilingual✗ Structured Output

Supported Frameworks

vllmtgi

Supported Precisions

BF16 (default)FP8INT4

Where to Deploy CodeGen2 16B

Similar Models

DeepSeek V2 Lite

15.7B params · moe

Quality: 50

Larger contextCompare →

DeepSeek MoE 16B

16.4B params · moe

Quality: 50

Larger contextCompare →

OctoCoder 15B

15.5B params · dense

Quality: 50

Larger contextCompare →

StarCoder2 15B

15.5B params · dense

Quality: 42

from $0.30/M

Larger context, Lower qualityCompare →

Nemotron 15B

15B params · dense

Quality: 72

from $0.30/M

Larger context, Higher qualityCompare →

Frequently Asked Questions

How much VRAM does CodeGen2 16B need for inference?

CodeGen2 16B requires approximately 32.0 GB of VRAM at BF16 precision, 16.0 GB at FP8, or 8.0 GB at INT4 quantization. Additional VRAM is needed for KV-cache (417792 bytes per token) and activations (~1.20 GB).

What is the best GPU for CodeGen2 16B?

The top recommended GPU for CodeGen2 16B is the H100 SXM using FP8 precision. It achieves approximately 1.1K tokens/sec at an estimated cost of $1794/month ($0.65/M tokens). Score: 100/100.

How much does CodeGen2 16B inference cost?

CodeGen2 16B inference costs vary by provider and GPU setup. Use our calculator for detailed cost estimates across all providers.