Benchmark Management
Admin
A
Benchmark Management
Submit, review, and manage community benchmark datasets.
Total Datasets
146
Total Datapoints
4053
Models Covered
73
GPUs Covered
10
Submit Benchmark Results
Contribute your benchmark data for model + GPU combinations.
| ISL | OSL | Concurrency | Throughput (t/s) | TTFT (ms) | |
|---|---|---|---|---|---|
Trust Score: 45/100
Model ID is required
GPU ID is required
Framework is required
Precision is required
Invalid throughput: 0
Existing Benchmark Datasets
| Model | GPU | GPUs/Node | Nodes | Framework | Precision | Datapoints | Run Date |
|---|---|---|---|---|---|---|---|
| snowflake/arctic-480b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| baai/bge-large-en-v1.5 | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/codellama-13b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/codellama-34b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/codellama-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| cohere/command-r-plus | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| deepseek/deepseek-coder-v2-236b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| deepseek/deepseek-r1 | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-02-20 |
| deepseek/deepseek-r1 | nvidia-h200-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| deepseek/deepseek-v3 | nvidia-a100-80gb-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| deepseek/deepseek-v3 | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-02-15 |
| deepseek-ai/deepseek-v3 | nvidia-h200 | 8 | 1 | vllm | bf16 | 18 | 2025-02-10 |
| intfloat/e5-mistral-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| falcon/falcon-180b | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| falcon/falcon-40b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| falcon/falcon-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-2-27b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-2-27b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-28 |
| google/gemma-2-27b | nvidia-h200-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-2-2b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-2-9b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-2-9b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-3-12b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-3-1b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| google/gemma-3-4b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| falcon/falcon-7b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| falcon/falcon-7b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| google/gemma-1.1-2b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| google/gemma-1.1-2b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| google/gemma-2-2b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| google/gemma-2-2b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| google/gemma-2-9b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| sensetime/internlm-20b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| meta-llama/llama-2-7b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| meta-llama/llama-2-7b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| meta-llama/llama-3-8b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| meta-llama/llama-3.1-8b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| mistral/mistral-7b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| mistral/mistral-7b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| mistral/mistral-7b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| microsoft/phi-1.5 | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| microsoft/phi-1.5 | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| microsoft/phi-1.5 | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| microsoft/phi-3-mini-3.8b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| microsoft/phi-3-mini-3.8b | nvidia-t4 | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| alibaba/qwen-1.5-moe-a2.7b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-0.5b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-0.5b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-0.5b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-1.5b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-1.5b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-1.5b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| alibaba/qwen-2.5-14b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| alibaba/qwen-2.5-32b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-7b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-7b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-2.5-7b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-3-4b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| qwen/qwen-3-4b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| google/recurrentgemma-2b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| google/recurrentgemma-2b | nvidia-t4 | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| stabilityai/stablelm-2-12b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| stabilityai/stablelm-zephyr-3b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| stabilityai/stablelm-zephyr-3b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| 01-ai/yi-1.5-34b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| 01-ai/yi-1.5-9b | nvidia-a10g | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| 01-ai/yi-1.5-9b | nvidia-a100-80gb-sxm | 1 | 1 | pytorch | bf16 | 48 | 2024-12-13 |
| 01-ai/yi-1.5-9b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| huggingface/zephyr-7b | nvidia-a10g | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| huggingface/zephyr-7b | nvidia-t4 | 1 | 1 | pytorch | fp16 | 48 | 2024-12-13 |
| internlm/internlm2.5-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| ai21/jamba-1.5-large | nvidia-h100-sxm | 8 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| kimi-k2.5 | nvidia-b200 | 8 | 1 | vllm | bf16 | 90 | 2025-03-01 |
| meta-llama/llama-3.1-405b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 19 | 2025-01-20 |
| meta-llama/llama-3.1-405b | nvidia-h200-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| meta-llama/llama-3.1-70b | nvidia-a100-80gb-sxm | 2 | 1 | vllm | bf16 | 19 | 2025-01-22 |
| meta-llama/llama-3.1-70b | nvidia-h100-sxm | 8 | 1 | vllm | bf16 | 18 | 2025-01-15 |
| meta-llama/llama-3.1-70b | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 19 | 2025-02-10 |
| meta-llama/llama-3.1-70b | nvidia-h200-sxm | 4 | 1 | vllm | bf16 | 19 | 2025-02-18 |
| meta-llama/llama-3.1-70b | nvidia-l40s | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.1-8b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-22 |
| meta-llama/llama-3.1-8b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| meta-llama/llama-3.1-8b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-20 |
| meta-llama/llama-3.1-8b | nvidia-h200-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-02-18 |
| meta-llama/llama-3.1-8b | nvidia-l40s | 1 | 1 | vllm | bf16 | 15 | 2025-02-05 |
| meta-llama/llama-3.1-8b | nvidia-rtx-3090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.1-8b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 17 | 2025-01-25 |
| meta-llama/llama-3.2-11b-vision | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.2-1b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.2-1b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.2-1b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.2-3b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.2-90b-vision | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.3-70b | nvidia-a100-80gb-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-3.3-70b | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 19 | 2025-02-12 |
| meta-llama/llama-3.3-70b | nvidia-h200-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| meta-llama/llama-4-maverick-400b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| meta-llama/llama-4-scout-17b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| nvidia/minitron-8b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mistral-7b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-20 |
| mistral/mistral-7b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| mistral/mistral-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-18 |
| mistral/mistral-7b | nvidia-h200-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mistral-7b | nvidia-l40s | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mistral-7b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mistral-large | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mistral-nemo-12b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-01-30 |
| mistral/mistral-small-24b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mixtral-8x22b | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mixtral-8x7b | nvidia-a100-80gb-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| mistral/mixtral-8x7b | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 19 | 2025-01-25 |
| nvidia/nemotron-15b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| nvidia/nemotron-340b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| nvidia/nemotron-70b | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-2 | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-2 | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-3-mini-3.8b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-4 | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-4 | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-02-01 |
| microsoft/phi-4 | nvidia-h200-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-4 | nvidia-l40s | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| microsoft/phi-4 | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-14b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-32b | nvidia-a100-80gb-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-32b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-3b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-3b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| alibaba/qwen-2.5-3b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-72b | nvidia-a100-80gb-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-72b | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 19 | 2025-02-05 |
| qwen/qwen-2.5-72b | nvidia-h200-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-7b | nvidia-a100-80gb-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-02-08 |
| qwen/qwen-2.5-7b | nvidia-h200-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-7b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-2.5-vl-72b | nvidia-h100-sxm | 4 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-3-235b | nvidia-h100-sxm | 8 | 1 | vllm | fp8 | 18 | 2025-03-15 |
| qwen/qwen-3-4b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| qwen/qwen-3-8b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| bigcode/starcoder2-15b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| bigcode/starcoder2-7b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| tinyllama/tinyllama-1.1b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| tinyllama/tinyllama-1.1b | nvidia-rtx-4090 | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| nvidia/vila-1.5-40b | nvidia-h100-sxm | 2 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| 01-ai/yi-1.5-34b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |
| 01-ai/yi-1.5-9b | nvidia-h100-sxm | 1 | 1 | vllm | bf16 | 18 | 2025-03-15 |