GPU Inference Glossary

Automatically adjusting GPU count based on traffic demand.

Activation-aware Weight Quantization — protects salient weight channels from quantization error.

Number of requests processed simultaneously by the GPU.

Brain Float 16 — 16-bit floating point optimized for deep learning.

Minimum price per million tokens to cover infrastructure costs.

Dynamic batching where new requests join mid-batch without waiting.

Cost per million tokens — the unit economics of inference.

Running identical model replicas across GPUs, each processing different requests.

The memory-bandwidth-bound phase where tokens are generated one at a time.

Distributing MoE experts across GPUs so each GPU holds a subset of experts.

Memory-efficient attention algorithm — 2-4x speedup over naive attention.

Half-precision floating point — 16-bit format with 5 exponent bits and 10 mantissa bits.

8-bit floating point — 2x memory savings with minimal quality loss on Hopper+ GPUs.

Post-training quantization method using second-order information for weight rounding.

Grouped Query Attention — shares KV-heads across query heads to reduce memory.

High Bandwidth Memory — stacked DRAM used in data-center GPUs.

8-bit integer quantization — reduces model size 2x with calibration.

Combining multiple GPU operations into a single kernel launch to reduce overhead.

Key-Value cache storing attention state for previously generated tokens.

Profit margin — the percentage of revenue that is profit after costs.

Rate at which data can be read from GPU memory, measured in TB/s.

Mixture of Experts — sparse architecture that activates only a subset of parameters per token.

Multi-Query Attention — uses a single KV-head shared by all query heads.

NVIDIA's high-bandwidth GPU interconnect — 600-900 GB/s bidirectional.

NVIDIA switch chip enabling all-to-all NVLink between all GPUs in a node.

Virtual memory for KV-cache — eliminates memory fragmentation.

Time required to recoup the initial GPU infrastructure investment.

Peripheral Component Interconnect Express — standard bus connecting GPUs to the host CPU.

Splitting model layers across GPUs in a pipeline for sequential processing.

The compute-intensive phase where the prompt is processed in parallel.

Reusing KV-cache from shared prompt prefixes across requests.

Power Usage Effectiveness — ratio of total facility power to IT equipment power.

Reducing model precision (e.g., FP16 to INT4) to decrease memory and increase speed.

Return on Investment — percentage profit relative to infrastructure cost.

Performance model that predicts throughput based on compute and memory bandwidth limits.

Using a small draft model to generate candidate tokens verified by the large model.

Time Between Tokens — inter-token latency during generation (decode phase).

Total Cost of Ownership — all-in cost including hardware, power, networking, and labor.

Thermal Design Power — maximum sustained power draw of a GPU in watts.

Specialized GPU cores for matrix multiply-accumulate operations.

Splitting a model's layers across multiple GPUs for parallel computation.

NVIDIA's optimized inference library with aggressive kernel fusion and quantization.

Text Generation Inference — Hugging Face's production serving framework.

Throughput metric — how many tokens the system generates per second.

Time To First Token — latency from request to first generated token.

Percentage of time GPUs are actively processing requests.

High-throughput LLM serving engine with PagedAttention and continuous batching.

Video RAM — GPU memory used for model weights and KV-cache.