vram-requirements

Calculating VRAM Requirements for vLLM Deployment

Example Model: Qwen/Qwen3-32B

This guide helps you estimate the GPU VRAM needed to deploy large language models using vLLM, using Qwen3-32B as a practical example.

Quick Guide

If you're looking for a quick back-of-the-envelope calculation, here's what you need to know:

Model Specifications to Look For

When viewing a model card on Hugging Face (e.g., Qwen/Qwen3-32B), look for:

1. Number of Parameters: Listed as "Number of Parameters" or model size
   • For Qwen3-32B: 32.8B parameters
2. Context Length: Listed as "Context Length" or "Max Position Embeddings"
   • For Qwen3-32B: 32,768 tokens (native), up to 131,072 with RoPE scaling

Simple Calculation

1. Model Weights

Rule of Thumb:

VRAM for weights (GB) ≈ Number of Parameters (B) × Bytes per Parameter

Common precisions:

• FP16/BF16: 2 bytes per parameter → 32.8B × 2 = 65.6 GB
  • Best quality, full precision inference
  • Official model: Qwen/Qwen3-32B
• FP8: 1 byte per parameter → 32.8B × 1 = 32.8 GB
  • Minimal quality loss, 50% memory savings
  • Official model: Qwen/Qwen3-32B-FP8
• AWQ 4-bit: 0.5 bytes per parameter → 32.8B × 0.5 = 16.4 GB
  • Moderate quality loss, 75% memory savings
  • Official model: Qwen/Qwen3-32B-AWQ

💡 Tip: Start with official quantizations (Qwen3-32B-FP8 or Qwen3-32B-AWQ) which are tested and optimized by Qwen. Avoid GGUF format—it's not compatible with vLLM.

For Qwen3-32B:

• BF16 (unquantized): ~66 GB (best quality)
• FP8: ~33 GB (official quantization available)
• AWQ (4-bit): ~16 GB (official quantization available)

Note on Quantization: Lower precision quantization reduces memory usage but may impact model performance. The extent of quality loss depends on the quantization method and your specific use case. Always test quantized models with your workload before production deployment.

⚠️ Important for vLLM: Use quantization formats compatible with vLLM such as FP8, AWQ, or GPTQ. Avoid GGUF format as it is not supported by vLLM (GGUF is designed for llama.cpp). Qwen provides official FP8 and AWQ quantizations that are thoroughly tested and recommended for production use.

2. KV Cache

The KV cache stores computed key-value pairs for the attention mechanism. It grows with:

• Batch size (number of concurrent requests)
• Sequence length (input + output tokens)

Note on Batch Sizing: vLLM handles batch sizing dynamically under the hood using its PagedAttention algorithm. You typically don't need to manually configure batch sizes—vLLM will automatically batch concurrent requests to maximize throughput within your available KV cache memory. The max_num_seqs parameter sets an upper limit, but vLLM will dynamically adjust based on available resources.

Simplified Formula:

KV Cache (GB) ≈ Batch Size × Context Length × 0.0005 GB per token

Example scenarios for Qwen3-32B (BF16 precision):

Scenario	Batch Size	Context Length	KV Cache Size
Single request, short	1	4,096	~2 GB
Single request, full	1	32,768	~16 GB
Batch inference, short	8	4,096	~16 GB
Batch inference, medium	8	16,384	~66 GB

3. Total VRAM Estimate

Formula:

Total VRAM = Model Weights + KV Cache + Overhead (15-20%)

Example for Qwen3-32B:

• Scenario: BF16 weights, batch size 4, context 8,192 tokens
• Model weights: 66 GB
• KV cache: 4 × 8,192 × 0.0005 ≈ 16 GB
• Overhead (20%): 16 GB
• Total: ~98 GB (requires 2× H100 80GB or 2× A100 80GB with tensor parallelism)

Practical Recommendations:

• For single GPU (e.g., A100 80GB): Use AWQ 4-bit quantization + limit context/batch size
• For 2 GPUs (e.g., 2× A100 80GB): Use FP8 or BF16 with tensor parallelism
• For 4+ GPUs: Use BF16 for best quality with full context length

The recommendations lean towards quantized models if one were to deploy additional models or require high throughput.

Advanced Guide

⚠️ Important Note on Calculators: Many online VRAM calculators and GPU memory utilization UIs provide rough estimates that may be significantly inaccurate for your specific deployment. This is because:

• No universal formula exists: Each model architecture (MHA vs GQA vs MoE) has different memory characteristics
• Framework differences: vLLM, TensorRT-LLM, HuggingFace TGI, and others have different memory management strategies
• Real-world factors: Dynamic batching, prefix caching, memory fragmentation, and CUDA overhead vary significantly
• Deployment-specific variables: Your actual batch size, sequence length distribution, and concurrency patterns differ from calculator assumptions

Best practice: Use this guide's formulas as a starting point, then validate with actual deployment testing. Errors such as CUDA out of memory can be found under Logs from our portal.

Detailed Model Architecture Analysis

To accurately calculate VRAM, you need these specifications from the model card or config.json:

Where to Find Model Specifications

1. Hugging Face Model Card (e.g., Qwen/Qwen3-32B):

   • Check the "Model Details" or "Model Overview" section
   • Look for parameter count, architecture type, and context length

2. config.json file:

   • Click "Files and versions" tab on Hugging Face
   • Open config.json to see complete architecture details
   • This is the authoritative source for technical specifications

3. Official Quantization Availability:

   • Search for model variants on Hugging Face (e.g., -FP8, -AWQ, -GPTQ)
   • Check the model author's organization page for official quantizations
   • For Qwen models: Look under the Qwen organization

Qwen3-32B Architecture

From Hugging Face model card:

{
  "num_parameters": "32.8B",
  "num_parameters_non_embedding": "31.2B",
  "num_layers": 64,
  "num_attention_heads": 64,       // Query heads
  "num_key_value_heads": 8,        // KV heads (GQA)
  "hidden_size": 5120,
  "intermediate_size": 13824,
  "max_position_embeddings": 32768,
  "vocab_size": 151936
}

Key observations:

• Uses Grouped-Query Attention (GQA): 64 query heads, 8 KV heads
• Head dimension: d_head = hidden_size / num_attention_heads = 5120 / 64 = 80
• GQA reduces KV cache by factor of 8× compared to Multi-Head Attention (MHA)

Precise VRAM Calculations

1. Model Weights Memory

Formula:

Memory_weights (bytes) = num_parameters × bytes_per_parameter

Precision options:

Precision	Bytes/Param	Qwen3-32B Memory	Official Model Available
FP32	4	131.2 GB	No (not recommended)
BF16/FP16	2	65.6 GB	✅ Yes (base model)
FP8	1	32.8 GB	✅ Yes (Qwen3-32B-FP8)
AWQ (4-bit)	0.5	16.4 GB	✅ Yes (Qwen3-32B-AWQ)
GPTQ (4-bit)	0.5	16.4 GB	Community versions available

Recommendation: Use official Qwen quantizations when available. They are professionally calibrated and tested. For other models, prefer quantizations from well-known providers like TheBloke, bartowski, or the original model authors.

2. KV Cache Memory (Detailed)

The KV cache is the dominant memory consumer during inference. For Grouped-Query Attention models:

Per-token KV cache formula:

KV_cache_per_token (bytes) = 2 × num_layers × num_kv_heads × head_dim × bytes_per_parameter

Total KV cache formula:

KV_cache_total (bytes) = batch_size × sequence_length × KV_cache_per_token

For Qwen3-32B (BF16 precision):

KV_cache_per_token = 2 × 64 layers × 8 KV_heads × 80 head_dim × 2 bytes
                   = 2 × 64 × 8 × 80 × 2
                   = 163,840 bytes
                   ≈ 0.164 MB per token
                   ≈ 0.00016 GB per token

Comparison with MHA: If Qwen3-32B used MHA (64 KV heads instead of 8):

KV_cache_per_token_MHA = 2 × 64 × 64 × 80 × 2 = 1,310,720 bytes ≈ 1.31 MB per token

GQA provides 8x reduction in KV cache!

3. KV Cache Examples for Qwen3-32B

vLLM's Dynamic Batching: The batch sizes shown below are for illustration purposes. In practice, vLLM automatically manages batching through its continuous batching algorithm with PagedAttention. You set max_num_seqs as an upper bound, and vLLM dynamically batches concurrent requests to maximize GPU utilization while staying within memory limits. The actual batch size varies moment-to-moment based on incoming requests and their sequence lengths.

Formula application:

KV_cache (GB) = batch_size × seq_length × 0.00016 GB/token

Batch Size	Sequence Length	KV Cache (GB)	Use Case
1	2,048	0.33	Single short chat
1	8,192	1.31	Single medium document
1	32,768	5.24	Single full context
1	131,072	20.97	Extended context (YaRN)
4	8,192	5.24	Batch processing
8	8,192	10.49	High throughput
16	4,096	10.49	API serving
32	2,048	10.49	Maximum throughput
8	32,768	41.94	Batch long context

4. Additional Memory Components

vLLM requires additional memory for:

a. Activation Memory:

Activation_memory ≈ 5-10% of model weights

For Qwen3-32B (BF16): ~3-7 GB

b. CUDA Context & Framework Overhead:

Overhead ≈ 1-2 GB base + 10-15% of (weights + KV cache)

c. Temporary Buffers:

• Depends on max_num_batched_tokens parameter
• Typically: 2-5 GB

Complete Memory Breakdown

Total VRAM formula:

Total_VRAM = Model_Weights + KV_Cache + Activations + Overhead

Where:

• Activations ≈ 0.07 × Model_Weights
• Overhead ≈ 2 GB + 0.12 × (Model_Weights + KV_Cache)

Example Calculation: Production Deployment

Scenario:

• Model: Qwen3-32B in BF16
• Batch size: 8
• Average sequence length: 8,192 tokens
• GPU: 2× NVIDIA A100 80GB (tensor parallelism)

Step 1: Model Weights

Weights = 65.6 GB (BF16)
Weights_per_GPU = 65.6 / 2 = 32.8 GB per GPU

Step 2: KV Cache

KV_cache = 8 × 8,192 × 0.00016 = 10.49 GB
KV_cache_per_GPU = 10.49 / 2 = 5.25 GB per GPU

Step 3: Activations

Activations = 0.07 × 32.8 = 2.3 GB per GPU

Step 4: Overhead

Overhead = 2 + 0.12 × (32.8 + 5.25) = 6.57 GB per GPU

Total per GPU:

Total = 32.8 + 5.25 + 2.3 + 6.57 = 46.92 GB per GPU

Result: Fits comfortably on 2× A100 80GB with ~33 GB headroom per GPU for dynamic batching.

Memory Optimization Strategies

1. Quantization

Quantization reduces the precision of model weights to decrease memory usage. Always test quantized models with your specific workload as performance impacts vary by use case.

Official Qwen3-32B Quantizations

FP8 Quantization (✅ Recommended):

• Memory: 32.8 GB (50% reduction from BF16)

• Model: Qwen/Qwen3-32B-FP8

• Quality: Minimal loss (~1-2% on most benchmarks)

• vLLM compatibility: Excellent, native support

• Usage:

  # Using official FP8 model
  vllm serve Qwen/Qwen3-32B-FP8 --dtype auto
  
  # Or quantize on-the-fly
  vllm serve Qwen/Qwen3-32B --quantization fp8

AWQ 4-bit Quantization:

• Memory: ~16 GB (75% reduction from BF16)

• Model: Qwen/Qwen3-32B-AWQ

• Quality: Moderate loss (~3-5% on complex reasoning tasks)

• vLLM compatibility: Excellent, native support

• Best for: Single GPU deployments with memory constraints

• Usage:

  vllm serve Qwen/Qwen3-32B-AWQ --quantization awq

Alternative Quantization Providers

If official quantizations aren't available for your model, use quantizations from reputable providers:

• TheBloke/turboderp: Well-known for GPTQ quantizations
• bartowski: Provides various quantization formats
• NousResearch: High-quality quantizations for research models

vLLM Quantization Compatibility

Format	vLLM Support	Use Case	Notes
FP8	✅ Native	Production	Best quality/size trade-off
AWQ	✅ Native	Single GPU	Good for 4-bit
GPTQ	✅ Native	Single GPU	Alternative 4-bit method
INT8	⚠️ Limited	Specialized	Via quantization libraries
GGUF	❌ Not supported	N/A	Use llama.cpp instead
GGML	❌ Not supported	N/A	Legacy format

⚠️ Critical: GGUF and GGML formats are designed for llama.cpp and are not compatible with vLLM. If you see .gguf files on Hugging Face, they cannot be used with vLLM. Look for SafeTensors (.safetensors) format models instead.

Quantization Performance Impact

Expected quality degradation:

• FP8: 0-2% loss on most tasks
• AWQ/GPTQ (4-bit): 3-8% loss, more on complex reasoning
• Lower bits: Generally not recommended for production

Always validate quantized models by:

1. Running your specific evaluation benchmarks
2. Testing on representative production queries
3. Monitoring user feedback after deployment

2. KV Cache Quantization

FP8 KV Cache:

--kv-cache-dtype=fp8

• Reduces KV cache by 50%
• New KV cache per token: 0.00008 GB
• Minimal quality impact on most tasks

3. Context Length Management

Using max_model_len:

--max-model-len=16384

• Reduces maximum KV cache allocation
• Increases available memory for larger batches

4. GPU Memory Utilization

Adjusting gpu_memory_utilization:

--gpu-memory-utilization=0.95

• Default: 0.90 (90% of GPU memory)
• Increase to 0.95 for more KV cache space
• Risk: May cause OOM if set too high

5. Tensor Parallelism

Sharding across GPUs:

--tensor-parallel-size=2

Tensor parallelism (TP) is vLLM's primary method for distributing large models across multiple GPUs within a single node. Here's how it works:

What gets distributed:

• Model weights: Split across GPUs (each GPU holds 1/N of the weights)
• Computations: Matrix multiplications are performed in parallel across GPUs
• KV cache: Also distributed across GPUs, reducing per-GPU memory pressure

How it works:

• Each layer's weight matrices are split along specific dimensions
• For attention layers: Query, Key, Value projections are split across heads
• For feed-forward layers: The large intermediate matrices are split
• GPUs synchronize results using high-speed interconnects (NVLink/NVSwitch)

Memory benefits:

Memory_per_GPU = (Total_Weights / TP_size) + (KV_Cache / TP_size) + Overhead

Example for Qwen3-32B with TP=2:

• Each GPU holds: 65.6 GB / 2 = 32.8 GB of weights
• KV cache is also split by 2
• Communication overhead: ~2-5 GB per GPU

When to use Tensor Parallelism:

• Model weights don't fit on a single GPU
• You need larger batch sizes or context lengths than single GPU allows
• You have multiple GPUs in the same node (PCIe or NVLink connected)

Considerations:

• Adds latency due to inter-GPU communication (minimal with NVLink)
• Most efficient with 2, 4, or 8 GPUs (powers of 2)
• Limited to GPUs in the same physical node
• Each GPU must be the same model and have identical VRAM

Note: For multi-node deployments, you'll need pipeline parallelism or data parallelism, which vLLM also supports but is beyond the scope of this guide.

Key Formulas Summary

Model Weights:

Memory_weights (GB) = num_parameters × bytes_per_param / 1e9

KV Cache (GQA):

Per_token (bytes) = 2 × L × H_kv × D_h × P
Total_KV (GB) = B × T × Per_token / 1e9

Where:

• L = num_layers (64)
• H_kv = num_kv_heads (8)
• D_h = head_dim (80)
• P = precision in bytes (2 for BF16)
• B = batch_size
• T = sequence_length

Total VRAM:

Total = Weights + KV_Cache + (0.07 × Weights) + 2 + (0.12 × (Weights + KV_Cache))

Practical Decision Matrix

GPU Setup	Precision	Model Variant	Max Batch	Max Context	Best For
1× A100 40GB	AWQ 4-bit	Qwen3-32B-AWQ	4-8	8K	Development/testing
1× A100 80GB	AWQ 4-bit	Qwen3-32B-AWQ	8-16	16K	Cost-optimized production
1× A100 80GB	FP8	Qwen3-32B-FP8	4-8	16K	Balanced single GPU
2× A100 80GB	FP8	Qwen3-32B-FP8	16-32	32K	High-quality production
2× A100 80GB	BF16	Qwen3-32B	8-16	32K	Maximum quality
4× A100 80GB	BF16	Qwen3-32B	32+	32K	High throughput
8× H100 80GB	BF16	Qwen3-32B	64+	131K	Enterprise/extended context

Quantization Selection Guide:

• BF16: Best quality, use when you have sufficient GPU memory and need maximum accuracy
• FP8: Recommended for most production deployments—excellent quality/memory trade-off
• AWQ 4-bit: Best for single GPU deployments with memory constraints, acceptable quality loss for most tasks
• Avoid GGUF: Not compatible with vLLM; use SafeTensors format models only

References and Further Reading

1. vLLM Documentation: https://docs.vllm.ai
2. Qwen3 Model Card: https://huggingface.co/Qwen/Qwen3-32B
3. KV Cache Deep Dive: "LLM Inference Series: 4. KV caching, a deeper look" by Pierre Lienhart
4. PagedAttention Paper: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al., 2023)
5. GQA Paper: "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (Ainslie et al., 2023)