Simple LLM VRAM calculator for model inference

How LLM VRAM calculator works

The LLM Memory Calculator is a tool designed to estimate the memory requirements for deploying large language models on GPUs. It simplifies the process by allowing users to input the number of parameters in a model and select a precision format, such as FP32, FP16, or INT8. Based on these inputs, the calculator computes the memory required to store the model in GPU memory and perform inference.

For instance, a 70-billion parameter model in FP32 precision is estimated to need between 280 GB and 336 GB of VRAM for inference. The "From" value represents the memory required for the model's parameters alone, while the "To" value accounts for additional overhead needed for activations, CUDA kernels, and workspace buffers. This range ensures all components of GPU memory usage are included in the estimate.

Different precision formats determine how much memory is used to store the model weights. FP32 (32-bit floating point) is the most precise format but requires the most memory, making it less common for inference tasks. FP16 (16-bit floating point) is widely used for storing model weights during inference as it offers a good balance between precision and memory efficiency. INT8 (8-bit integer) is even more memory-efficient and is often used in scenarios where slight reductions in precision are acceptable. These precision options allow practitioners to tailor their memory usage based on the specific requirements of their application.

The calculator is useful because it does not require detailed knowledge of the transformer model's architecture. Instead, it uses the number of parameters and precision to provide a general estimate applicable across different architectures. This makes it an essential tool for researchers and engineers planning GPU workloads and avoiding out-of-memory errors.

To understand how the calculator estimates memory, it helps to break down the major contributors to GPU memory usage. The first and most significant factor is the model parameters themselves. These weights and biases require substantial memory. For example, a 70-billion parameter model in FP16 precision occupies 140 GB of memory for its parameters alone. Larger models, such as GPT-3 with 175 billion parameters, require even more space, making efficient memory management critical.

Another major factor is the memory needed for activations. During inference, as input data flows through the model, intermediate outputs - or activations - are generated and temporarily stored. While activations are smaller during inference compared to training, they still contribute significantly to memory usage, especially for deeper models with long input sequences.

CUDA kernels and temporary buffers also add to the total memory requirements. These are responsible for executing computations and managing intermediate data. Factors like batch size and sequence length amplify memory usage further by increasing the number of activations and computational demands per forward pass. Additionally, memory fragmentation arises as GPUs allocate and release memory blocks during computation, leaving gaps that cannot be effectively utilized. Extra memory is often needed to handle this fragmentation and ensure smooth operation.

The estimates provided by the calculator align with the practical guideline of reserving 1.2 times the model's memory size to account for overhead and fragmentation. For instance, GPT-3 with 175 billion parameters stored in FP16 precision requires 350 GB for the model and an additional 70 GB to meet the 1.2x requirement. This results in a total memory requirement of 420 GB, typically requiring a multi-GPU setup using high-end GPUs such as NVIDIA A100s.

Smaller models, such as LLaMA 2-13B with 13 billion parameters, need about 26 GB of memory in FP16 and approximately 31.2 GB with the 1.2x rule applied. This fits comfortably on GPUs like the NVIDIA RTX 6000 Ada, while consumer-grade GPUs such as the GeForce RTX 4090 may require lower precision to accommodate the model. However, a newer consumer NVIDIA GeForce RTX 5090 Blackwell GPU with 32 GB VRAM might be able to serve the model. Even smaller models like BERT-Large, with 340 million parameters, require approximately 816 MB of memory under the 1.2x rule, making them suitable for consumer-grade GPUs like the NVIDIA GeForce RTX 3060.

For users with limited GPU resources, the calculator highlights the importance of optimization techniques. Quantization, for example, reduces precision from FP16 to INT8, nearly halving memory usage. Offloading computations to the CPU, using model parallelism to split the model across multiple GPUs, and optimizing sequence lengths are other viable strategies. Libraries like FlashAttention further reduce memory usage by optimizing the computation of attention mechanisms.

By combining these techniques, users can maximize the efficiency of their hardware and deploy models effectively. The LLM Memory Calculator simplifies planning for these deployments, ensuring memory requirements are met and avoiding potential errors. For further insights into these principles, refer to the article on Transformer Math by EleutherAI.

Examples

How much VRAM for a 7B model at FP16?

~14 - 16.8 GB (range includes typical overhead)

How much VRAM for a 13B model at FP16?

~26 - 31.2 GB

How much VRAM for a 70B model at FP16?

~140 - 168 GB

How much VRAM for a 405B model at FP16?

~810 - 972 GB

Supported Precisions

FP32 (Single-Precision Floating-Point)

FP32 is the standard for general-purpose computing. It uses 32 bits and offers a great balance of range and precision, making it suitable for scientific simulations, high-fidelity graphics, and traditional machine learning.

BF16 (Bfloat16)

BF16 is a 16-bit format that keeps the same wide dynamic range as FP32 by using 8 bits for the exponent but reduces the mantissa to 7 bits. This is great for deep learning as it avoids overflow/underflow issues, allowing for faster computation and reduced memory usage.

FP16 (Half-Precision Floating-Point)

FP16 is another 16-bit format, but it uses 5 bits for the exponent and 10 bits for the mantissa. This gives it a smaller range than BF16 but higher precision, making it ideal for inference and certain training tasks where dynamic range isn't a primary concern. GPUs often have Tensor Cores that speed up FP16 operations.

FP8 (8-Bit Floating-Point)

FP8 is a format that further reduces memory and computational requirements. It comes in two main flavors: E4M3 (4 exponent bits, 3 mantissa bits) for smaller range and higher precision, and E5M2 (5 exponent bits, 2 mantissa bits) for a wider range and less precision. It's becoming crucial for large language model (LLM) inference.

INT8 (8-Bit Integer)

INT8 is a fixed-point format using 8 bits. It offers the highest performance and lowest memory footprint. It's mainly used for inference after a model has been trained and "quantized" from a higher precision like FP32 to INT8 to maximize throughput.

FP6 (6-Bit Floating-Point)

FP6 is an experimental 6-bit format that sits between FP4 and FP8. It's designed for ultra-low-precision scenarios, aiming to strike a balance between memory, speed, and accuracy for specific deep learning workloads.

FP4 (4-Bit Floating-Point)

FP4 is an extremely low-precision format. It offers the greatest potential for memory savings and speed-ups but with a significant reduction in accuracy. This is a topic of ongoing research, primarily for inference tasks where a small drop in accuracy is acceptable for a massive boost in performance and efficiency.

Last updated: August 12, 2025