How the numbers are made

Every estimate in the calculator comes from one plain TypeScript model, engine.ts, which runs in your browser. This page writes out its formulas, the constants they use and the published measurements the speed constants are fitted to.

Overview

Memory is counted part by part: weights, KV cache, activations, runtime overhead and, when used, a speculative drafter. Speed follows from the hardware: generating a token means reading the weights and the KV cache from memory, so decode is limited by memory bandwidth; reading a prompt is a large matrix multiplication, so prompt processing is limited by compute. Fixed costs per step and per layer (kernel launches, scheduling) are added on top.

How close a runtime gets to the hardware's peak depends on the runtime and the GPU, so those efficiencies are fitted to 123 published measurements. The results are estimates. Drivers, kernels, settings and the prompts you send all move real numbers.

Model shape

Each model is described by its Hugging Face config.json: layers L, hidden size h, attention heads n_h, KV heads n_kv, head size d_h, feed-forward size d_ff, vocabulary V, and the parameter count P from the model's published weights. Hybrid models also give the number of sliding-window layers (window w), linear-attention or state-space layers, and Multi-head Latent Attention (MLA) models give the size of their compressed KV latent.

For a mixture-of-experts (MoE) model with E experts and k active per token, the parameters used per token P_active come from the model card when published, and otherwise from:

Memory

Weights

Weights take their bits per weight b. GGUF figures are whole-file averages, which already include the embeddings. Other quantized formats keep the embeddings and output head (P_emb = V·h, twice when they aren't tied) in 16-bit:

KV cache

Each sequence of s tokens stores keys and values in every attention layer:

c

values per token per layer: 2 × n_kv × d_h, or the latent size for MLA (576 by default)

β

bytes per value: 2 for FP16/BF16, 1 for FP8 and INT8, 0.5 for INT4

S

the fixed recurrent state of a linear-attention or state-space layer, whatever the length

One copy of the model serves n_seq = ⌈users / copies⌉ × batch sequences at once. With prefix caching, the shared share f of the prompt is stored once: KV(s) × (n_seq × (1 − f) + f). Tensor parallelism over tp GPUs splits the KV heads, so each GPU keeps ⌈n_kv / tp⌉ / n_kv of the cache (MLA latents are copied to every GPU). Pipeline parallelism over pp stages divides it by pp.

Activations and runtime overhead

Runtimes process prompts in chunks of C tokens (8,192 for vLLM, 512 for llama.cpp), and keep the logits of every running sequence:

Without fused (flash) attention, llama.cpp, Ollama and Transformers materialize the score matrix of a chunk in FP32, C × s × n_h × 4 bytes. Each runtime also holds a fixed amount for its CUDA context, graphs and buffers, plus 0.5 GB when a copy spans several GPUs.

Speculative decoding drafters

A draft model adds its parameters in BF16. Learned drafters are sized from the target model, with one transformer layer counted as 12h² parameters: an EAGLE-3 head is one layer plus a 32K-token output head (12h² + 32,000h), a DFlash drafter about five layers, and multi-token prediction (MTP) one extra layer at the model's own precision. N-gram and suffix lookup add nothing.

Usable memory and verdict

Per GPU, the total is W × (1 − o) / (tp × pp) plus the KV cache, activations, scores, overhead and drafter, where o is the share of weights offloaded to CPU memory (offloading the KV cache keeps 30% of it on the GPU). It is compared with the memory the runtime lets itself use. Apple Silicon can only wire about 75% of unified memory to the GPU, two thirds on machines with 36 GB or less.

Below 50% of usable memory the setup is comfortable, then moderate up to 75%, high up to 90%, very high up to 100%, and it doesn't fit beyond. The number of users that fit is the memory left after everything but the KV cache, divided by the KV cache of one user, times the number of copies.

Decode speed

Each decode step produces one token for each of the b sequences in flight. Its time is the larger of reading memory and computing, plus communication and fixed costs:

B

the GPU's memory bandwidth from its spec sheet

η

the share of it the runtime achieves (fitted, see Calibration)

F

dense FP16/BF16 tensor throughput with FP32 accumulate

t_step, t_layer

fixed cost per step and per layer (launches, scheduling); MoE layers add routing and small expert matmuls

W_read is the weights streamed per step. The input embedding table is only looked up, so it isn't streamed when it is separate from the output head. An MoE model always reads its shared weights, plus the experts the batch touches. With b tokens each picking k of E experts, the expected number touched is:

Weights and KV cache offloaded to the CPU are read at 60 GB/s (dual-channel DDR5), which adds t_cpu.

Several GPUs

Tensor parallelism splits every layer across tp GPUs, which then all-reduce activations twice per layer. An all-reduce of x bytes over n GPUs costs a latency term plus the ring algorithm's traffic:

So t_tp = 2L × AR(2bh, tp). Pipeline parallelism passes activations between pp stages: t_pp = (pp − 1) × AR(2bh, 2). Tensor parallelism that spans machines uses the network between them.

Several copies of a model (replicas) each get tp × pp GPUs and split the users between them. Per-user speed is 1 / t(b); total throughput is that speed times the sequences in flight times the copies. Requests per second assume 512 output tokens per request.

Speculative decoding

A drafter proposes k tokens and the model checks them in one step over k + 1 tokens per sequence, so that step is t(b) with b(k + 1) tokens: more compute and, for MoE models, more experts touched. With acceptance rate a per drafted token, a step yields on average (1 − a^k+1) / (1 − a) tokens (Leviathan et al.). Drafting takes its own time: each pass reads the drafter and runs its layers, t_draft = passes × (bytes read / (tp × B × η) + layers × t_layer). Draft models, EAGLE-3 heads and MTP make one pass per drafted token, DFlash one pass for the whole block; MTP and DFlash also run the model's output head, which is large for big vocabularies. N-gram and suffix lookup take no GPU time. Verifying costs about 3% more per drafted token (rejection sampling, KV writes):

Measured gains shrink with the batch size faster than this, as kernels don't overlap compute and memory perfectly, so the speedup used is 1 + (g − 1) / (1 + b / 8). vLLM reports an acceptance length (tokens per step); the calculator's acceptance rate is the a that gives the same length. The rates it suggests (from 0.4 for n-gram lookup to 0.85 for MTP) are typical values; yours depend on the model and the prompts.

Prompt processing

Reading an s-token prompt costs 2P_active FLOPs per token plus attention, which grows with the square of the length:

Time to first token is the larger of that compute and re-reading the weights once per chunk, plus communication and a fixed cost:

φ

1.6 for FP8 weights on GPUs with FP8 tensor cores, 2.4 for NVFP4 and MXFP4 on FP4 GPUs, otherwise 1

μ

the share of peak compute the runtime achieves on prompts (fitted)

t₀

scheduling, tokenization and sampling: 15 ms unless fitted

Calibration

The efficiencies η, μ, t_layer and t₀ start from typical values per runtime. Where published measurements exist, they are fitted to them by scripts/fit-speed.ts. The fit minimizes the squared log error, so being 2× too fast costs as much as being 2× too slow. It runs first over all GPUs of a runtime, then per GPU class (vendor plus consumer, workstation or datacenter), but only for a class with more GPUs than constants to fit. That matters: on prompts, llama.cpp reaches about 1.2× the catalog's FP16 figure on GeForce cards (its integer kernels beat the half-rate figure GeForce cards are rated at), about 0.5× on NVIDIA workstation cards and 0.15× to 0.3× on datacenter GPUs. Ollama runs llama.cpp and uses its constants.

Empty cells use the runtime's All GPUs line. Runtimes without measurements here (SGLang, Transformers) keep the starting values:

Accuracy

Typical error is exp(mean |log(predicted / measured)|) − 1, so 10% means predictions are usually within about 10%. The figures below are on the measurements used for the fit. On a GPU left out of the fit, which is what a GPU without measurements gets, decode speed is typically within 10% and llama.cpp prompt processing within 25%.

Known gaps: llama.cpp prompt processing on datacenter GPUs varies more than one constant per class can follow (the H100 is predicted about 2× too fast, the L40 about 2.4× too slow), very large MoE models are on the optimistic side, and speeds for SGLang and Transformers are not checked against measurements yet.

Measurements from more hardware make the fit better. The benchmark script measures a running vLLM server or a GGUF file with llama-bench and writes a report to paste into a benchmark issue.

Fine-tuning

Memory

Full fine-tuning keeps the weights at the training precision (2 bytes for BF16, 4 for FP32, 1 for FP8), gradients of at least 2 bytes, an FP32 master copy (4 bytes) in mixed precision, and Adam's two moments (8 bytes, 2 with an 8-bit optimizer, 4 when the trainer stays in pure BF16). ZeRO stage 1 splits the optimizer states across GPUs, stage 2 also the gradients, stage 3 also the weights.

LoRA freezes the model in BF16 (QLoRA stores it in 4-bit NF4 with double quantization, 0.516 bytes per parameter, and keeps embeddings in 16-bit) and trains rank-r adapters on the attention (q, k, v, o) and MLP (gate, up, down) projections of every layer:

with q = n_hd_h and kv = n_kvd_h. Adapters take 2 bytes per parameter, their gradients 2, and their optimizer states 12 (4 for the FP32 copy and 8 for Adam, or 2 with an 8-bit optimizer).

Activations kept for the backward pass follow Korthikanti et al.: 34 × s × m × h bytes per layer for micro-batch m, plus 5 × n_h × s² × m without flash attention. Gradient checkpointing keeps only each layer's input (2smh bytes) and recomputes one layer at a time; Unsloth also offloads those checkpoints, keeping about 30% on the GPU. Logits take 6 bytes per vocabulary entry per token, 0.5 with chunked loss (Unsloth, torchtune). The trainer gets 95% of GPU memory plus 1 GB of overhead.

Speed

κ is 6 FLOPs per parameter per token for full fine-tuning (forward 2, backward 4) and 4 for LoRA, which skips weight gradients, plus 2 when checkpointing recomputes the forward pass. φ is 1.4 for FP8 training on FP8 GPUs and q is 0.75 for QLoRA's dequantization. Training time is samples × epochs × sequence length divided by tokens per second.

Embedding and reranking

Encoders read each input once and keep no KV cache. Memory is the weights, activations of the batch (batch × s × (4h + 2d_ff) × 2 bytes) and 0.75 GB of overhead, out of 92% of GPU memory. A pass over n tokens takes:

with an MFU of 25% under 200M parameters, 40% under 2B and 50% above, and 85% scaling efficiency across GPUs.

Several models

The Advanced view places several models on a pool of machines with a greedy plan that is simple to follow rather than optimal. Pinned models are placed first, then the largest. Each model takes the fewest GPUs that fit and meet its targets (minimum speed per user, maximum time to first token), adding tensor parallelism or copies when one GPU group is too slow; a model spans identical machines with pipeline parallelism only when no single machine works. Automatic precision starts every model at the highest quality and steps all of them down together while that places more models or meets more targets, then lets each step back up if nothing gets worse. Idle GPUs end up as tensor parallelism or extra copies for the model that gains the most speed per user. Every placement is scored with the same engine.

Find hardware runs the current model on every GPU of the catalog at full, 8-bit and 4-bit precision; Find models runs every catalog model on your hardware.

Cost and energy

Load is 75% for inference, 90% for training and 60% for embedding. Energy per day is power × 24 h; emissions are that energy times the grid's carbon intensity (380 g CO₂ per kWh unless you change it); electricity costs $0.15 per kWh unless you change it. Cloud rental is a rough on-demand price per GPU-hour times the GPUs, over 720 hours a month.

Launch commands

The commands under Run it are built from the same setup: tensor and pipeline parallelism, context length, users, precision, KV cache format, speculative decoding, offloading and, for several copies, one process per copy with its own GPUs and port. They cover vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama, Transformers and Text Embeddings Inference, and Unsloth, TRL and torchtune for fine-tuning. Flags follow each project's documentation; check them against the version you install.

Sources

Data

• Models: the Hugging Face Hub API, each model's config.json and parameter count, rebuilt weekly by build-catalog.ts.
• GPUs: vendor spec sheets for memory, bandwidth, tensor throughput and power, with rough rental prices, in gpus.py.
• Speed measurements: llama.cpp scoreboards for CUDA, ROCm and Apple Silicon; NVIDIA NIM benchmarks; SemiAnalysis InferenceX. All rows are in benchmarks.json with their links.
• The quantization guide's perplexity figures are indicative; the cost of a format varies by model.

Methods

• Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, 2022. Activation memory.
• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, 2020.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023.
• Dao et al., FlashAttention, 2022. Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention, 2023.
• DeepSeek-AI, DeepSeek-V2 (Multi-head Latent Attention), 2024, and DeepSeek-V3 (multi-token prediction), 2024.
• Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, 2023. Li et al., EAGLE-3, 2025. Chen, Liang and Liu, DFlash: Block Diffusion for Flash Speculative Decoding, 2026.
• Patarasuk and Yuan, Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations, 2009. Ring all-reduce traffic.

Related work on predicting LLM speed

• Patwari, Sirasao and Das, Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling (LIFE), 2025.
• WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs, 2026.
• Bambhaniya et al., Demystifying Platform Requirements for Diverse LLM Inference Use Cases (GenZ), 2024.
• Agrawal et al., Vidur: A Large-Scale Simulation Framework for LLM Inference, 2024.

Cite

If the calculator helps your work, please cite it:

Marius Brouty. VRAM Calculator: memory and speed estimates for LLM inference and fine-tuning. 2026. github.com/Marius-brt/VRAM-Calculator

@software{brouty_vram_calculator_2026,
  author = {Brouty, Marius},
  title  = {VRAM Calculator: memory and speed estimates for LLM inference and fine-tuning},
  year   = {2026},
  url    = {https://github.com/Marius-brt/VRAM-Calculator}
}