VRAM Calculator

Model

Parameters: 8B
Architecture: Dense
Attention: GQA, 8 KV
Context: 128K

Hardware

Memory: 24 GB
Bandwidth: 1,008 GB/s
FP16 compute: 165 TFLOPS

GPU, 24 GB total

Precision

Model weights

KV cache

Workload

Context length2,048tokens

Batch size1

Concurrent users1

Runtime

Advanced

Prefix cachingReuse the KV cache of a shared system prompt across users.

Speculative decodingGuess several tokens cheaply, check them in one pass; typically 1.5 to 3× faster for few users.

Offload to CPU memoryKeep some layers in system RAM to run models larger than VRAM, at a speed cost.

Flash attentionFused attention kernels. Without them llama.cpp, Ollama and Transformers build the full attention matrix.

Fits on 1× RTX 4090 (24GB)

6.46 GB of 21.6 GB usable (24 GB total). Plenty of headroom for longer prompts or more users.

30% used

21.6 GB usable

Model Weights4.53 GB70%
KV Cache0.25 GB4%
Activations0.17 GB3%
Framework Overhead1.5 GB23%
Free15.14 GB70%

Speed

Per user

117.2

tok/s

Each of 1 user

Throughput

117.2

tok/s

All users combined

Single request

117.2

tok/s

When running alone

First token

473

Prompt processing

Token latency

8.5

Between tokens

Estimates fitted to published benchmarks. Measured your own setup? Share the numbers to make them more accurate.

Run it

vllm serve Llama-3.1-8B-Instruct-Q4_K_M.gguf --tokenizer meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 1 --max-model-len 2048 --max-num-seqs 1

Needs a Q4_K_M GGUF file of meta-llama/Llama-3.1-8B-Instruct; most popular models have one on Hugging Face.

Memory as you scale

Total memory KV cache Usable memory

Serving capacity

Users that fit at this context: 61
KV cache per user: 0.25 GB
KV cache space still free: 15.1 GB
Requests per second: 0.23 at 512 output tokens
Risk of queueing users: None

Cost and energy

Cloud rental: $0.4 per hour, $288 per month
Electricity, always on: $36 per month at $0.15/kWh
Power draw: 338 W
Energy: 8.1 kWh per day
Emissions: 92.5 kg CO₂e per month