Gemma 4 GGUF: pick the quant first, download second
GGUF is the file format llama.cpp, Ollama, LM Studio and KoboldCpp all read, and it is almost always what you want for local inference. The only real decision is which quantization to download, and that decision is driven by one number: how much VRAM you can spare.
This page gives you the answer in one table instead of four blog posts. Q4_K_M is the right default for nearly everyone — it loses roughly 5-7% quality against the full model while cutting the file to a quarter of the size. Drop to Q5_K_M or Q8_0 only when VRAM is genuinely free, and reach for BF16 only when you are evaluating the model rather than using it.
Which quant should I download?
Quantization trades file size and VRAM for output quality. In practice the quality curve is flat until you go below 4-bit, which is why Q4_K_M is the community default.
| Quant | Bits | Quality vs BF16 | Pick it when |
|---|---|---|---|
| BF16 / F16 | 16-bit | 100% (reference) | You are publishing benchmark numbers, fine-tuning, or comparing quantisation effects. These files are 3-4x larger than Q4 and rarely worth it for chat. |
| Q8_0 | 8-bit | ~99% | You have the VRAM to spare and want near-lossless output. Roughly halfway between Q4 and BF16 in size. |
| Q5_K_M | 5-bit | ~96-97% | You can afford about 25% more VRAM than Q4 and want a visible edge on long-form writing or code. |
| Q4_K_M ← recommended | 4-bit | ~93-95% | Almost always. It is the best quality-per-gigabyte point in the K-quant family, and it is what Ollama downloads by default.Best default for most people |
Below 4-bit (Q3_K, Q2_K) quality degrades sharply on reasoning-heavy prompts. If you cannot fit Q4_K_M, run a smaller model variant in Q4 rather than a larger variant in Q3 — the smaller model wins.
Gemma 4 GGUF file sizes and VRAM
Community quantization listings, cross-checked against Google's official Gemma 4 model cards. File sizes change as repositories are revised; verify on the linked repository before downloading.
| Model | Params | Q4_K_M | Q5_K_M | Q8_0 | BF16 | VRAM @ Q4_K_M | Hugging Face repo |
|---|---|---|---|---|---|---|---|
| Gemma 4 E2B-it | 5B | 3.11 GB | 3.36 GB | 5.05 GB | 9.31 GB | ~4 GB | unsloth/gemma-4-E2B-it-GGUF |
| Gemma 4 E4B-it | 8B | 4.98 GB | 5.48 GB | 8.19 GB | 15.1 GB | ~6 GB | unsloth/gemma-4-E4B-it-GGUF |
| Gemma 4 26B-A4B-it | 27B MoE | 16.9 GB | 21.2 GB | 26.9 GB | — | ~20 GB | unsloth/gemma-4-26B-A4B-it-GGUF |
| Gemma 4 31B-it | 33B Dense | 18.3 GB | 21.7 GB | 32.6 GB | — | ~22 GB | unsloth/gemma-4-31B-it-GGUF |
VRAM figures assume Q4_K_M with an 8K context and roughly 1 GB of headroom for the KV cache. Long contexts grow the KV cache — a 32K context on E4B costs about 2 GB more than 8K, so budget for the context you actually use, not just the weights.
How to download and run Gemma 4 GGUF
Four ways, ordered from easiest to most control. Every command below is copy-pasteable.
1. Ollama (fastest)
Ollama picks a quantization that fits your hardware and handles the llama.cpp build for you. Start here unless you have a reason not to.
# fastest path — Ollama picks the quant for your hardware
ollama pull gemma4:e4b
ollama run gemma4:e4b
# pin a specific quant
ollama run gemma4:e4b-q8_02. Hugging Face CLI (pick the exact file)
Use this when you want one specific quant, a resumeable download, or you are deploying into a container that has no Ollama.
pip install -U huggingface_hub
# one file only — the recommended 4.98 GB build
huggingface-cli download unsloth/gemma-4-E4B-it-GGUF \
--include "gemma-4-E4B-it-Q4_K_M.gguf"
# the whole quant set, if you want to compare
huggingface-cli download unsloth/gemma-4-E4B-it-GGUF --local-dir ./gemma4-e4b-gguf3. llama.cpp directly (most control)
Bypasses Ollama entirely. Use -hf to stream a repo file without downloading it first, or run llama-server when you want an OpenAI-compatible endpoint.
# straight from the hub, no manual download
./llama-cli -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M
# server mode, OpenAI-compatible on :8080
./llama-server -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M --port 8080 -c 327684. LM Studio (GUI)
Search for "gemma-4" in the Discover tab, then pick the quant from the dropdown on the model card. LM Studio shows an estimated VRAM requirement per quant, which is easier than reading a table.
Four things that break a first GGUF run
Almost every failed first attempt is one of these. Each one has a single fix.
The model loads, then the machine freezes or the process is killed
You are out of memory. GGUF will happily try to load a file larger than your VRAM and spill into system RAM, which on a small machine means an OOM kill rather than a clean error.
Fix: Drop one quant (Q5_K_M → Q4_K_M) or move to a smaller variant. If you have enough system RAM but not enough VRAM, add --n-gpu-layers to offload only part of the model to the GPU.
Output is fast but the model loops or repeats phrases
Temperature or repetition penalty, not the GGUF file. Quantization affects coherence, but loops are almost always sampling settings.
Fix: Raise the repetition penalty to about 1.1 and keep temperature in the 0.6-0.9 range for chat. Our presets page has tested values per task.
Generation is extremely slow even though the file loaded fine
The model is running on CPU. Chat and coding prompts can look correct on CPU and still take minutes per answer, which reads as a hang.
Fix: Confirm GPU offload is actually on (llama.cpp prints the layer split at startup, and Ollama logs report the same). On Apple Silicon use MLX instead of GGUF — it is measurably faster on unified memory.
Answers get worse as the conversation gets longer
Context overflow. When the prompt exceeds the configured context window the oldest turns are silently dropped or truncated mid-instruction.
Fix: Set an explicit context length (-c 32768, or num_ctx in an Ollama Modelfile) that matches your use case, and check the VRAM cost from the table above before raising it.
Gemma 4 GGUF FAQ
Which Gemma 4 GGUF file should I download first?
E4B-it in Q4_K_M, unless your GPU has less than about 6 GB of usable VRAM, in which case start with E2B-it Q4_K_M. Q4_K_M is the recommended default because it keeps roughly 93-95% of full-precision quality at a quarter of the file size, and every major runner supports it.
How much VRAM does Gemma 4 GGUF need?
At Q4_K_M: about 4 GB for E2B, 6 GB for E4B, 20 GB for the 26B-A4B MoE, and 22 GB for the 31B Dense model. Those numbers include an 8K context and a small safety margin. A 32K context adds roughly 2 GB on E4B, so raise the context only if you have the headroom.
Are Gemma 4 GGUF files official Google releases?
Partly. Google publishes official QAT checkpoints on Hugging Face, and the GGUF quantizations most people download are community conversions of those weights — usually from the unsloth repositories linked in the table above. They are redistributed under the same Apache 2.0 licence. Check the repository's model card if you need the provenance for a commercial deployment.
Is Q8_0 worth the extra size over Q4_K_M?
Only if VRAM is genuinely free and you are doing something quality-sensitive, such as long-form code generation or evaluating model behaviour. In blind comparisons most people cannot reliably separate Q8_0 from Q5_K_M, and Q4_K_M is close enough for chat, summarisation and drafting.
Can I run Gemma 4 GGUF without a GPU?
Yes, but expect minutes per response on the larger variants. CPU inference works, and llama.cpp will use every available core, but the 26B and 31B models are impractical on CPU. E2B in Q4_K_M is the only variant that stays usable on CPU-only hardware.
Does GGUF work with Ollama and LM Studio?
Yes, both read GGUF natively — that is the format they were built around. Ollama manages its own copy of the file, while LM Studio points at a file on disk. If you already have a GGUF file downloaded, LM Studio will load it directly without re-downloading.
What is the difference between GGUF and SafeTensors for Gemma 4?
GGUF is a single-file format built for local inference with llama.cpp, with quantization baked in and metadata that tells the runtime how to load it. SafeTensors is the training and serving format used by Transformers and vLLM. Pick GGUF for a laptop or workstation, SafeTensors for a server or anything you plan to fine-tune.
Need the model files for every format, not just GGUF?
The download hub covers SafeTensors, GPTQ and MLX alongside GGUF, with the official Google sources and the matching VRAM notes for each.