Gemma 4

Getting Started with Gemma 4

Quick start guides for running Gemma 4 with Ollama, Transformers, vLLM, llama.cpp, and LM Studio.

Get Gemma 4 running in minutes with your preferred framework. Choose the method that best fits your workflow below.

Ollama

The fastest way to get started locally. Install Ollama and run:

# 26B A4B (MoE) — recommended for most users
ollama run gemma4:26b

# 31B Dense — highest quality
ollama run gemma4:31b

# E4B — lightweight, great for laptops
ollama run gemma4:e4b

# E2B — ultra-light, runs on phones
ollama run gemma4:e2b

Hugging Face Transformers

Use the standard Transformers API for fine-tuning or custom pipelines:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "google/gemma-4-31B-it"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)

outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Available model IDs:

  • google/gemma-4-31B-it — 31B Dense instruction-tuned
  • google/gemma-4-26B-A4B-it — 26B MoE instruction-tuned
  • google/gemma-4-E4B-it — E4B instruction-tuned
  • google/gemma-4-E2B-it — E2B instruction-tuned

vLLM

For high-throughput serving with OpenAI-compatible API:

pip install vllm

# Serve the 31B Dense model
vllm serve google/gemma-4-31B-it

# Serve the 26B MoE model (lower VRAM requirement)
vllm serve google/gemma-4-26B-A4B-it

Then query the API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

llama.cpp

For CPU and quantized inference with GGUF models:

# Clone and build llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release

# Download a GGUF model (e.g., 26B A4B Q4_K_M)
huggingface-cli download google/gemma-4-26B-A4B-it-GGUF \
  gemma-4-26b-a4b-it-q4_k_m.gguf \
  --local-dir models/

# Run interactive chat
./build/bin/llama-cli \
  -m models/gemma-4-26b-a4b-it-q4_k_m.gguf \
  -c 4096 \
  --chat-template gemma

LM Studio

A graphical desktop application for running models locally:

  1. Download and install LM Studio
  2. Open the app and search for "gemma-4" in the model browser
  3. Choose the variant and quantization that fits your hardware
  4. Click Download, then start chatting

LM Studio automatically selects optimal settings for your hardware and provides a built-in chat interface as well as a local API server.

Next Steps

  • Models — Detailed specifications and hardware requirements for every variant
  • Benchmarks — Performance comparisons across reasoning, coding, and vision tasks
  • Downloads — Direct download links for all model formats
Getting Started with Gemma 4 | Gemma 4