How to Run Gemma 4 Locally: A Step-by-Step Guide

Running AI models locally gives you complete control over your data, eliminates API costs, and enables offline use. Gemma 4 makes this practical across a wide range of hardware — from smartphones to workstations. This guide walks you through every method, from the easiest one-command setup to advanced production serving.

Prerequisites

Before you begin, consider your hardware and which model fits:

For GPU inference, NVIDIA GPUs with CUDA support or Apple Silicon Macs with Metal are recommended. CPU-only inference works but is significantly slower for larger models.

Method 1: Ollama (Easiest)

Ollama is the fastest way to get started. It handles model downloading, quantization selection, and serving in a single tool.

Install Ollama:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Or download from https://ollama.com/download

Pull and run a model:

# Pull the model (one-time download)
ollama pull gemma4:26b

# Start an interactive chat
ollama run gemma4:26b

Available tags:

ollama pull gemma4:31b    # 31B Dense — highest quality
ollama pull gemma4:26b    # 26B A4B — best balance
ollama pull gemma4:e4b    # E4B — lightweight multimodal
ollama pull gemma4:e2b    # E2B — edge/mobile

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1 for integration with other tools.

Method 2: LM Studio (GUI)

LM Studio provides a polished desktop application with a chat interface, model management, and a local API server.

1. Download and install LM Studio from lmstudio.ai
2. Open the app and search for "gemma-4" in the model browser
3. Select your preferred variant and quantization level
4. Click Download, then start chatting in the Chat tab

LM Studio also provides a local server mode that exposes an OpenAI-compatible API, making it easy to integrate with existing applications.

Method 3: llama.cpp (Advanced)

llama.cpp gives you maximum control over inference parameters and supports the widest range of hardware.

Build from source:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)

# For CUDA support:
make -j$(nproc) GGML_CUDA=1

# For Metal (macOS) support:
make -j$(nproc) GGML_METAL=1

Download a GGUF model:

Download pre-quantized GGUF files from HuggingFace. Look for community uploads with Q4_K_M (balanced), Q5_K_M (higher quality), or Q8_0 (near-lossless) quantization.

Run inference:

./llama-cli -m gemma-4-26b-a4b-it-Q4_K_M.gguf \
  -p "Explain the difference between MoE and dense architectures" \
  -n 512 -ngl 99

The -ngl 99 flag offloads all layers to GPU. Reduce this number if you have limited VRAM and want to split between CPU and GPU.

Method 4: vLLM (Production)

vLLM is designed for high-throughput production serving with features like continuous batching, PagedAttention, and tensor parallelism.

Install and serve:

pip install vllm

vllm serve google/gemma-4-26b-a4b-it \
  --max-model-len 8192 \
  --tensor-parallel-size 1

Query the OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-26b-a4b-it",
    "messages": [{"role": "user", "content": "Hello, Gemma!"}]
  }'

For multi-GPU setups, increase --tensor-parallel-size to match your GPU count. vLLM automatically handles model sharding across devices.

Choosing the Right Model

When in doubt, start with the 26B A4B model via Ollama. It offers the best balance of quality and resource efficiency for most users.

Conclusion

Running Gemma 4 locally has never been easier. Whether you choose the one-command simplicity of Ollama, the visual interface of LM Studio, the fine-grained control of llama.cpp, or the production-grade serving of vLLM, you can have a powerful multimodal AI running on your own hardware in minutes.

The E2B model makes AI accessible on virtually any device, while the 31B Dense model delivers quality that competes with the best proprietary models — all running privately on your machine with zero API costs and no data leaving your network.