Why Edge AI on Raspberry Pi Actually Matters Now
For years, "edge AI" meant running a compressed MobileNet on a microcontroller and calling it a day. That era is over. With the Raspberry Pi 5's quad-core ARM Cortex-A76 CPU and a mature on-device inference runtime, you can now run a full LLM locally — no API keys, no round-trip latency, no data leaving the device.
This guide walks through deploying Gemma 4 E2B on a Raspberry Pi 5 using LiteRT (Google AI Edge's production inference runtime) and the LiteRT-LM orchestration layer. By the end you'll have a working setup capable of real-time speech and vision tasks — the same stack that powers the Reachy Mini robot demo.
If you're coming from a cloud-first background, the mental shift is this: your bottleneck is no longer network I/O, it's memory bandwidth and thermal headroom. Everything below is written with that constraint in mind.
The Hardware Reality Check: CPU vs GPU on Raspberry Pi 5
Before writing a single line of code, understand what you're working with.
| Component | FP32 GFLOPS | INT8 TOPS | Notes |
|---|---|---|---|
| Cortex-A76 CPU (4-core) | ~153.6 | ~2.0 | Raw compute powerhouse |
| VideoCore VII GPU | ~76.8 | ~0.24 | 800 MHz, heterogeneous parallelism |
The CPU wins on raw throughput, but the GPU gives you heterogeneous parallel execution — the ability to run vision/audio models concurrently with LLM inference instead of saturating one core cluster. This is the architectural insight behind the Reachy pipeline: offload continuous vision to the GPU, keep CPU cycles for LLM decode and orchestration.
LiteRT exposes GPU inference via its WebGPU (Vulkan) backend through ML Drift, so you can run MediaPipe, Ultralytics YOLO, Moonshine, and embedding models side-by-side with Gemma.
Installing LiteRT CLI
# Recommended: use a virtual environment
python -m venv litert-env
source litert-env/bin/activate
# Install the unified LiteRT CLI
pip install litert-cli
Running Gemma 4 E2B from Hugging Face
The LiteRT Hugging Face Community hosts ready-to-run models. Export your HF token, then invoke the model directly:
# Authenticate with Hugging Face
export HUGGING_FACE_HUB_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx
# Run Gemma 4 E2B with an image attachment and a Reachy-style prompt
litert lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--attachment=image.jpg \
--prompt="You are Reachy Mini. Identify the main object in front of you, \
state its location (Left/Right/Center), and suggest head action in \
10 words or less."
That's it — no separate tokenizer setup, no manual quantization step. LiteRT-LM handles orchestration on top of the runtime.
Performance Numbers You Should Expect
On a Raspberry Pi 5 running Gemma 4 E2B via LiteRT-LM:
- Prefill: ~99 tokens/sec
- Decode: ~9 tokens/sec
- Peak memory: ~1432 MB
- End-to-end generation: ~27.3 chars/sec (~300 wpm) — roughly 2x normal human speech
That decode rate is the number that matters. At ~300 wpm, you're comfortably above real-time speech throughput, which is why voice translation and conversational agents are viable on this hardware.
Caveats, Pitfalls, and What Nobody Tells You
1. The 1432 MB memory ceiling is real. A Raspberry Pi 5 with 4 GB RAM leaves you very little headroom once the OS, Python, and the runtime are loaded. If you plan to run vision models concurrently, go with the 8 GB variant. This is not a soft recommendation.
2. Decode speed ≠ responsiveness. 9 tokens/sec decode sounds fine on paper, but agentic workflows that chain multiple reasoning steps compound that latency. Design your prompts to minimize round-trips — batch instructions, avoid unnecessary tool calls.
3. GPU offload is not free. Moving a model to VideoCore VII saves CPU cycles but adds memory transfer overhead between the two processors. Profile before assuming offload is faster — for small models it often isn't.
4. Thermal throttling will bite you. Sustained LLM inference on a Pi 5 without active cooling will throttle within minutes. Budget for a heatsink and fan if you're targeting anything beyond burst workloads.
5. Quantization trade-offs. The default LiteRT distributions are pre-optimized, but if you bring your own model, INT8 quantization can degrade reasoning quality noticeably on smaller Gemma variants. Validate on your actual task, not on perplexity benchmarks.
Where to Go Next
- Multi-modal pipelines: Combine YOLO on the GPU with Gemma on the CPU for a full perceive-reason-act loop.
- Agentic orchestration: Wire the LiteRT CLI into your coding agent (e.g., Google Antigravity) so it can autonomously run conversion → quantization → benchmark → inference workflows.
- Hailo acceleration: LiteRT integration with Hailo AI accelerators (AI HAT+ / AI HAT+ 2) is coming — same workflows, dedicated NPU offload. Worth waiting for if you're building production robotics.
- Voice translators: The Gemma Translator repo is a good reference implementation for fully offline speech-to-speech.
For a broader look at how agentic reasoning models are reshaping cost structures in production, see our analysis of NVIDIA Nemotron 3 Ultra and its 30% agent cost reduction. And if you're thinking about the UX layer of the systems you're building, Cloudflare's Turnstile redesign deep dive is worth a read.
TL;DR
You can run a real LLM on a $80 single-board computer today. LiteRT + Gemma 4 E2B on Raspberry Pi 5 delivers ~99 tok/s prefill, ~9 tok/s decode, and ~300 wpm generation at under 1.5 GB peak memory — all offline.
The install is three commands. The hard part is architectural: knowing when to split work between CPU and GPU, how to budget memory, and how to keep thermal headroom. Get those right and you have a genuinely autonomous edge system.
Start with the LiteRT CLI, grab a model from the Hugging Face community, and ship something that never phones home.
Source: Mastering Edge AI on Raspberry Pi with LiteRT and Gemma