I built a desk-top voice assistant called Pi-Esp Jarvis: say “Jarvis” to a little ESP32 screen, and a Raspberry Pi 5 answers, with Claude stepping in for the hard questions. It sets timers, wakes you up, reads the weather, keeps your shopping list and plays music. You can build one in an afternoon without writing any code.
What it does
Jarvis is a smart speaker you own completely: a small board listens and talks, and your Raspberry Pi does the thinking. An animated robot face called Visor shows what it’s doing.
- Answers questions privately. Everyday chat is handled by an AI model running on the Pi itself, so it never leaves your house.
- Calls in Claude for the hard ones. Say “Use Claude, …” and the question goes to Claude. Ask something live, like last night’s score, and Claude searches the web.
- Timers, alarms and reminders. “Set a pasta timer for 9 minutes” or “wake me up at 7 every weekday”. When one is due, the board wakes up by itself and rings.
- Weather, news and a morning briefing. Say “good morning” and hear the date, the forecast, your reminders and three headlines.
- Lists and memory. “Add milk to the shopping list” or “remember the spare key is under the blue pot”. There’s a page for your phone too.
- Music. Your own files, or internet radio by genre or station name.
- Pi control. “How’s the Pi doing?”, “open YouTube”, “reboot the Pi” (it asks first).
- Gestures. Double-tap the screen to talk, turn it face-down to snooze, shake it to stop.
How it works
The trick is splitting the job in two: a cheap board does the listening and talking, and the Pi does the heavy thinking. The board can’t run an AI model (it has less than 1% of the Pi’s memory), but it’s brilliant at hearing a wake word and playing sound.
- You say “Jarvis”. The ESP32 board hears the wake word on its own chip, then streams your voice to the Pi over Wi-Fi.
- Whisper turns your speech into text. It’s an open-source speech recogniser running on the Pi.
- A router decides who answers. Timers, lists, weather, music and Pi control are handled by simple built-in skills (instant, no AI needed). Everyday chat goes to Llama 3.2, a small AI model running on the Pi. “Use Claude” questions, and anything that needs the internet, go to Claude.
- Piper turns the answer into speech and streams it back. The board plays it while Visor changes expression.
A typical answer starts within 2 – 6 seconds. Everything on the Pi runs in two Docker containers that restart by themselves after a power cut.
The board runs xiaozhi-esp32, a popular open-source voice-assistant firmware that already supports this exact board. Normally it talks to the vendor’s cloud servers; this project points it at your Pi instead. It also adds the Visor face, double-tap, motion gestures and a way for the Pi to wake the board when an alarm goes off.
What you’ll need
One Raspberry Pi, one ESP32 board and a cable. A Claude account is optional.
| Part | Why | Notes |
|---|---|---|
| Raspberry Pi 5 (8 GB) | Runs the AI, speech and skills | 4 GB works too; the setup picks a smaller AI model |
| microSD card (32 GB+) or SSD | Holds the system and AI models | An SSD loads models faster |
| Official Pi 5 power supply (27 W) | Steady power under load | Weak supplies make the Pi slow itself down |
| Waveshare ESP32-S3-Touch-AMOLED-1.8 | Microphone, speaker, screen, touch and motion sensor in one | Revision 1 or 2 both work |
| USB-C data cable | Flashing the board from the Pi | Some cables only charge; you need a data one |
| Claude API key (optional) | “Use Claude” and live web answers | From console.anthropic.com, pay as you go |
You don’t need to solder anything or write code. Budget about an hour, most of it waiting for downloads and the firmware build.
Build it in five steps
Two scripts do almost all the work: one sets up the Pi, the other builds and flashes the board.
1. Prepare the Pi
Install Raspberry Pi OS (64-bit) with Raspberry Pi Imager. In the Imager’s settings, give the Pi a name (I used jarvis), turn on SSH and enter your Wi-Fi details. Boot it and open a terminal, either on the Pi or from your computer with ssh you@jarvis.local.
2. Install Pi-Esp Jarvis
git clone https://github.com/nativebreed/pi-esp-jarvis.git
cd pi-esp-jarvis
bash setup.shThe script installs Docker if you don’t have it, then asks three optional questions: your Claude API key, your town (for the weather) and your units. Then it builds and starts everything. When it asks whether Jarvis may control the Pi, say y to get the Pi health and reboot commands.
The first start downloads about 2 GB of AI models in the background. Run docker compose logs -f assistant to watch, and wait for “Local model … ready”.
3. Flash the ESP32 board
Plug the board into the Pi with the USB-C data cable and run:
bash flash-esp32.sh # board revision 1
bash flash-esp32.sh v2 # board revision 2The first run builds the firmware on your Pi using Espressif’s official tools in Docker. It takes 20 – 30 minutes, so it’s a good time for a cup of tea. Later runs take a minute. Not sure which revision you have? Look for “V2” on the board or box; if the screen stays black after flashing v1, run it again with v2.
4. Connect the board to Wi-Fi
The board starts its own Wi-Fi network called Xiaozhi-XXXX. Join it from your phone, and a setup page opens (if it doesn’t, browse to http://192.168.4.1). Pick your 2.4 GHz Wi-Fi and type its password. The board restarts and Visor appears on a black screen.
5. Say “Jarvis”
Try “Jarvis, what’s the weather tomorrow?” You’ll hear a chime, Visor’s eyes light up, and a few seconds later it answers. You can also double-tap the screen instead of saying the wake word.
Things to try
Say “Jarvis” (or double-tap the screen) before each of these.
| Say | What happens |
|---|---|
| “What’s the weather tomorrow?” | A spoken forecast for your town |
| “Use Claude, explain black holes simply” | Claude answers; Visor’s eyes become an amber bar |
| “Who won the football last night?” | Claude searches the web for a live answer |
| “Set a pasta timer for 9 minutes” | The board rings when it’s done, even if it’s gone to sleep |
| “Wake me up at 7 every weekday” | A repeating alarm; stopping it with “stop” plays your morning briefing |
| “Remind me in 20 minutes to check the oven” | A reminder that wakes the board |
| “Add eggs and bread to the shopping list” | Saved; check it on your phone at http://jarvis.local:8000/lists |
| “Remember the spare key is under the blue pot” | Later, “where’s the spare key?” just works |
| “Play some soul music” | Your own music, or a soul radio station |
| “Good morning” | Date, weather, today’s reminders and the headlines |
| “How’s the Pi doing?” | Temperature, memory, disk and whether everything is running |
To silence a ringing alarm, double-tap and say “stop” or “snooze”, turn the board face-down to snooze it, or give it a shake to stop it.
How I built it with Claude Code
I didn’t write this code myself: I built the whole thing in one long conversation with Claude Code, in an afternoon. I described what I wanted, Claude wrote and tested the code, and I ran the commands on my Pi and reported back what happened.
The build went roughly like this:
- The question. I started by asking whether an ESP32 could be an AI assistant on its own. Claude explained that it can’t run a language model, but makes a great “ears and mouth” for a Raspberry Pi. That became the design.
- Finding the firmware. Instead of writing firmware from scratch, Claude found an open-source project that already supported my exact board, and wrote a small patch to point it at my Pi.
- The Pi server. Claude wrote the Python server that speaks the board’s protocol, plus the speech, routing and Docker setup. It tested everything against a simulated board before I touched any hardware.
- The first real test. I flashed the board, said “Jarvis”, and it worked. Then I asked it to play soul music. It said it was playing, then apologised that it couldn’t. The local model had made it up! Claude added a proper music player and told the local model never to pretend.
- Making it mine. One-line requests became features: “use Claude” as the trigger phrase, double-tap to wake, then timers, alarms, weather, lists, a morning briefing, Pi control and gestures in a single batch.
- The face. I asked for a big animated character. Claude designed Visor, an original robot helmet, and showed me a to-scale preview on the real screen size, with two size options, before writing any code.
A few things I learned that will help if you try this yourself:
- Describe the outcome, not the code. “I want it to wake me up at 7” worked better than naming technologies.
- Ask for a preview before big changes. Seeing the face at real size saved a round of rework.
- Report real-world results. The most useful messages were the simple ones, like “it said it was playing but it wasn’t”.
- Ask what it couldn’t test. Claude was upfront about what it had checked against a simulator and what only real hardware could prove.
Costs, privacy and limits
Running Jarvis costs nothing beyond electricity, unless you use Claude.
- Claude is pay-as-you-go. A short spoken answer usually costs around a cent or less, and web searches add about $10 per 1,000. You can set a monthly spending limit in the Claude Console. Without an API key, everything stays local and free.
- What stays at home. Wake-word detection happens on the board. Speech recognition, the local AI, your lists, memories and alarms all live on your Pi.
- What leaves the house. Only “use Claude” questions and live web questions (sent to Anthropic), plus weather, news and radio lookups (from free public services).
- The local AI is small. Llama 3.2 3B is fine for chat and quick facts, but it can be wrong, and it’s slower than the cloud. That’s exactly why Claude is one phrase away.
- It’s English-first. The speech models and phrases are set up for English.
- Keep it plugged in for alarms. On battery, the board switches itself off after 5 minutes idle and can’t ring until you turn it back on.
Make it your own
Most changes are one line in the .env settings file, then docker compose up -d.
| To change | How |
|---|---|
| The voice | Set PIPER_VOICE, e.g. en_GB-alan-medium (pick from the Piper voices) |
| Speaking speed | SPEECH_SPEED=1.2 for 20% faster |
| The local AI | LOCAL_MODEL=llama3.2:1b for speed, or any other Ollama model |
| Claude model | CLAUDE_MODEL=claude-haiku-5-5 for faster, cheaper answers |
| Your news | NEWS_FEEDS = any RSS feed addresses, comma-separated |
| The wake word | Reflash with WAKE_WORD=hiesp bash flash-esp32.sh to say “Hi ESP” |
| The face | Swap the GIFs in firmware/face/emoji/ (one per mood) and reflash |
| What it can do on the Pi | Add apps, websites or your own scripts to pi-helper/actions.json |
And if you have Claude Code yourself, you can keep going the way I did: open the project folder and describe the next feature you want.
If something goes wrong
Most problems come down to the board not finding the Pi, or a cable that only charges.
| Problem | Fix |
|---|---|
| The screen says it can’t connect to the server | Your network can’t find the Pi by name. Reflash with the Pi’s IP address: OTA_URL=http://$(hostname -I | cut -d' ' -f1):8000/xiaozhi/ota/ bash flash-esp32.sh |
| “No ESP32 found” when flashing | Use a data cable. Still nothing? Unplug the board, hold BOOT while plugging it back in, and retry |
| “Permission denied” from Docker | Log out and back in (or reboot) after the first setup |
| It hears you but never answers | Run docker compose logs -f assistant to see what it heard and who answered |
| “Jarvis” rarely triggers | Speak from about 1 m away, or switch to “Hi ESP” (see above) |
| Alarms don’t ring | Keep the board on USB power and on Wi-Fi |
The README has more.
Get the code
Everything is free and open source on GitHub: github.com/nativebreed/pi-esp-jarvis. The README there has the full command list, every setting, and the parts list again.
If you build one, I’d love to see it. Share a photo, open an issue if you get stuck, or send a pull request with your own improvements.
Pi-Esp Jarvis is MIT-licensed and builds on some brilliant open-source work: xiaozhi-esp32 (the board firmware), Ollama with Meta’s Llama 3.2, faster-whisper, Piper, Open-Meteo weather data and the radio-browser.info station directory. The “use Claude” answers come from Claude by Anthropic. It’s a hobby project, not affiliated with Anthropic, Raspberry Pi, Espressif or Waveshare.
First dropped: | Last modified: October 15, 2026