Most people meet AI through a phone or a laptop. Xiaozhi puts it somewhere else: inside a small board with a microphone, a speaker and, if you want, a tiny screen. You say a wake word, ask a question, and the device answers out loud. It feels like a smart speaker, but you build it, you choose the board, and you can read every line of the code.
Xiaozhi, published as xiaozhi-esp32, is open-source firmware under the MIT license. It has grown into one of the most widely used ESP32 voice projects, with support for a long list of boards from many makers. This guide explains what it is, how it works in plain words, what you need, and how to get your first one talking, including how an AI coding tool can help.
What Xiaozhi is
Xiaozhi is the software that runs on the ESP32 chip. It handles the parts that must happen on the device: listening for the wake word, recording your voice, compressing the audio, sending it over the network, playing the reply and drawing faces or text on a screen.
The thinking happens on a server. By default the firmware talks to the official Xiaozhi service, where you can register a device and pick a model. The project notes that personal users can use a Qwen real-time model there for free. You can also run your own server, which is covered in a separate guide.
The firmware supports several ESP32 chips, including the ESP32, ESP32-C3, ESP32-C5, ESP32-C6, ESP32-S3 and ESP32-P4. The ESP32-S3 is the most common choice, because it has enough memory and speed for audio work.
How it works in plain words
Think of the device as a pair of ears and a mouth with a short memory. The brain lives somewhere else.
First, the board listens all the time for a wake word. This part runs on the chip itself, using Espressif's ESP-SR speech library, so nothing leaves the device while it waits. The wake word can be changed.
Second, once it hears the wake word, it records what you say and compresses it with the Opus audio codec. Compressed audio is small, which matters on a slow or busy Wi-Fi network.
Third, it sends the audio to the server over WebSocket, or over a mix of MQTT and UDP. The server turns speech into text, sends the text to a language model, turns the answer back into speech and streams it to the board. Some setups use a real-time voice model that does all of this in one step.
Fourth, the board plays the answer and can show an emoji face or text on its screen.
There is one more part that makes Xiaozhi interesting for makers. It uses the Model Context Protocol, or MCP. On the device side, MCP lets the language model call simple tools on the board, such as changing the speaker volume, turning on an LED or moving a servo. On the cloud side, MCP lets the model reach outside tools, such as smart home controls, a search tool or apps on your computer. In plain words, the model can do things, not just talk.
What you need
A supported ESP32-S3 board with a microphone and speaker. Many ready-made boards already have both. The project lists boards from Espressif, M5Stack, Waveshare, LILYGO, Seeed and others, plus a breadboard design for people who want to wire their own.
A USB cable that carries data. This sounds obvious, but it is the most common first problem. Many cables only carry power.
A 2.4 GHz Wi-Fi network. The ESP32 family used here does not join 5 GHz networks, so make sure your router offers 2.4 GHz.
A computer with a Chrome-based browser for the simple flashing route, or a full ESP-IDF setup if you want to build from source.
If you choose the breadboard route, you will also need an I2S microphone module, a small I2S amplifier, a speaker and, if you want faces, a small OLED or LCD screen.
The easy route: flash ready-made firmware
The project recommends that beginners start with ready-made firmware, with no development setup at all. You download the build for your exact board, connect the board, and flash it with a browser tool or Espressif's flashing tool.
A sensible order looks like this.
Find your exact board in the supported list. Buy the board from that list if you have not bought one yet. This saves hours later.
Back up the factory firmware before you flash anything, if your board shipped with its own software. Bricked boards are a common story, and a backup lets you go back.
Flash the Xiaozhi build that matches your board.
When the board starts, it creates its own Wi-Fi hotspot. Join it from your phone and enter your home 2.4 GHz network details.
The board shows a code. Enter it in the Xiaozhi console to bind the device to your account, then choose a model and a voice.
Say the wake word and ask a simple question.
Using an AI coding tool to help
You do not need to write code to get Xiaozhi running, but a terminal agent or AI code editor can save a lot of time. Builders who are new to hardware often let an agent run the setup commands: installing the flashing tool, finding the serial port, running the backup, flashing the image and reading the serial log when something fails.
A good way to work with the agent is to give it a clear plan. Tell it the exact board name, your operating system and what you want: back up the current firmware, flash the matching Xiaozhi build, then open the serial monitor. Ask it to explain each command before it runs, and approve them one by one. When the board misbehaves, paste the serial log into the chat and ask what the error lines mean.
If you go further and build from source, the project suggests Cursor or VS Code with the ESP-IDF extension, and it now requires ESP-IDF version 6 or later. An agent is useful here too. You can ask it to add a new board definition by following the custom board guide in the repository, or to add a small MCP tool, such as one that reads a temperature sensor.
Common pitfalls
The board is not detected. Try another USB cable first. Then check that you are using the right port, and try holding the BOOT button while you plug the board in.
Wi-Fi will not connect. Check that the network is 2.4 GHz and that the password is right. Some mesh routers merge both bands under one name, which can confuse small devices.
The board restarts over and over after changing the wake word. Some wake-word models need more memory than a board can give. Builders have reported that switching to a different wake-word model fixed a reboot loop. If you change the wake word and the board starts looping, go back to the default and try another model.
Tools that run code. Some example MCP tools are written to show what is possible, not to be safe. If an example takes text from the model and runs it as code, add your own checks before you use it, or leave it out. The model's output should never be trusted as if it were your own code.
Privacy. With the default setup, your voice goes to a cloud service once the wake word is heard. If that matters to you, read what the service stores, or run your own server.
Variations to try
Change the wake word, the face images and the fonts. The project offers a web tool for custom assets, so you can give the device its own look without touching code.
Add a camera on a supported board, so the model can look at what is in front of it and describe it.
Write a device-side MCP tool to control something on your desk, such as a lamp through a relay or a small servo that waves.
Point the device at your own server to choose your own models, keep your data at home, or connect it to your own AI agent.
Use a 4G-capable board if you want the assistant to work away from Wi-Fi. The firmware supports some cellular modules.
The short version
Xiaozhi turns a supported ESP32-S3 board into a voice assistant that listens for a wake word on the chip and sends your question to a language model on a server. Start with ready-made firmware for a board on the supported list, use a data cable and a 2.4 GHz network, back up the original firmware, and let an AI coding tool handle the commands while you approve each one. Once it talks, change the wake word, add a tool through MCP, and decide whether you want to keep using the hosted service or run your own.
Project resources: https://github.com/78/xiaozhi-esp32