A Xiaozhi voice assistant has two halves. The ESP32 board listens, records and speaks. A server does the heavy work: it turns speech into text, asks a language model for an answer and turns that answer back into speech. By default, the board uses the official hosted service. For many people that is enough.
Others want more control. They want to pick their own models, keep conversations on their own computer, connect the assistant to their own tools or learn how the whole system fits together. That is where xiaozhi-esp32-server comes in. It is a community back end for Xiaozhi devices, led by a university research team, written mainly in Python with Java and Vue parts for the management console. It is released as open source.
This guide explains what the server does, how the pieces connect, what you need to run it and how to set it up step by step, with an AI coding tool as a helper.
Who this is for
The project says it plainly: it is for people who already have an ESP32 Xiaozhi device that works with the official service and now want to run their own back end. If your board has never talked to the hosted service, get that working first. It proves that the hardware, the cable and the Wi-Fi are fine, so any later problem is in the server.
What the server does
When the board sends audio, the server runs a pipeline of modules. Each module can be swapped.
Voice activity detection decides when you have stopped talking.
Speech recognition turns your voice into text. The free setup in the project's guide uses FunASR, which runs locally.
A language model writes the answer. The free setup uses a free cloud model, and the faster setup uses a paid streaming model.
A vision model can describe images on boards with a camera.
Speech synthesis turns the answer into audio. The free setup uses Microsoft's Edge TTS voices.
Intent handling and function calling let the model decide when to call a tool instead of just replying.
Memory lets the assistant remember recent conversations, either locally or through an outside memory service, and it can summarise past talks.
On top of this, the server supports voiceprint recognition, which tries to tell who is speaking so the model can answer each person in a more personal way. It can use a knowledge base through RAGFlow, so the model looks up your own documents before it answers. It supports MCP in several forms, so the model can call tools on the device, tools on the server and tools exposed by other MCP services. There is also a plugin system for custom functions.
The project offers two configurations. A free starter setup uses free components and suits personal home use. A streaming setup uses paid streaming services and suits demos or several devices at once. The project reports that its streaming mode responds noticeably faster than its early versions, and it publishes its own performance tests so you can measure on your own hardware.
Two ways to install
The minimal install runs only the core server. Settings live in a configuration file, and no database is needed. It is good for a single device and a low-powered computer.
The full install adds a web management console, a database, user accounts, several agents and device management. It is good when you want to manage more than one device or switch settings without editing files.
Both can run with Docker or directly from source. Docker is the easier path for most beginners, because it packages the right versions of everything.
What you need
A computer that stays on. An old laptop, a mini PC or a home server is fine. The minimal install does not need much power, but local speech recognition is faster on a stronger machine.
Docker, if you take the container route.
API keys for any cloud models you choose. Keep these keys in the configuration file on your machine and never paste them into public places.
Your Xiaozhi device, already working with the hosted service.
A way to tell the device where your server is. In practice, this means pointing the firmware at your server's OTA address and WebSocket address, either in the device settings or by rebuilding the firmware with your addresses.
Step by step
Start with the minimal Docker install. Follow the deployment document in the repository and get the server running on your computer.
Fill in the configuration file. Choose the free modules first, so you only need one or two keys.
Use the test tools before you touch the board. The project includes an audio interaction test page and a script that measures how fast your speech, language and voice modules respond. If the test page cannot talk to your server, the board will not either.
Point the device at your server. Restart it and watch both the device log and the server log as you speak.
When it works, change one thing at a time. Swap the language model, then the voice, then add memory. If something breaks, you will know which change caused it.
How an AI coding tool helps
Setting up a server like this involves many small steps: Docker commands, configuration files, ports, firewall rules and log reading. A terminal agent or AI code editor is good at exactly this kind of work, as long as you stay in charge.
Open the project folder in your editor or start an agent in it. Ask it to read the deployment document and explain the steps for the minimal Docker install on your operating system. Then ask it to prepare the configuration file with placeholders for your keys, and fill in the keys yourself. When a container fails to start, ask the agent to read the logs and explain the error before it changes anything.
You can also ask an agent to write a small plugin or MCP tool. For example, a tool that reads your calendar for today, or one that switches a smart plug through your home automation system. Ask it to follow the plugin examples already in the repository, and test the tool on its own before you let the model call it.
Pitfalls
Do not expose the server to the internet without protection. The project warns that it has not passed a security review and should not be used in production. If you open it to the public web, put it behind authentication and a firewall, or use a private network tunnel instead.
Watch your API costs. Some modules call paid services every time you speak. Set spending limits with each provider.
Keep keys out of shared files. If you put the project in a Git repository, make sure the configuration file with keys is not committed.
Check the speed of each module. A slow voice or a slow model makes the assistant feel broken, even when it works. The included speed test helps you find the slow part.
Respect the people around you. Voiceprint recognition and memory store personal data. Tell people in your home what the device records, and give them a way to delete it.
Variations
Run everything locally. Use local speech recognition, a local language model on a strong computer and a local voice, so nothing leaves your network.
Connect the assistant to your own AI agent through MCP, so a spoken request can start a task on your computer.
Add a knowledge base with your household manuals, recipes or study notes.
Try the Java or Go versions of the server, which other community members maintain, if they fit your skills better.
The short version
xiaozhi-esp32-server is an open-source back end for Xiaozhi voice devices. It lets you choose each part of the speech pipeline, add memory, tools, a knowledge base and speaker recognition, and keep your data at home. Get the device working with the hosted service first, start with the minimal Docker install and the free modules, use the test tools before the board, change one thing at a time and keep the server off the open internet unless it is properly protected.
Project resources: https://github.com/xinnan-tech/xiaozhi-esp32-server