Talking toys are not new. What is new is a toy that can listen, think and answer in a natural voice, with a personality you choose. Until recently, building one meant gluing together a speech recogniser, a language model, a voice generator and a server, and then fighting delays at every step.
ElatoAI is an open-source project that packages this into one stack. A small ESP32-S3 board inside the toy streams your voice to an edge server, which talks to a real-time voice model and streams the answer back. The project supports several speech-to-speech services, including the OpenAI Realtime API, the Gemini Live API, xAI's Grok voice agent, ElevenLabs conversational agents and Hume's voice interface, plus pipelines that combine separate speech, language and voice models.
This guide explains what the project does, how its parts fit together, what you need and how a beginner could build a first version.
What ElatoAI is
ElatoAI has three main parts.
The device client. Firmware for the ESP32-S3, written with the Arduino framework and built with PlatformIO or the Arduino IDE. It records audio, compresses it with the Opus codec, sends it over a secure WebSocket and plays back the reply.
The edge server. Small functions that run close to the user, on Deno's edge platform or on Cloudflare Workers. They hold the WebSocket connection with the device and talk to the voice model provider. Your API keys live here, not on the toy.
The web app. A Next.js app where you create characters, choose voices and personalities, see transcripts of past conversations, manage devices and control things like volume. User accounts and data live in a Supabase database.
The project also lists features that matter in a real product: over-the-air firmware updates, a captive portal for Wi-Fi setup, factory reset from the web app, button or touch control, conversation history, tool calling from the device and a pitch control for cartoon-style voices. It says the device does not need PSRAM, which widens the range of boards you can use.
How it works in plain words
Picture a phone call between the toy and an AI. The toy holds the phone. The edge server is the operator who connects the call. The voice model is the person on the other end.
You press a button or touch the toy. The board opens a secure connection to the edge server. As you speak, the board sends small packets of compressed audio. The edge server passes them to the voice model. The model listens, decides when you have finished, and starts speaking back almost at once. The audio streams back to the toy and plays through its speaker.
Because the model hears and speaks directly, rather than going through separate text steps, the conversation feels more natural. The project also uses server-side voice activity detection, so the model knows when you have stopped talking.
What you need
An ESP32-S3 board. A development board with plenty of flash is a safe choice.
An I2S microphone module and an I2S amplifier with a small speaker, unless your board has them built in.
A button or a touch pad to start a conversation, and a battery with a charging board if the device must be portable.
A soft toy, a 3D-printed case or any shell you like.
Accounts for the services you use: a database, an edge function host and at least one voice model provider. Each provider charges in its own way, so read the pricing before you start.
How a beginner could build one
Begin with the quick start in the project documentation. It walks through setting up the web app, the database and the edge server, then flashing the firmware.
Build the electronics on a breadboard before you put anything in a toy. Wire the microphone, the amplifier and the button to the pins named in the firmware configuration. Short, tidy wires make audio problems less likely.
Flash the firmware with PlatformIO. The documentation has guides for both PlatformIO and the Arduino IDE.
Connect the device to Wi-Fi through its setup portal, register it in the web app and pick a character.
Press the button and say hello. When the basic loop works, move the parts into the toy.
Using AI coding tools along the way
This project has many moving parts, which is where an AI code editor or terminal agent helps most. Open the repository and ask the agent to explain the folder layout and how the device, edge server and web app talk to each other. Ask it to trace one conversation from the button press to the reply. That gives you a map before you change anything.
When you wire your own board, ask the agent to find where the pin numbers are defined and to update them for your microphone and amplifier. Then build and flash, and paste the serial output back if it fails.
For the server side, an agent can help you set up environment variables, deploy the edge functions and read their logs. Keep your API keys in the provider's secret settings, not in the code. Ask the agent to check that no key is committed to the repository.
Pitfalls
Audio wiring is fussy. A wrong I2S pin, a missing ground or a long loose wire can cause silence, crackling or a voice that sounds sped up. Check each connection against the firmware settings.
Power matters. The speaker can draw sudden bursts of current. A weak battery or thin cable can cause the board to reset in the middle of a sentence.
Costs can grow quietly. Real-time voice models are billed by usage. A toy that a child talks to for an hour a day can cost more than you expect. Set limits with your provider.
Children's privacy needs care. If the toy is for a child, think carefully about what is recorded, where transcripts are stored and who can read them. Tell parents clearly how it works. Some places have laws about collecting data from children.
Network dependency. With a cloud model, the toy is silent without Wi-Fi. Make that clear to the person using it, and give the toy a friendly sound or light that shows when it is offline.
Variations
Give each toy its own character, voice and rules, and switch them from the web app.
Add tool calling so the toy can tell the time, read a story from a list or control a light.
Use the project's newer options to run on Cloudflare with a range of speech and language models, or try a local setup with local models and voices for a fully home-based toy.
Use the same stack for things that are not toys: a desk assistant, a language practice partner or a talking reminder board for an older relative.
The short version
ElatoAI connects an ESP32-S3 board to real-time voice models through an edge server, with a web app to manage characters and devices. Build the audio parts on a breadboard first, follow the quick start, keep API keys on the server, and use an AI coding tool to map the code and adapt it to your own wiring. Watch costs, power and privacy, especially for children, and test the full conversation loop before you sew anything into a toy.
Project resources: https://github.com/akdeb/ElatoAI