Most AI gadgets built on the ESP32 are thin clients. The chip listens and speaks, but the real model runs on a server far away. The esp32-ai project does the opposite. It runs a small language model on the chip itself, with no network at all, and shows the words on a small screen as they are generated.
The project describes a model of 28.9 million parameters running on an ESP32-S3 with 512 KB of fast SRAM, 8 MB of PSRAM and 16 MB of flash. It reports about 9.88 tokens per second end to end, with the model taking 14.9 MB at 4-bit precision. Those are the project's own figures. Just as important, the author is clear about what the model cannot do. This guide explains the idea in plain words, what you would need to try it and what a beginner can learn from it.
The memory problem
A language model is mostly a very large table of numbers. To run, the chip must read those numbers again and again. Fast memory on a microcontroller is tiny. The ESP32-S3 has 512 KB of SRAM, which is far too small to hold a model of this size.
The chip does have other memory. PSRAM is larger and slower. Flash is larger still and slower again. Normally flash is for storing programs, not for reading model weights many times per word.
How the project fits the model
The trick is to look at how often each part of the model is read.
Some values are touched many times for every token, such as the working numbers being calculated and some small normalisation weights. These live in the fast SRAM.
The dense core of the model and its output layer are read once for each position. These live in PSRAM.
The largest part is a set of embedding tables. In this design, most of the parameters sit in these tables, and the model only looks up a few rows for each token instead of reading the whole table. The project says it reads about six rows per token, roughly 450 bytes. So the 25 million parameter table can stay in slow flash, and only the tiny pieces needed are pulled in.
This approach is called Per-Layer Embeddings. It comes from Google's Gemma 3n work, where it was used to fit models on phones. Here it is applied to the memory layout of a microcontroller.
In plain words: keep the things you read all the time close by, keep the things you read rarely far away, and design the model so that most of it is the kind of thing you read rarely.
What it does and what it does not
The demo model was trained on TinyStories, a dataset of short, simple stories. It writes short stories and mostly keeps them coherent. The project offers other small models too, including one tuned to answer questions about espresso.
The author states that the story model will not answer general questions, follow instructions, write code or know facts. The memory trick changes how big a model can fit, but the part that does the reasoning is still very small. That honesty is useful. It tells you this is a lesson in architecture, not a replacement for a chat assistant.
The same repository also includes a different kind of demo: a port of part of a fruit fly's brain wiring, with tens of thousands of neurons, that reacts to a looming spider on screen. It is another example of fitting a large structure into the flash of a small chip.
What you need to try it
An ESP32-S3 board with 16 MB of flash and 8 MB of PSRAM. Boards with less memory will not hold the model.
A small display wired to the board if you want to see the output, as in the demo.
A computer that can run the project's scripts and the ESP-IDF build tools.
The repository separates downloading from flashing. One script downloads a chosen model, checks it against a fixed checksum and size, and installs it only if every check passes. Another script builds the firmware with that model and flashes the board. The board holds one model at a time.
How a beginner can approach it
You do not need to understand every detail of the model to learn from this project.
Start by reading the README slowly. Then open an AI code editor in the repository and ask it to explain the memory tiers in your own words: which data goes where, and why. Ask it to point to the code that reads an embedding row from flash.
Next, run the scripts as documented on a board that meets the memory requirements. Watch the output speed on the screen and compare it with the project's figures. Do not expect to match them exactly on different hardware or settings.
Then try small changes. Ask the agent to help you change the prompt the story model starts from, or to print timing information to the serial monitor.
Pitfalls
Wrong board. The most common failure will be a board without enough flash or PSRAM. Check the exact module on your board, not only the board name.
Expecting a chatbot. The model writes simple stories. Set your expectations by the README, not by excited summaries elsewhere.
Flash wear and time. Flashing a large model takes time. Do not reflash in a loop while you experiment with code that does not touch the model.
Copying claims. Some write-ups of this project add their own speed or capability claims, or mix it with other ideas. Trust the repository's own numbers and limits.
Variations and ideas
Train a tiny model on your own narrow topic, such as plant care tips or commands for a single device, following the same layout, and see whether it stays useful.
Combine a tiny on-chip model with a cloud model: let the chip handle simple fixed phrases offline and send harder requests to a server.
Use the same memory idea for other large lookup tables, such as dictionaries or sound samples, in projects that are not about AI at all.
The short version
esp32-ai runs a small language model entirely on an ESP32-S3 by keeping its large embedding tables in flash and reading only a few rows per token. It needs a board with 16 MB of flash and 8 MB of PSRAM. The story model writes simple stories and does not answer questions or follow instructions, and the project says so clearly. Treat it as a hands-on lesson in how memory shapes what AI can run on tiny hardware, and use an AI coding tool to help you read the code and try small changes.
Project resources: https://github.com/slvDev/esp32-ai