We measured where the time goes on an ESP32-S3 and rewrote language-model inference for this chip. On the same board and the same weights our engine produces the same text as esp32-ai, several times faster. Our own models, designed for this chip, write better than theirs at the same speed.
The texts match character for character: 759 of 759. Only the time differs: 18.8 s vs 2.3 s.
To write one token the model has to read all of its weights. The chip has three kinds of memory whose speeds differ almost 10x, and the PIE vector unit computes 21x faster than plain code. Once PIE is in use, the processor mostly waits for data.
That gives the rule behind all of the work: speedup = fewer bytes per token. Bytes per token = bits per weight × fraction of weights read ÷ tokens per weight pass. Each technique below shrinks one of these factors.
A per-operation cost model: reads from each memory plus compute. On third-party measurements it was not fitted to, the error is 3–5% (only two check points); on our engine it underestimates time by 10–14%. It let us drop ideas on paper before writing code.
Our own benchmark firmware: SRAM, PSRAM and flash speed, PIE kernels, DMA. Findings: PIE is 21x faster than plain C (~13x faster than esp32-ai's kernel); after it the bottleneck is PSRAM (83 MB/s on two cores); the PSRAM bus gives ~90 MB/s in total, so DMA adds no bandwidth.
27 models trained on a laptop, every compared variant in two runs (except one repeated 2-bit run). Per-layer embeddings (PLE) make the model smarter for the same bytes read; ReLU lets 79% of neurons be skipped at no cost; training for 4 bits is almost free; 4–6 layers beat 12.
C code shared by PC and board, plus PIE assembly kernels. Three versions: 70 → 98 → 110 tokens/s on our model. Every change was measured on its own; failed ones were reverted and logged.
Their published weights, not requantized. Their firmware and our engine on one board, on their own benchmark: 94.1 vs 11.5 ms per token, identical text.
Speculative decoding does not pay on this chip (negative result). Batching tokens known in advance (the prompt, unambiguous pieces of JSON) gives 1.5–2.8x.
The cost model, calibrated on the board, predicted the speed of the variants within 1–5%. The best variant (vocabulary 8192, PLE table twice as wide, 8.6M parameters), trained on 70M tokens, reaches 0.63 bits per byte vs 0.86 for esp32-ai's 29M model; our 3.4M model already beats it at almost the same speed.
| Technique | Quality (val loss) | Speed (board or cost model) | Verdict |
|---|---|---|---|
| PLE: a per-token table in flash, one row read per token | −0.022…−0.039 | +0.5 ms, board | works |
| ReLU instead of SwiGLU, skipping silent neurons (79%) | −0.002 | 1.3x, board | works |
| Training for 4 bits from the start (QAT) | +0.024 | ~7x fewer bytes than fp32 (incl. scales) | works |
| Mixture of experts (4 experts, 1 read) | −0.009 | ≈, cost model | weak |
| 2 bits at the same size (with twice the weights and the same bytes read: +0.13) | +0.305 | 1.5x, cost model (not run on the board) | too costly in quality |
| 12 layers instead of 6 | +0.069 | −15%, cost model | worse on both |
The combined model combo = PLE + ReLU + 4 bits: val loss 2.010 vs 2.024 for a plain 4-bit model (mean of two runs, better in both), and 22% faster on the board (98 vs 80 tok/s, engine v2).
Models with different vocabularies are compared in bits per byte of text: how much information the model is missing to predict each byte. First 5,000 held-out stories, each model with its own tokenizer and its own 4-bit weights, as the engine runs them. Speed is the mean over the whole 256-token context on our engine (two-stage output layer, PSRAM 80 MHz).
| Model | Parameters | Bits per byte | tok/s (mean, board) |
|---|---|---|---|
| esp32-ai TinyStories (their open model) | 28.9M | 0.863 | ~94 |
| combo (ours: PLE + ReLU + 4 bits) | 3.4M | 0.732 | ~91 |
| PLE table twice as wide | 4.9M | 0.716 | ~84 (previous engine version) |
| vocabulary 8192 + PLE twice as wide | 8.6M | 0.704 | ~81 |
| same, trained on 70M tokens instead of 20M | 8.6M | 0.625 | ~81 |
| vocabulary 16384 + PLE twice as wide | 16.0M | 0.706 | — (predicted ~74) |
Fairness check: our score for their model (2.45 nats per token) matches what its authors publish (perplexity 11.4 before quantization, "~+1" after). Almost all of their parameters are a 32k-token vocabulary and the PLE table, with a trunk of only 0.56M; children's stories barely use such a vocabulary, and our 16384-vocabulary variant showed that the PLE table saturates. We run their model only on our engine: their firmware runs it at 10.6 tok/s and cannot run our models.
Weights are stored in 4 bits and unpacked by vector instructions. esp32-ai uses 8 bits and plain C: twice the bytes and ~13x more cycles per operation (6.2 vs 0.47).
Two-stage output layer: 256 clusters of tokens, score the clusters first, then compute only the best ones in full. Agrees with the full output layer at 99.4–99.8% of positions; the rows that are read match the full computation bit for bit.
Every matrix-vector product is split between the cores; the second core spins waiting for work, so handing it a job costs 83–93 cycles.
Engine buffers first, then the most-read weights go to fast internal memory (~210 KB), the rest to PSRAM; the PLE table stays in flash.
ReLU "gate first": compute the gate, then read the other two matrices only for live neurons. A new kernel on the QACC accumulator for the transposed matrix.
Work outside the matrix products: quantization, GELU by table, cluster selection. Guided by the profile, not guesses, it went from 2.2 to 1.2 ms per token.
We built esp32-ai's firmware their way (arduino-cli, -O3, their flash partition) and measured it on our board. The model fingerprint a9bdd778 matched their metadata, and three runs gave the same value. Their own published figure is 94.9 ms; we measured 94.1 ms.
| Engine | Output layer | PSRAM | ms/token | tok/s | Speedup |
|---|---|---|---|---|---|
| esp32-ai | full, 8-bit | 80 MHz | 94.10 | 10.6 | 1x |
| ours | full, 4-bit | 80 MHz | 25.00 | 40.0 | 3.8x |
| ours | two-stage | 80 MHz | 11.52 | 86.8 | 8.2x |
| ours | full | 120 MHz* | 18.90 | 52.9 | 5.0x |
| ours | two-stage | 120 MHz* | 9.17 | 109.0 | 10.3x |
* PSRAM at 120 MHz is marked experimental in ESP-IDF ("works at a stable temperature"). Measured at room temperature, no faults seen.
| Idea | Expected | Measured | Why |
|---|---|---|---|
| Guess 2–4 next tokens and verify them in one weight pass | 1.3x by the calculator | 0.8–1.05x | after the optimizations a third of the time is per-token compute; the drafter yields only 1.4–1.6 tokens per pass |
| DMA prefetches weights while the core computes | −30% | no faster | the PSRAM bus gives ~90 MB/s in total; DMA (77 MB/s) only shares it with the cores, and two cores already read 83 |
| 16-bit weight scales instead of 32-bit | −6% bytes | +5% time | converting 8192 numbers per token costs more than the read it saves |
| Keep recent attention values in SRAM | −1 ms | −0.06 ms attention | the writes were not the bottleneck, PSRAM read latency was (−0.39 ms in total, the rest unexplained) |
| Weights from a seed generator (SeedLM), weight compression | a win on GPUs | a loss on paper | the chip has no spare cycles: decompression costs more than the read |
| Project | Where the model runs | Speed on ESP32-S3 | Has | Lacks |
|---|---|---|---|---|
| This project | ESP32-S3, fully offline | 87–110 tok/s (29M), 110–134 (2M); ~72 at the end of the context | PIE kernels, 2 cores, two-stage output layer, sparse FFN, batching, cost model, bit-exact checks | an applied model (children's stories so far) |
| slvDev/esp32-ai | ESP32-S3 | 10.6 tok/s (measured by us) | TinyStories and Barista models, PLE | PIE, 4-bit execution, work skipping |
| esp32-tinyLLM (Rust) | ESP32-S3 | ~10.5 tok/s (their figure) | bit-exact test methodology | vector unit, large byte savings |
| Needle ports (tool calling) | ESP32-S3 | ~1.5 tok/s (their figure) | schema-based tool calls | speed |
| Deeploy (compiler) | RISC-V MCU with NPU | — | tiling, DMA, small LMs | ESP32 / Xtensa PIE |
On the same weights and the same text: 3.8x with identical computation and 8.2x with the two-stage output layer. We found no other open LLM engine using PIE.
Architectures compared at equal bytes read per token and checked on the board. Our 3.4M model writes better than esp32-ai's open 29M model at almost the same speed on our engine (0.73 vs 0.86 bits per byte); the best one reaches 0.63.
The ideas come from server and mobile LLM inference: candidate screening for the output layer, ReLU activation sparsity and reading only active rows (LLM in a flash), per-layer embeddings. We did not find them measured on a microcontroller, and show which help here, which hurt, and why.
A cost model validated on data it was not fitted to, bit-exact agreement between PC and board, every kernel checked against a C reference, every experiment repeated.
"Not found" means a search of GitHub, Hugging Face, arXiv and patents; it is not proof of absence.
PC, the board with PIE on two cores, on one core, and in plain C give identical digests of all tokens and all logits in every run.
The engine's loss on held-out text differs from PyTorch by at most 0.003. The chosen token matches at 98–99% of positions; differences only at near-ties.
Spread between repeats of the same run. This is not the spread across texts: speed drops 1.6x from the start to the end of the context. All measurements are kept as logs in the repository.
An ESP32-S3-N16R8 board connects to any computer over USB. A page in Chrome or Edge flashes the board and then talks to it directly, nothing to install. You type the start of a story, the board continues it in real time, and the page shows the time of every token.
All models were trained on children's stories. Two images are offered: our varC model (8.6M, 0.63 bits per byte) and esp32-ai's model on our engine. The serial protocol is in engine/demo/README.md.
PC → G 120 700 40 <seed> 0 7 433 447 259 405 … board → P 29.6 board → T 11 10320 board → T 612 10288 … board → E 120 10.32
The models are small (3–29M parameters) and trained on children's stories, 20M tokens per model (70M for the best one), one learning rate for all; the quality comparison with esp32-ai uses text from our distribution (their score on it matches their own); speed depends on context length ("110 tok/s" is positions 13–60; the esp32-ai comparison uses their benchmark, positions 4–203); PSRAM at 120 MHz is experimental; energy per token was not measured, and the second core busy-waits; the Arduino core version for the esp32-ai measurement was 3.3.3 (the authors used 3.3.10).