Menu

On-Device AI

On-Device AI: Models That Run on the Product, Not in the Cloud

NotesNovember 19, 202616 min readSenova Systems Research

On-device AI answers in 1–50 ms; the cloud answers in 50–500 ms, and when the network is down, the cloud never answers at all. A visual wake-words model runs in 38 ms for 387 microjoules on a microcontroller, and quantizing it from FP32 to INT8 makes it four times smaller for under 1% accuracy. The radio is the battery's biggest expense, not the model.

First, a simple picture

A brain in the head, not a phone call to a brain somewhere else.

A person who sees a ball coming does not phone a friend, describe the ball, wait for the friend to think, and then catch it. The thinking happens where the seeing happens, in the head, in milliseconds. A person who needs to write a novel does phone a friend: the job is too big for one head, and nobody is in a hurry.

On-device AI, also called edge AI, is the model that lives inside the product instead of phoning home. The trained model runs on the device itself: a microcontroller, a phone, a camera, a robot. The data is born on the device, the thinking happens on the device, and the answer comes back in milliseconds. Cloud AI is the phone call: the device sends the data to a data centre, a big model thinks about it there, and the answer comes back over the network.

Inference is the word for the thinking part. Training is teaching the model; inference is the model doing its job on new data. This article is about inference, where it runs, what it costs, and when the phone call is worth making.

What on-device AI really is

A model is a list of numbers, weights, plus the arithmetic that turns an input into an answer. The size of a model is the size of that list. A parameter is one number in the list. A small language model has billions of parameters. A TinyML model, the field of running machine learning on microcontrollers, has under 2 million weights, and runs on chips that draw sub-milliwatt to low-milliwatt power.

The constraint is the whole story. A microcontroller has kilobytes to a couple of megabytes of memory, not gigabytes. An STM32H7, a practical high-end Cortex-M7, has 512 kB of SRAM and 2 MB of flash, running at 480 MHz.

An ESP32-S3, the $8 prototyping favourite, has 512 kB of SRAM plus up to 8 MB of PSRAM. The model must fit in that memory, run in that power budget, and answer in time, and the model that fits is a small one, trained for one job: is there a person in this picture, is that the wake word, is this vibration normal.

MLPerf Tiny is the benchmark that makes these claims comparable. Built by the MLCommons Tiny Working Group with EEMBC, more than 50 organisations from industry and academia, including Harvard, Google, STMicroelectronics, Qualcomm, Syntiant, Renesas, Infineon, CERN, and Silicon Labs, over roughly 18 months, it runs the same four tasks on every device: image classification, keyword spotting, visual wake words (is there a person in this picture), and anomaly detection. It measures three things, accuracy, latency, and energy per inference, and each task carries a fixed quality target, so submissions are compared on speed and energy at equivalent accuracy rather than trading one against the other. Energy is treated as a first-class metric, because in TinyML a device that runs an inference in 10 ms at twice the joules of a competitor is often the wrong choice: battery life is the binding constraint.

The four tasks are the whole field in miniature. Keyword spotting is the wake word, "hey" something, that a device listens for around the clock without sending any audio to the cloud. Visual wake words is the person detector. Image classification is the sorter. Anomaly detection is the watchman that knows what normal sounds like and raises its hand when it doesn't. If your product's job is one of those four, on-device AI is not a research project. It is a solved problem with a benchmark.

When it wins: latency, privacy, battery, dead zones

Where the answer is computed decides how fast it arrives (published 2026 edge and cloud comparisons). At 60 mph, 200 ms is 5.4 m of travel.

WhereTypical round trip
On the device1 to 10 ms
Edge, one short LAN or Wi-Fi hop5 to 50 ms
Cloud, same region50 to 200 ms
Cloud, another region or overseas100 to 300+ ms

Latency. Edge inference typically lands at 1–50 ms; cloud inference typically lands at 50–500+ ms once the network is included. The ladder, from the published 2026 comparisons: on-device 1–10 ms (control loops, on-device vision, safety-critical tasks); edge over a short LAN or Wi-Fi hop 5–50 ms; regional cloud 50–200 ms; long-distance cloud 100–300+ ms. Light in optical fibre covers roughly 200 kilometres per millisecond, so a data centre 1,000 kilometres away adds about 10 ms of round-trip propagation before any queueing or computation, and practical network round trips to a cloud endpoint run 20–80 ms. A concrete case from the comparisons: an INT8-quantized YOLOv8n model running on edge hardware hits sub-3 ms inference latency, which supports perception rates above 300 Hz and total closed-loop latency under 20 ms. Push that same pipeline to the cloud, and you add 100 ms or more.

The physics makes the point better than the numbers do. A cloud-based inference call takes 100–500 ms round-trip. At 60 mph, 26.8 metres a second, a 200 ms delay means the car has travelled 5.4 metres before the answer arrives.

For a manufacturing line spitting out 600 parts a minute, one part passes every 100 milliseconds. The rule the guides state plainly: if your application needs predictions in under 100 ms, edge AI is likely the only option, network round trips add 50–500 ms of latency before the model even starts running. Hard real-time applications requiring responses within 10 ms must run at the edge.

Privacy. On-device inference keeps the data where it is created. The raw image, the raw audio, the raw sensor stream never leaves the device. That is not a feature. It is the absence of a risk: no data to intercept, no data to store, no data to leak, and no sovereignty rule to break. A wake-word detector that listens around the clock is a different product when the audio never leaves the room.

Battery. The radio is the battery's biggest expense, not the model, the demo proves it below. And the corollary: a device that only thinks when something happens, and only transmits when the thinking says so, can run for years on a coin cell. A dedicated neural processing unit spotted keywords 25× faster than the median Cortex-M4, per the analysis of 243 MLPerf Tiny results, and the fastest chips now run visual wake words at 22.2 µJ per inference, which the vendor's own arithmetic puts at over three years on a single CR2032 coin cell at one inference a second.

Dead zones. When the network is down, the cloud check never answers. The on-device check answers in 38 ms, exactly as it always did. A fence-post sensor in a field, a machine on a factory floor with flaky Wi-Fi, a wearable in a basement: the product that needs the cloud to think is a product that stops thinking when the network does.

When it loses: size, accuracy, and the job that is too big

On-device AI loses on three things, and the honest article names all three.

Size. The model must fit in the device's memory. TinyML models run 10 KB to 500 KB; a keyword spotter is about 25 KB; a MobileNetV2 at 96×96 is about 320 KB. Anything needing more than about 2 MB of model weights does not belong on a microcontroller. A large language model does not fit, and the small ones that do fit are small for a reason. Google's Gemma 4 family launched in 2026 with explicit edge variants, the E2B (2 billion parameters) and E4B (4.5 billion effective parameters), and Liquid AI's LFM2.5-VL-450M, a 450-million-parameter vision-language model, hits sub-250 ms inference on hardware from NVIDIA Jetson Orin boards to Snapdragon phones. These are real, and they are the ceiling: a 450-million-parameter model is the biggest thing that runs on the edge today, and it runs on a phone or a Jetson, not on a microcontroller.

Accuracy. The small model is less accurate than the big one, and the quantization that shrinks it costs a little more. The numbers are in the next section, under 1% for the standard path, more for the hard cases. The question is never "is the small model as accurate as the cloud model." The question is "is the small model accurate enough for this job, and is the cloud model's extra accuracy worth the latency, the battery, and the dead zone." For a person detector, 89.6% is enough. For a medical diagnosis, it is not.

Retraining and updates. The cloud model can be retrained overnight and redeployed in minutes. The on-device model is baked into firmware, and updating it means updating the firmware, which means the update path from Article #5's board design and Article #7's enclosure (can a technician open it, replace the battery, and close it to the same rating?) matters to the AI, not just to the hardware. A model that cannot be updated is a model that rots, and the rotting is silent, which is Article #13's failure mode, applied to the model instead of the workflow.

Quantization

Quantization: the same model, four times smaller, for under 1% accuracy (FP32 to INT8)

EffectFigure
Size4× smaller (32 to 8 bits a weight); INT8 files 60 to 75% smaller
Speed1.9 to 3.8× faster than FP32 on integer hardware (MLPerf Tiny)
Accuracy0.2 to 0.8% mAP lost (YOLOv8, ResNet) with per-channel post-training quantization

Quantization is the process of reducing the numerical precision of a model's weights and activations, from 32-bit floating point (FP32) to 8-bit integers (INT8), or lower, so the model fits in less memory, runs faster, and uses less power. It is the single most important technique in on-device AI, and it is nearly free.

The arithmetic is simple: FP32 stores each weight in 4 bytes; INT8 stores it in 1 byte. The model becomes 4× smaller. Integer arithmetic is faster and cheaper than floating-point arithmetic on the hardware that runs it, so inference speeds up 2–4× on integer-only hardware, and the analysis of 243 MLPerf Tiny results found INT8 made Cortex-M inference 1.9× to 3.8× faster than FP32 with no loss in classification accuracy.

A real hardware test on a Snapdragon 8 Gen 3, running a MobileNet-style person-detection model, cut the model file from 1.07 MB to 322 KB, a 70.5% reduction, while keeping inference latency comparable.

The accuracy cost, measured on the standard path: post-training quantization (PTQ) with per-channel weight quantization typically loses under 1% accuracy on classification models, and needs only a small calibration dataset (100–500 representative images). For standard convolutional vision models like YOLOv8 or ResNet, PTQ with KL-divergence calibration loses 0.2–0.8% mAP (mean average precision) against FP16, while halving the memory footprint and doubling throughput. Quantization-aware training (QAT), simulating quantization during training so the weights adapt, loses under 0.5% at INT8, at the cost of a full retraining cycle.

Per-tensor INT8 (one scale factor for a whole layer) loses 1–3%; per-channel INT8 (one scale factor per output channel) loses under 1%. The guides' verdict: do not fear INT8; demanding FP16 execution on low-power edge devices wastes 50% of your memory bandwidth and quadruples thermal dissipation for an imperceptible gain.

Two traps, both real. First: naive INT8 frequently causes severe accuracy degradation on vision transformers and modern language models, the architectures whose activations have outlier values that a single scale factor cannot represent. Per-channel quantization fixes most of it; the rest needs QAT or a mixed-precision approach.

Second: the operator fallback. The NPU accelerates only the operators it supports; when a quantized model uses an operator outside that set, the runtime silently falls back to CPU execution, and latency increases 3–10×. The model file is smaller, the accuracy is fine, and the device is slower, and nothing tells you.

The test for this is the same as the test for everything else in this track: run it on the real hardware, measure it, and believe the measurement.

The scale ladder, for orientation: a 70-billion-parameter language model in FP16 needs 140 GB of memory, far beyond any consumer GPU. With 4-bit quantization it fits in 35 GB and can run on two RTX 4090 GPUs. A keyword spotter is 25 KB. Between those two numbers is the whole field, and the job decides where on the ladder you stand.

The benchmark numbers, read honestly

The MLPerf Tiny v1.4 results (July 2026) and the published analyses give us numbers that can be checked, and the honest way to read them is as a range, not a leaderboard.

  • A cross-layer optimization on an STM32H743ZI (Cortex-M7, 480 MHz, 512 kB SRAM, 2 MB flash) reaches 91.7% image classification, 95.4% keyword spotting, 89.6% visual wake words, and 0.913 anomaly-detection AUC, using 198 kB of flash and 174 kB of peak SRAM, at 387 µJ per inference and 38 ms latency (MDPI, July 2026, a peer-reviewed paper, labelled as such). The MCUNet baseline on the same chip spends 923 µJ per inference on visual wake words.
  • The record for visual wake words in MLPerf Tiny v1.4 is 22.2 µJ per inference (ASYGN ColibryNPU, July 2026, a vendor submission, labelled as such), with the vendor's claim of over three years on a CR2032 at one inference a second.
  • The preprint analysis of 243 MLPerf Tiny results (v1.0–v1.3) found energy per inference tracks latency closely (log-log slope 0.97), and that 4-bit weights raised validation loss by only 0.09–0.55%, a preprint, labelled as such, not yet peer-reviewed.
  • One vendor's filing claims 17× to 56× lower energy per inference than competitors on MLPerf Tiny, a vendor press claim, labelled as such, and not a benchmark result.

The numbers that matter for a product decision are the middle of the range, not the record: a few hundred microjoules and a few tens of milliseconds per inference, on a microcontroller that costs a few dollars. The record is the ceiling. The middle is the product.

Example Labs is a made-up company we use for worked examples.

DEMO

This product is made up. Call it the Example Labs field sensor hub again: the fence-post unit from Articles #6 and #7, the weather sensor, the event camera from Article #2, the battery from Article #6, the enclosure from Article #7, now with a job: watch a gate, and tell the office when a person walks through it. The demo's arithmetic is the article's own, verified before writing, on the published figures cited above; the hub and its schedule are the worked example, not a measured product.

Step 1, The job. The hub must decide, once a second, whether a person is in the camera's view. The model is a visual wake-words CNN, the MLPerf Tiny task, running at 89.6% accuracy, 198 kB of flash, 174 kB of peak SRAM. The battery is a CR2032: 230 mAh at 3 V, which is 2,484 joules of energy. The hub checks once a second, 24 hours a day.

Step 2, On-device: 74 days. The model runs on the hub's microcontroller: 387 µJ per inference, 38 ms. At one check a second, the battery lasts 2,484 ÷ 387 µJ = 74 days, and that is compute-only, before the idle losses, so it is a ceiling, not a promise. The image never leaves the device. The answer arrives in 38 ms. When the network is down, the answer still arrives in 38 ms.

Step 3, Cloud: 23 hours. The hub sends the image to the cloud instead. One Wi-Fi transmission costs about 30,000 µJ (Article #6's radio figure), and that is before the cloud even runs the model. At one check a second, the battery lasts 2,484 ÷ 30,000 µJ = 23 hours. The cloud check costs 77× the energy of the on-device check. The answer arrives in 100–300 ms, when the network is up. When the network is down, the answer never arrives.

Step 4, The event camera changes everything. The hub's eye is an event camera (Article #2): it only reports change. Most of the time, nothing moves, and the model sleeps. Suppose movement happens 5% of the time. The on-device hub now lasts 74 ÷ 0.05 = 1,486 days, about 4.1 years. The cloud hub lasts 23 hours ÷ 0.05 = 19 days. The gating is what makes the product possible, and the gating is only possible because the model lives where the data is born, the event camera's silence is the trigger, and the trigger only works if the thinking is local. And the neuromorphic option (Article #1): Loihi 2 runs 2,400 inferences per joule, which is 417 µJ per inference, the same order as the microcontroller's 387 µJ, on a chip that thinks in spikes. The neuromorphic chip is not magic. It is one more way to run the same model, at the same energy, on different hardware.

On-device, gated by event camera1497.0 daysOn-device (387 µJ a check)74.0 daysCloud, gated by event camera19.0 daysCloud (30,000 µJ a check)1.0 daysLog scale. Longer is better.
The worked example's hub on one CR2032 (2,484 J), one check a second: battery life, compute only

The decision test

Four questions. They sort the products that should think on-device from the ones that should phone home:

  1. Does the answer need to arrive in under 100 ms, or under 10 ms, for a control loop or a safety decision? Yes, the cloud cannot answer in time, and the model belongs on the device. No, the latency is not the constraint, and the cloud is still in the running.
  2. Must the product work with no network, a dead zone, a basement, a field, a factory floor with flaky Wi-Fi? Yes, the model belongs on the device, because the cloud check never answers when the network is down. No, the network is reliable, and the cloud is still in the running.
  3. Does the raw data have to stay on the device, audio, images, sensor data that cannot leave the site? Yes, the model belongs on the device, because the data never leaves it. No, the data can travel, and the cloud is still in the running.
  4. Does the job fit in the device, the model under about 2 MB, the accuracy good enough at INT8, the update path through firmware? No, the job is too big, and the cloud is the honest answer, or the product needs a bigger chip. Yes, the model fits, the accuracy holds, and the update path exists.

If the answers are yes, yes, yes, yes, the model belongs on the device, and the assessment is a final check, not a rescue. If the answer to question 4 is no, the model is a 450-million-parameter vision-language model, or the accuracy at INT8 is not enough, the cloud is the honest answer, and the assessment says so. And if the answer to question 1 is "the job can wait a second" and the answer to question 2 is "the network is reliable", the cloud may be the cheaper answer, and the assessment will say that too.

The job decides. Not the fashion.

"But the cloud model is smarter", and two more fair doubts

"The cloud model is smarter." It is. The cloud model is bigger, more accurate, and retrained overnight. The question is not whether it is smarter. The question is whether its extra accuracy is worth the latency, the battery, and the dead zone, and whether the product's job needs it. A person detector at 89.6% that answers in 38 ms beats a person detector at 98% that answers in 300 ms, when the job is to catch a person walking through a gate. The smarter model is the one that answers in time.

"We can just send the data to the cloud." You can, and the battery will last 23 hours instead of 74 days, the answer will arrive in 100–300 ms instead of 38 ms, and the product will stop thinking when the network does. Sending the data is not free. The radio is the battery's biggest expense, and the network is the product's biggest dependency. The cloud is the right answer for the job that can wait, that has a reliable network, and that needs a model too big to fit. It is the wrong answer for the job that cannot wait.

"Our model is too big to fit on a microcontroller." Then the question is whether the model needs to be that big, or whether the product needs a bigger chip. Quantization takes a model from FP32 to INT8 at 4× smaller for under 1% accuracy. Pruning and distillation take it smaller still. A Jetson Orin or a Snapdragon phone runs a 450-million-parameter vision-language model at sub-250 ms. The ladder has rungs between the microcontroller and the data centre, and the assessment's job is to find the rung your product stands on, not to force it onto the smallest one.

Sources

  1. MLCommons, "MLPerf Tiny: Benchmarking AI at the Edge" (July 2026)
    What we used

    TinyML defined as models under 2M weights on microcontrollers drawing sub-milliwatt to low-milliwatt power; MLPerf Tiny measures accuracy, latency, and energy per inference with fixed quality targets; built by 50+ organisations over ~18 months; first results v0.5 June 2021; energy is a first-class metric because battery life is the binding constraint

  2. MDPI Electronics, "Cross-Layer Resource Optimization for Ultra-Low-Power TinyML Inference on ARM Cortex-M Microcontrollers" (July 2026, peer-reviewed)
    What we used

    STM32H743ZI (Cortex-M7, 480 MHz, 512 kB SRAM, 2 MB flash): 91.7% image classification, 95.4% keyword spotting, 89.6% visual wake words, 0.913 anomaly-detection AUC; 198 kB flash, 174 kB peak SRAM, 387 µJ per inference, 38 ms latency; MCUNet baseline 923 µJ per inference on visual wake words

  3. ASYGN, "ColibryNPU"
    What we used

    Announcement (July 2026, vendor submission), record-low 22.2 µJ per inference on MLPerf Tiny v1.4 Visual Wake Words; over 3 years on a CR2032 (220 mAh) at one inference per second; real-time video processing under 1 mW at several frames per second

  4. Preprints.org, "Subscription-Free AI: Energy-Efficient On-Device Inference of Quantized Language Models on Low-Cost Edge Hardware"
    What we used

    (November 2026, preprint, not peer-reviewed, labelled as such), analysis of 243 MLPerf Tiny results (v1.0–v1.3): INT8 made Cortex-M inference 1.9×–3.8× faster than FP32 with no loss in classification accuracy; a dedicated NPU spotted keywords 25× faster than the median Cortex-M4; energy per inference tracked latency with a log-log slope of 0.97 (R² = 0.72); 4-bit weights raised validation loss by only 0.09–0.55%

  5. Eduonix, "Edge AI: Run AI Models Locally Without the Cloud" (2026)
    What we used

    Edge inference typically 1–50 ms, cloud 50–500+ ms with network; pure edge wins for real-time control loops, offline operation, and sensitive data; cloud dominates for frontier-scale models, frequent retraining, and multi-step reasoning

  6. Nano-GPT, "Edge AI vs. Cloud AI: Cost and Latency Comparison"
    What we used

    Edge 1–50 ms, cloud 50–500+ ms; edge local 1–10 ms, LAN/Wi-Fi 5–50 ms, regional cloud 50–200 ms, long-distance 100–300+ ms; INT8 YOLOv8n sub-3 ms inference, perception above 300 Hz, closed-loop under 20 ms; the same pipeline to the cloud adds 100 ms or more; edge/cloud break-even around 30–50 GB/day for some devices

  7. BeyondTMRW, "Edge AI vs Cloud: When the Cloud Isn't Fast Enough" (2026)
    What we used

    Cloud inference 100–500 ms round-trip; edge inference 5–20 ms; voice AI cloud pipeline around 915 ms total; 63% of newly launched Android flagships carry discrete NPU cores capable of running quantized LLMs on-device (reported figure, labelled as such); Google Gemma 4 edge variants E2B (2B parameters) and E4B (4.5B effective), up to 35× faster inference on mobile with LiteRT; Liquid AI LFM2.5-VL-450M (450M parameters) sub-250 ms on Jetson Orin to Snapdragon phones

  8. TechStoriess, "Edge AI vs Cloud AI for AI Workloads: 2026 Cost & Latency Decision Guide"
    What we used

    Edge inference below 10 ms, cloud 50–200 ms for real-time applications; hard real-time under 10 ms must run at the edge; approximately 90% of IoT data is now processed locally at the edge before transmission (reported figure, labelled as such)

  9. AIPlusInfo, "Edge AI vs Cloud AI Differences"
    What we used

    Edge AI market USD 33.4 billion in 2026, up from USD 26.9 billion in 2025 (P&S Intelligence, market projection, labelled as such); light in fibre roughly 200 km per millisecond; network round trips to a cloud endpoint 20–80 ms (Spheron guide); a 3-billion-parameter on-device model can generate each token in under 20 ms, cloud LLM first token 200–500 ms (practitioner analysis, labelled as such)

  10. Edge Silicon Lab, "INT8 vs FP16 on Edge NPUs: Quantization Accuracy Loss"
    What we used

    PTQ with KL-divergence calibration: under 0.5–1.0% mAP loss on YOLOv8/ResNet vs FP16, memory footprint halved, throughput doubled; INT8 reduces mathematical energy consumption by over 75%; 4× more INT8 MACs on the same die; per-channel quantization virtually eliminates accuracy loss; ViTs and LLMs suffer severe degradation under naive INT8

  11. ML Systems Universe, "Model Optimizations"
    What we used

    INT8 PTQ per-channel under 1% accuracy loss on classification, 100–1,000 calibration samples; per-tensor 1–3%; QAT INT8 under 0.5%; QAT INT4 1–2%; LLaMA-2 70B FP16 140 GB, 4-bit GPTQ 35 GB on two RTX 4090s

  12. EdgeGate, "Model Quantization Testing for Edge AI: FP32 vs INT8 on Real Hardware"
    What we used

    Person-detection model on Snapdragon 8 Gen 3: FP32 1.07 MB vs INT8 322 KB (70.5% reduction), peak memory 121.51 MB vs 124.66 MB; operator fallback increases latency 3–10× when the NPU lacks a quantized operator variant

  13. ECN Apps, "TinyML On-Device AI Guide 2026"
    What we used

    TinyML memory 16 KB–2 MB SRAM, 64 KB–2 MB flash, microwatts to milliwatts; model size 10 KB–500 KB; a $2 microcontroller can detect keywords, classify sounds, recognise gestures, spot anomalies, entirely offline, on a coin cell for years

  14. TildAlice, "TinyML on ESP32: Ship Your AI MVP for Under $10"
    What we used

    ESP32-S3 (dual-core Xtensa LX7 @ 240 MHz, 512 KB SRAM, up to 8 MB PSRAM) runs TFLite models at 20–30 fps; keyword CNN 25 KB, 28 ms, 94%; anomaly 1D-CNN 18 KB, 15 ms, 89%; MobileNetV2 96×96 320 KB, 180 ms, 78%; TFLite Micro tensor arena 5–10× the model size

  15. SightSys, "Wake-Word Voice Recognition on STM32H7 with Edge Impulse" (May 2026)
    What we used

    Always-on wake-word stack uses ~63 KB of 2 MB flash (3%), 137 KB of 1 MB SRAM (13%), and 25% of one Cortex-M7 core, classifying every 500 ms