On-Device AI: When the Round Trip Is the Wrong Architecture

Sending everything to an API is the default for AI features, and for many products it is correct. But there are cases where the round trip is the wrong architecture, and mobile is where they concentrate.

When on-device wins

Privacy that is structural, not promised. Health, financial and intimate data carry a different standard. "We do not store your data" is a policy. "Your data never leaves the device" is an architecture, and it is the stronger claim because it does not depend on anyone's conduct.

Latency that has to feel instant. Anything responding to typing, to camera input, or to a gesture cannot afford a network round trip. Two hundred milliseconds on-device beats eight hundred over a network, regardless of which model is better.

Working offline. Connectivity is not a given everywhere, and a feature that fails on a train is a feature users stop trusting.

Cost at scale. Inference on the user's hardware costs you nothing per call. For a high-frequency feature in a consumer app, that is often the difference between a viable product and one whose unit economics never work.

When it does not

Smaller models are genuinely less capable. For open-ended reasoning, long context or broad world knowledge, a hosted frontier model will outperform anything that fits in a phone's memory budget.

On-device also means shipping the model. Binary size grows, updates require an app release rather than a server deploy, and you inherit a device matrix — the oldest supported phone sets your real performance floor, not the newest.

The hybrid that usually wins

Run the frequent, latency-sensitive, privacy-sensitive work on-device. Escalate the rare, hard requests to a hosted model — with the user's awareness when data leaves the device.

Classification, embedding, keyword extraction, transcription, suggestion ranking and simple rewriting are all well within reach locally. Long-form generation and multi-step reasoning generally are not.

Practical constraints worth knowing early

• Memory is the binding limit, not compute. A model that loads fine on a current flagship may be killed by the OS on a three-year-old device.
• Quantisation is a quality decision. Smaller weights mean faster loads and lower memory, with real accuracy cost. Measure it on your task rather than trusting a general benchmark.
• First load is the worst moment. Loading a model during app launch is a good way to be uninstalled. Load lazily, keep it warm, and show something useful meanwhile.
• Thermals are real. Sustained inference heats a phone, and the OS will throttle you. Benchmark after ten minutes of use, not on the first call.
• Battery is a review score. Users attribute drain to whichever app they were using.

Deciding

Ask what happens when the network is gone, what happens if the data leaks, and what the feature costs at a hundred times current usage. If any of those answers is uncomfortable, on-device deserves a serious look — even if it means accepting a less capable model for a better product.