Local vs Cloud AI: When to Run Models on the Device

Run a model on the user's device when the data is sensitive, the task is bounded and the feature must work offline or answer instantly. Call a hosted model when the task needs the best available reasoning, very long context or behaviour you cannot test on every device your users own. Most serious products end up with both, and the hard part is the routing rule between them.

"Local" in this guide means the model's weights run on hardware the user controls: a laptop, a phone, a workstation or a server inside their own network. "Hosted" means you send the request to a model provider's API and pay per token or per call. Neither is the modern choice and the other the legacy one. They are different cost structures with different failure modes.

The question is rarely "which model is smartest?" It is closer to: which parts of this feature need a frontier model, which parts a small model can do well when it receives good evidence, and which parts should be ordinary deterministic code? A model should not count bytes, compute hashes or decide whether a file exists. Software does that reliably. The model interprets, explains and drafts.

This guide gives you a decision table, the memory maths for local models, the trade-offs of quantisation, a neutral map of runtimes, and the hybrid patterns we use. The examples come from Neel, the local-first AI coding agent we are building. It is in development with no public release; the first installable build is planned for January 2027.

Seven factors decide most cases. Score your feature on each one before anyone picks a model or a vendor.

Two rows deserve a second look.

Cost per request is not total cost. A local model moves cost from your inference bill to engineering time: download flows, memory checks, runtime packaging and testing on real hardware. For a feature used rarely, a hosted call is often cheaper overall. For a feature used hundreds of times a day per user, the hosted bill dominates. Our companion guide on what an AI feature really costs works through the per-request formula.

Update cadence cuts both ways. A hosted model can improve overnight, and it can also change behaviour overnight or be retired. A local model behaves the same until you ship a new one. If your feature is tested against exact outputs, such as structured extraction feeding a database, a pinned local model is easier to qualify. If you want the newest capability without lifting a finger, hosted wins.

A local model needs memory for three things: its weights, its KV cache, and working space for the runtime and the rest of the machine. Weights are easy to estimate. The KV cache is the part teams forget, and it grows with every token of context.

Weights

The rule of thumb is:

weight bytes ≈ parameters × bits per weight ÷ 8

An 8-billion-parameter model at 16 bits is about 16 GB. At 4 bits it is about 4 GB before overheads. Quantised formats also store scaling factors. A common scheme keeps a 16-bit scale and a 16-bit offset for every group of 64 weights, which adds half a bit per weight, so "4-bit" really costs about 4.5 bits. Treat the formula as a floor, not a forecast. Embedding tables, layers kept at higher precision and the way a model family counts its parameters all move the real file size.

KV cache

When a transformer reads your prompt, it stores a key and a value vector for every token in every layer, so it does not recompute them for each new token. That store is the KV cache:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

As an illustration, take a model with 32 layers, 8 KV heads, a head dimension of 128 and 16-bit values. Each token costs 2 × 32 × 8 × 128 × 2 = 131,072 bytes, which is 128 KiB. An 8,192-token context then needs 1 GiB of cache, and a 32,768-token context needs 4 GiB. That is as much as the weights of a 4-bit 8B model. Models that share KV heads across query heads or use sliding-window attention on some layers need less, which is why the figure has to be worked out per model.

The practical consequence: context length is a memory decision. Doubling the window you allow can matter more than choosing the next model size up.

A worked example from Neel

Neel's current development rule for a user-added GGUF model budgets the file size, plus 192 KiB per token of context, plus 1 GiB. For an illustrative 4.0 GiB file:

Same model, same file, and the memory need jumps by 4.5 GiB because of the context setting alone.

Neel's curated local models are three sizes of Gemma 4, run through MLX at 4 bits on Apple silicon. Each one has a documented free-memory requirement, and the gap between file size and memory need is the KV cache and safety headroom at work.

Each figure adds up from three parts in the current development configuration:

The runtime reservation covers the weights, the KV cache for the model's 32k-token context window, and the runtime's working buffers. A larger model has more and wider layers, so every token of context costs more memory as well as the weights growing. The 12B reservation is deliberately conservative.

Protected headroom is memory the engine refuses to take, so the operating system, the editor and the browser keep working while the model generates. A model that loads by pushing the machine into heavy swapping is slower than a smaller model that fits.

Free memory is not installed memory. An 8 GB Mac with a browser and an editor open rarely has 6.5 GiB free, which is why E2B is the floor model and tight at 8 GB, and E4B is the comfortable choice at 16 GB. Neel's model picker shows these needs before anything loads, and memory figures always say which measurement they come from, because the operating system and the engine count free memory differently.

These are documented figures for one runtime and one configuration. Other models, runtimes and devices need their own checks, and the figures may change before release.

Quantisation stores each weight in fewer bits so a model fits in less memory and reads faster from it. You pay for it in quality, and the loss is uneven: it shows up first in precise, structured output.

The table below shows illustrative weight sizes for an 8B model. Real files vary with the format and with which layers are kept at higher precision.

Four habits keep quantisation honest:

• Evaluate on your task, not a leaderboard. A quantised model can score close to the original on general benchmarks and still slip on your schema, your language or your tool definitions. Keep a set of real inputs with expected outputs and run it against every candidate.
• Watch structured output first. Tool calls, JSON and exact formats break before prose does. In Neel's development testing, the smallest curated model sometimes emits a tool call in a slightly wrong shape. The engine repairs it, and the reply's receipt shows a repaired ×1 marker so the repair is never hidden.
• Compare a smaller model at higher precision with a larger one at lower precision. At the same memory budget either can win. Test both.
• Quantise the cache, carefully. Some runtimes can store the KV cache at 8 bits or fewer, which roughly halves its memory. It extends the context you can afford, and it is one more thing to evaluate.

A runtime loads the weights and runs the maths on the device's hardware. Choose it by platform first, because the runtime decides which model files you can use and how well the hardware is used.

Neel uses MLX at 4 bits as its first local runtime on Apple silicon. Other model families can be explored through compatible user-added GGUF files or a supported local server, with their own memory and compatibility checks. The point for a product team is not which runtime is best in general. It is that the runtime is part of your product:

• Pin exact versions of the runtime and the model, and verify every download against a published SHA-256. Neel is built to do both.
• Do not bundle weights in the installer. Download them only when the user chooses a model, show the size first, and make the download resumable.
• Check memory before loading, using the arithmetic above, and refuse with a clear reason rather than letting the machine swap.
• Prove the route with a real request after loading. A model that appears in a list is not a model that works.

A hybrid product routes each request to a local or a hosted model by rule, not by guesswork. The most useful default is local first, with an escalation the user approves.

These are the patterns we reach for most:

1. Local first, escalate with consent. The local model handles the request. If it cannot (the context is too long, the task needs stronger reasoning, or the local model is not installed), the product says so and offers a hosted model. The user approves, once or for this kind of request. Nothing leaves the device silently.
2. Split by role. One model plans and another executes. A hosted model might draft a plan that a local model carries out against private files, or a local model might triage requests before a hosted one handles the hard cases. Neel's direction is to let you choose one model to plan and another to execute without rebuilding your workflow.
3. Prepare locally, reason remotely. A local model or plain code redacts names, strips secrets, extracts the relevant passages or summarises a long document, and only that reduced context goes to the hosted model. This often cuts token cost as much as it protects privacy.
4. Hosted to learn, local to run. Use a hosted model to explore the task and label evaluation data, then move the narrow, repetitive part to a small local model once you know what good looks like.
5. Offline fallback. The hosted model is the default and a local model keeps a reduced version of the feature working without a connection. Be explicit in the interface about which one answered.

Whatever the pattern, three rules keep it trustworthy:

Switching models must never change permissions. A hosted model gets exactly the tools and data access the local one had, and the user can always see which model produced an answer.

• Show the route. Every answer should say which model and which runtime or provider produced it.
• Make escalation a decision, not a default. Ask before the first hosted call, name the provider, and remember the answer at the scope the user chose.
• Respect the provider's terms. Each hosted provider has its own data handling, retention and residency terms, and a consumer chat subscription does not automatically include API access. Those terms are product requirements, not legal footnotes.

The same framework gives different answers for different features. The three cases below are illustrative, not client work.

Summarising private notes in a desktop app. The data is personal, the task is bounded, and users expect it to work on a train. Run it locally. Set a hardware floor, such as 16 GB of memory for a comfortable 4B-class model, offer a smaller model for 8 GB machines, and cap the context so the KV cache fits. A hosted model can be an opt-in extra for very long documents.

Drafting customer-support replies against a long policy library. Quality matters, the context is long, and the agents use managed office machines with no spare memory. Use a hosted model. Redact customer identifiers before the call, retrieve only the relevant policy passages, and record which model drafted each reply. Revisit local inference only if volume makes the bill painful.

Field inspections on low-end Android phones with patchy signal. Offline matters most and the hardware is weak. Keep the core workflow deterministic: forms, photos and checklists that work with no model at all. Add a small on-device model only for a narrow job such as tidying notes, and queue anything that needs a hosted model until the device is back online, with the user's consent.

Before you commit to local, hosted or hybrid, answer these questions in writing. They take a day and save a quarter.

1. Classify the data. What may leave the device, under which contracts, to which regions? If nothing may leave, the decision is made.
2. Test the smallest model first. Run your evaluation set on a small local model with good inputs and tools. You may find it is enough for most requests, and you will know exactly where it is not.
3. Set a hardware floor from real numbers. Use your analytics to see how much memory your users' devices have, then apply the weights-plus-KV-plus-headroom arithmetic to see which model fits at which context length.
4. Design the download. Explicit consent, the size shown before it starts, resumable transfer, checksum verification and a clear way to remove the model later.
5. Write the routing rule. When does a request escalate, who approves it, and how does the interface show which model answered?
6. Model the cost both ways. Hosted cost per request times expected volume, against the engineering and support cost of shipping local runtimes. Our cost guide has the formulas.
7. Plan for model change. Hosted models will be updated and retired; local models will need upgrades you ship. Either way, your evaluation set is what tells you a change is safe.

Neel is our own worked answer to these questions: local models run without an account or an inference bill, models download only when you choose one, every runtime download is checked against a SHA-256, and a hosted provider is an explicit choice with its own terms. You can follow its progress on the Neel page.

If you are deciding where your AI feature should run, our AI product development team can help you test it on real hardware before you commit. Talk to us.

How much RAM do I need to run an LLM locally?

Add the weights (parameters × bits per weight ÷ 8), the KV cache for the context length you allow, and headroom for the operating system and other apps. As a documented example, Neel's Gemma 4 E2B at 4 bits on MLX is 3.5 GiB on disk but needs about 6.5 GiB of free memory, so it is tight on an 8 GB Mac and comfortable on 16 GB. Larger models and longer contexts need proportionally more.

Is running AI locally more private than using a cloud API?

Usually, because the prompt and the data never leave the device unless the product sends them somewhere. A hosted call sends the data to the provider under its retention, training and residency terms. Local is only private if the product around the model is too, so check telemetry, logs and any optional hosted fallback.

Is a local model cheaper than a hosted model?

Per request, almost always: there is no inference bill, and the user's device does the work. In total, it depends on volume. Shipping local runtimes, download flows and hardware testing costs engineering time, so a rarely used feature is often cheaper on a hosted API, while a high-volume, repetitive task usually favours local.

What is the difference between MLX and GGUF?

MLX is Apple's array framework for Apple silicon, and MLX models are weights converted for it, often quantised to 4 or 8 bits. GGUF is a single-file model format used by llama.cpp and runtimes built on it, which run on CPUs and on Metal, CUDA and Vulkan GPUs across macOS, Linux and Windows. Choose by platform first, then by the models and quantisation levels you need.

Does 4-bit quantisation hurt model quality?

It costs some quality, and the loss is uneven. Prose often holds up well, while exact formats, JSON and tool calls slip first. Run your own evaluation set against each candidate, and compare a smaller model at higher precision with a larger one at lower precision at the same memory budget.

Can one product use both local and cloud AI?

Yes, and most mature products do. The common patterns are local first with a consented escalation, splitting planning and execution between models, preparing or redacting data locally before a hosted call, and using a local model as an offline fallback. Whatever the pattern, show which model answered and never let a model switch change permissions.