What an AI Feature Really Costs to Run, and How to Predict It

An AI feature costs the price of one completed task multiplied by the number of tasks, plus the fixed costs around the model: embeddings, storage, evaluation runs, logging and human review. Bills surprise teams when they price a single model request and forget everything else a task triggers.

A single "ask the assistant" button can fire several model calls, a retrieval query, a tool call, a retry when validation fails and, for some answers, a person checking the result. Each of those is cheap on its own. Multiplied by every user, every day, they become a line on the P&L that someone has to defend.

This guide gives you a formula you can put in a spreadsheet, a worked example with clearly made-up prices, the levers that change the result, and a checklist for a monthly cost review. It deliberately avoids real vendor prices. They change too often to print, and the structure of the calculation matters more than this month's rate card. Plug in your provider's current prices, as of the day you run the numbers.

The cost of an AI feature is the model cost per task, multiplied by tasks, plus review and fixed costs. Write it down with variables so the team argues about inputs, not arithmetic.

``` model cost per task = (1 + r) × Σ over calls ( tokens_in × P_in + tokens_out × P_out ) / 1,000,000

monthly cost = T × model cost per task + T × h × R (human review) + embeddings + vector storage + evaluation runs + logging and tracing

cost per successful task = monthly cost / (T × s) cost per active user = monthly cost / MAU ```

The terms:

• T: tasks per month, meaning user-visible jobs such as "answer this question" or "generate this deck", not API calls.
• Calls per task: an agent or a tool-using flow makes several model calls per task, and each call resends the conversation so far.
• tokens_in, tokens_out: tokens sent to and generated by the model on each call. Output tokens are usually priced higher than input tokens.
• P_in, P_out: price per million input and output tokens for the model you use.
• r: retry rate, covering timeouts, rate limits and responses that fail validation.
• h, R: the share of tasks a person reviews, and the loaded cost of one review.
• s: success rate, meaning the share of tasks that produce a result the user accepts.

Three costs hide outside the per-call price. Tool calls add a model round trip each time, because the result goes back into the context. Embeddings are paid when documents are indexed and re-indexed, and vector storage grows with the corpus. Evaluation runs are real traffic too: a regression suite run on every prompt change is a recurring line item.

Here is the formula applied to an illustrative feature: an assistant inside a B2B product that answers questions from a customer's own documents and makes one tool call per task. Every price below is invented for the example and is not any vendor's price. The unit "u" stands for any currency.

Assumptions (illustrative):

- 10,000 monthly active users, each completing 8 tasks: T = 80,000 - 2 model calls per task: - Call 1 sends 6,000 tokens (1,500 instructions, 4,000 retrieved passages, 500 conversation) and returns 300 tokens that request a tool call. - Call 2 sends 6,800 tokens (call 1 plus the tool request and a 500-token tool result) and returns a 500-token answer. - Per task: 12,800 input tokens and 800 output tokens - P_in = 1.00 u and P_out = 4.00 u per million tokens - Retry rate r = 5%, success rate s = 80% - 5% of tasks reviewed by a person at 0.40 u per review

Model cost per task: 12,800 × 1.00 / 1,000,000 = 0.0128 u, plus 800 × 4.00 / 1,000,000 = 0.0032 u, for 0.0160 u. With retries, 0.0160 × 1.05 = 0.0168 u.

That is 0.34 u per active user and, with 64,000 successful tasks, 0.053 u per successful task.

The lesson is in the shares. In this example the model calls are less than half the bill, and human review is the largest line. Teams that optimise only the prompt are working on the smaller number.

What moves the total

The agent row deserves attention. Calls 3 and 4 each resend everything before them, so doubling the steps took input from 12,800 to 29,200 tokens per task. Context grows with every step, which makes step count one of the most expensive design decisions in an agent.

Cost per successful task is the number that tells you whether a feature is efficient; cost per active user tells you whether it fits your pricing. Track both, because each hides something the other shows.

Cost per successful task charges failures to the work that succeeded. A failed answer still burns tokens, and a user who regenerates three times costs four tasks for one result. If quality drops, this number rises even when spend is flat, which makes it an early warning that a cost dashboard alone will miss. It is also the fair way to compare models: a cheaper model that fails more often can cost more per useful result.

Cost per active user connects the feature to revenue. Compare it with revenue per user, or with the price of the plan that includes the feature, and you know whether the feature is a margin problem before finance does.

Averages hide the distribution. Usage of AI features is usually uneven: a small group of heavy users, automated scripts or a single large tenant can drive a disproportionate share of tokens. Break the numbers down by plan, tenant and user decile before deciding anything. Very often the fix is a fair-use quota on the top tier, not a cheaper model for everyone.

Agree the definition of "successful" with the product owner. For a drafting feature it might be "sent without major edits"; for a search answer, "no rephrased query within a minute". Without that definition, cost per successful task becomes a matter of opinion.

The biggest savings usually come from sending less context, routing easy work to smaller models, and cutting agent steps; caching and batching help where the traffic suits them. Every lever trades something, so change one at a time and re-run your evaluation set after each.

Caching and batch terms are provider features, not universal rules. As of October 2026 several major providers offer both, with different discounts and conditions, so check current terms before counting the saving.

Routing, in practice

Routing works when you can tell easy from hard cheaply. Good signals include input length, the task type the user picked, and whether a small model's output passes validation. A common pattern is "try small, escalate on failure": the small model answers, a validator checks the result, and only failures go to the larger model. Track the escalation rate. If it climbs past the point where the extra call wipes out the saving, the router needs work.

Caching without leaking

Response caches must be keyed by everything that affects the answer, including the user's permissions. A cached answer built from one tenant's documents must never be served to another. When in doubt, cache retrieval results or embeddings rather than final answers.

Local and on-device models

A small local model can take whole categories of work off the per-token bill: classification, extraction, short rewrites, autocomplete. The cost does not disappear. It moves to engineering time, device support and model updates. Whether that trade is right depends on privacy, offline needs and volume, which we cover in local vs cloud AI.

Predictable AI spend comes from limits at three levels: per run, per user or tenant, and per feature. Each limit needs a designed behaviour for when it is hit, not just an error.

Tag every request. Record the feature, tenant, model, prompt version and cost on every call. Without tags, a spike shows up as one big number nobody can explain.

Cap each run. Set a maximum number of agent steps, a maximum token count and a maximum wall-clock time. Runaway loops are the most common cause of a surprising bill, and a step cap prevents them outright.

Set quotas per user and tenant. Fair-use limits by plan protect you from scripts and from a single heavy account. Show users their remaining allowance before they hit the limit, so the cap reads as a product rule rather than a failure.

Budget per feature. Give each AI feature a monthly budget with an owner. Alert at, say, 50%, 80% and 100% of the month's budget, and compare spend with the calendar.

Alert on unit cost, not just spend. A rise in total spend may just mean growth. A rise in cost per successful task, for example more than 20% week on week, means something changed: a longer prompt, a new retry loop, a quality drop. That is the alert worth waking someone for.

Degrade gracefully at hard limits. When a budget is exhausted, switch to a smaller model, queue non-urgent work, or fall back to the non-AI path. Keep the provider's own spending limit as a backstop, set above your internal cap.

For most products, buying model access through a hosted API is cheaper than running models yourself, until volume is high and steady enough to keep dedicated hardware busy. The deciding variable is utilisation, not the per-token price.

A rough break-even test for self-hosting:

self-hosted cost per million tokens =
  (GPU hours + share of engineering time + monitoring) per month
  / (millions of tokens actually served per month)

Compare that with the blended API price for the same quality. The trap is the denominator. Idle GPUs cost the same as busy ones, so a model that serves traffic eight hours a day pays for twenty-four. Engineering time is the other trap: someone has to patch, upgrade, evaluate and be on call for the serving stack.

Whatever you choose, keep model access behind one internal interface. Prices, models and providers change more often than your product does, and a clean provider boundary turns a switch into a configuration change plus an evaluation run rather than a rewrite. Fine-tuning or training your own model is a separate decision, and rarely the first one: most features should prove their value on a hosted or open-weight model before anyone pays for training.

Run this review monthly for every AI feature in production, and before widening access to a new one. It takes an hour if the tagging is in place.

1. Unit cost. What did one successful task cost this month, and how does that compare with last month?
2. Shares. What share of spend went to model calls, human review, embeddings and storage, evaluation runs and logging? Which line grew fastest?
3. Tokens per task. Has the average input grown? Check the prompt, the retrieved context and the conversation history separately.
4. Calls per task. Are agents taking more steps than last month? Is the step cap ever hit?
5. Retries and failures. What is the retry rate, and what is driving it: timeouts, rate limits or validation failures?
6. Routing. What share of traffic goes to each model, and what is the escalation rate?
7. Caching. What is the cache hit rate, and are cache keys still correct after recent permission changes?
8. Distribution. Which tenants, plans or users drive the top 10% of spend? Do quotas still fit?
9. Quality link. Did any cost-saving change move evaluation scores or acceptance rates?
10. Volatile inputs. Have provider prices, model versions or retirement dates changed since the last review? Record the date you checked.
11. Budget. Is the feature inside its budget with headroom for planned growth?

Write the answers down with the date. Six months of these notes are the best forecast you will have.

We treat cost and latency as acceptance criteria, measured per successful task, before a feature reaches more users. That is cheaper than discovering the bill after launch and redesigning under pressure.

In practice, the first milestone of an AI product development engagement is one useful job tested on real examples, with cost per task measured alongside quality. We choose the smallest model that meets the quality bar, limit context to what the task needs, cache where results allow, keep provider code behind a clear boundary and put usage and latency on a dashboard from day one.

Cost is one half of launching well. The other half is trust: designing for wrong answers, running evaluations and rolling out in stages, which we cover in shipping AI features people trust. If you have an AI feature whose costs you cannot yet predict, or one that is about to scale, talk to the team.

How much does it cost to run an AI feature?

It depends on tasks per month, model calls and tokens per task, the model's price, retries, and the costs around the model such as embeddings, evaluation runs, logging and human review. Calculate model cost per task with a formula, add the other lines, and divide by successful tasks. In many features the model calls are not the only large line; human review can be bigger.

How do you calculate LLM cost per user?

Add up the month's AI spend for the feature, including model calls, review, embeddings, storage, evaluations and logging, then divide by monthly active users of that feature. Break the result down by plan and tenant, because a small group of heavy users often drives a large share of usage.

What is the fastest way to reduce LLM costs?

Measure first: tag requests and find which line and which feature dominates. The usual wins are trimming retrieved context, routing easy requests to a smaller model and cutting agent steps, since each step resends the whole context. Re-run your evaluation set after every change so savings do not quietly cost quality.

Is self-hosting an open-weight model cheaper than an API?

Only when volume is high and steady enough to keep the hardware busy. Divide your monthly GPU, engineering and monitoring costs by the tokens you actually serve and compare that with the API price for the same quality. Low utilisation and on-call engineering time are what usually make self-hosting more expensive than expected.

How do you stop AI costs from spiking unexpectedly?

Cap each run with maximum steps, tokens and time, set quotas per user and tenant, and give each feature a budget with alerts at fixed percentages. Alert on cost per successful task as well as total spend, and design a graceful fallback, such as a smaller model or a queue, for when a hard limit is reached.

Do on-device or local models remove AI running costs?

They remove per-token fees for the tasks they handle, but the cost moves to engineering time, device support, app size and model updates. Local models suit classification, extraction, short rewrites and privacy-sensitive or offline work, and are usually paired with a hosted model for harder tasks.