The cost anatomy of an LLM feature

When people estimate what an LLM feature will cost, they price what they can see: a short question going in and an answer coming out. That’s the smallest part of the bill.

This article takes one ordinary feature, prices every line of it at current list prices, and shows where the money actually goes. Then it takes the same feature from about $1,200 a month to about $56 without making the answers worse, and does the same on Azure.

The feature

An “ask the docs” box. Someone types a question, the system finds the relevant passages in about 5,000 pages of documentation, and a model writes an answer with links to its sources. It might sit in a help center or in an internal tool. Follow-up questions are allowed, so there’s a short conversation history.

The volume is 20,000 questions a month. The prices are US list prices for October 2026, on AWS first.

One question, taken apart

Here’s the first version, built the way a lot of first versions are built: the flagship model for everything, ten retrieved chunks to be safe, the whole conversation resent every time, and no caching.

On Claude Sonnet 5.5, at $2 per million input tokens and $10 per million output tokens, one question costs 2.35 cents:

Part of the prompt	Tokens	Cost
Instructions	1,200	0.24¢
Retrieved chunks (10 × 500)	5,000	1.00¢
Conversation history	2,500	0.50¢
The question	50	0.01¢
The answer	600	0.60¢
Total	9,350	2.35¢

The question is under 1% of the tokens. The retrieved text costs more than the answer does, and the history nearly as much as the answer. Output tokens cost five times as much as input, so the answer is still a big line on its own, but most of the cost is context the user never sees.

The monthly bill

Multiply by 20,000 questions and add everything around the model:

Line item	Per month
Model calls, production (20,000 × 2.35¢)	$470
Model calls, evaluation runs (500 questions × 30 runs)	$353
Vector store, OpenSearch Serverless Classic (2 OCUs minimum)	$350
Guardrails, question and answer, two policies	$24
Logs and compute	about $1.50
Total	about $1,198

Two of those lines never show up in a back-of-the-envelope estimate.

Evaluation runs cost three-quarters as much as production. Every time someone changes the prompt, a good team re-runs a test set of questions to check nothing got worse. Run a 500-question suite on the flagship model 30 times a month and it adds 15,000 model calls, which is 75% of production volume. That’s the right habit with the wrong defaults.

The vector store bills by the hour, whether anyone asks a question or not. A Classic OpenSearch Serverless collection, long the default behind Bedrock knowledge bases, has a minimum capacity. A production-shaped one costs about $350 a month before the first query. At this volume, that’s three-quarters of what every production answer costs combined. (The next-generation collections, generally available since May 2026, scale to zero when idle. Check which kind you have.)

Version 2: the same feature for $56 a month

None of these should make the answers worse, and your evaluations are how you confirm that. Each change is one decision:

1. The small model by default. Claude Haiku 5.5 handles 90% of questions, at $0.10 per million input tokens and $0.50 per million output. Sonnet 5.5 takes the 10% where retrieval looks weak or the user rephrases the same question. For more on routing, see ATC-001.
2. Cache the instructions. They’re identical on every call, so put them first and turn on prompt caching. A Haiku cache read costs $0.01 per million tokens. The cache lasts about five minutes, so this assumes Haiku calls arrive often enough that 90% of them hit it. Sonnet sees only about one call every 20 minutes, so its cache would rarely hit, and the numbers below don’t count on it.
3. Retrieve five good chunks, not ten safe ones. Better chunking and a relevance cutoff do more for answer quality than volume does. Retrieved text drops from 5,000 tokens to 2,500.
4. Summarize the history. After a couple of turns, replace the full conversation with a summary capped at 800 tokens.
5. Cap the answer. Ask for concise answers and set a maximum output length. 350 tokens instead of 600 cuts the most expensive kind of token by about 40%.
6. Run evaluations on the default model, and fewer of them. A 200-question suite, run 20 times a month on Haiku, costs about $2. When results can wait, batch inference halves even that.
7. A vector store that’s free when idle. Amazon S3 Vectors charges for storage, writes and queries, not for hours. For 10,000 vectors and 20,000 queries a month, that’s about ten cents.

The new bill:

Line item	Per month
Model calls: Haiku for 90%	$9.64
Model calls: Sonnet for 10%	$25.20
Evaluation runs	$2.14
Guardrails, question and answer, two policies	$18.00
Vector store, S3 Vectors	$0.10
Logs and compute	about $1
Total	about $56

One question now costs about 0.17 cents in model calls instead of 2.35 cents.

Two things stand out. The 10% of questions that go to Sonnet are now 72% of the model spend, so the escalation rule is the decision to watch. And the guardrails now cost more than the default model. That’s not a reason to drop them, but it is a reason to check the question and the final answer only, not every step in between.

If you’d rather not run the retrieval side yourself, Bedrock’s new managed knowledge bases charge $5 per GB of source data a month plus $1 per 1,000 retrievals. That’s a little over $20 a month at this volume, and only the storage part keeps billing while idle.

How the bill scales

Version 2 at three volumes:

Questions a month	AWS	Azure
2,000	about $9	about $83
20,000	about $56	about $134
200,000	about $532	about $650

At small volume, the fixed costs decide the bill. At large volume, tokens do. If your feature is new and its traffic is uncertain, avoid anything with an hourly floor until the traffic justifies it.

The same feature on Azure

The model line can be identical. Claude Haiku 5.5 and Sonnet 5.5 are available in Microsoft Foundry, billed through the Azure Marketplace, so you can run exactly the same model routing on Azure. The usual Azure alternative is OpenAI’s GPT-5 mini, at $0.25 per million input tokens, $0.025 cached and $2.00 per million output. For this workload that’s about three times what Haiku 5.5 costs, and more in practice, because it’s a reasoning model and its reasoning tokens bill as output. Test both on your own questions before choosing.

The differences are in everything around the model:

• Search has a floor. Azure AI Search’s Free tier allows 50 MB of storage, which a real document set quickly outgrows. Basic costs $0.101 an hour, about $74 a month, and S1 about $245. That floor is most of the gap in the table above. A serverless option that scales to zero is in preview, at $0.24 per compute-unit hour; it’s worth watching for features like this one.
• Content safety is one price for everything. Azure AI Content Safety’s standard tier costs $0.375 per 1,000 text records, where a record is up to 1,000 characters. Prompt Shields is billed the same way. For this feature that’s about $22.50 a month, close to AWS.
• Logs cost more per gigabyte, after a free allowance. Log Analytics ingestion is $2.30 per GB after the first 5 GB, against $0.50 per GB in CloudWatch. At this volume, even full prompts stay inside the free allowance. At ten times the volume, log the metadata and a sample, not every prompt.

So the honest comparison at 20,000 questions a month: the same model costs the same on both clouds, and the difference is the search service. On AWS, S3 Vectors keeps that line near zero. On Azure, it’s about $74 a month until the serverless tier is generally available.

A cost model you can keep

• Price one request by part of the prompt, not by question and answer
• Put the instructions first and cache them
• Retrieve less, but better, and cap the history and the answer
• Make the small model the default, and write down the escalation rule
• Count evaluation traffic as real traffic, and tag it so you can see it
• Check every component for an hourly floor before it goes into a small feature
• Publish cost per question as a metric and alarm on it, with a monthly budget behind it

The cheapest token is the one you don’t send. Most of them were never going to be seen by the user anyway.

---

Prices are US list prices as of October 2026: global endpoints for Claude on Bedrock, Global deployments on Azure, East US for Azure services. Sources: Amazon Bedrock pricing, Claude pricing, Amazon S3 pricing (S3 Vectors), OpenSearch Service pricing, CloudWatch pricing, Claude models in Microsoft Foundry, Azure OpenAI pricing, Azure AI Search pricing, Azure AI Content Safety pricing, Azure Monitor pricing.