Lowering OpenAI & Gemini Token Costs in AI Apps

How to Reduce OpenAI & Gemini Token Costs in a Production AI App

We got a $14,000 surprise bill in month two. The feature looked harmless in staging. A small AI assistant inside the product, a few prompts, and normal usage tests. Nothing suggested a problem.

Then real users arrived. Conversations grew longer. Context kept getting appended. The same instructions were sent again and again. Token usage climbed quietly until the monthly invoice forced everyone to look closely.

After 16 years of building production software, we have learned that AI costs rarely explode because of one big mistake. They grow from small architectural decisions. Prompt design, memory handling, and request patterns. 

So, in this guide, I share the fixes we apply to keep OpenAI and Gemini token costs under control for our clients. Let’s first…

Understand What You’re Actually Paying For 

Most discussions about AI cost reduction jump straight to tactics. Fewer prompts. Smaller models. Caching. All useful ideas. But without a clear mental model of how tokens are billed, teams often optimize the wrong thing.

In production systems, token costs grow quietly because developers underestimate what actually gets counted.

Input Tokens vs Output Tokens

Every request to a model contains input tokens and generates output tokens.

Input tokens include the prompt, system instructions, conversation history, tool definitions, and any other context sent with the request. Output tokens are the words the model generates in response.

Most providers price output tokens higher than input tokens. In many cases, the difference can reach three to five times the cost. That means a verbose response can cost more than the prompt itself.

This changes the optimization strategy. Reducing prompt length helps, but controlling response length often saves more money.

The “Re-Read Tax” in Conversations

Another cost pattern appears in chat-style applications.

Each time a user sends a new message, the model must read the entire conversation again to understand context. That means the full message history is sent with every request.

A simple conversation can quickly look like this:

Request 1
System prompt + user message

Request 2
System prompt + message 1 + assistant reply + new user message

Request 3
System prompt + full conversation history again

The model is effectively re-reading the entire thread every time, and every token counts again. This is what many engineers call the re-read tax, and it becomes one of the biggest hidden drivers of cost in production chat features.

Tokens That Developers Forget About

Many teams estimate token usage by counting visible text. The problem is that several invisible components are also billed.

Common surprises include:

• System prompts that run on every request
• Tool definitions and function schemas
• JSON structures for tool calling
• Hidden formatting characters and whitespace
• Image tokens in multimodal requests

Each of these elements consumes tokens even though users never see them.

What We See in Real Production Audits

When our team audits AI features for clients, we rarely find a single catastrophic mistake. Instead, we see layers of small inefficiencies.

In many projects, actual token usage is two to three times higher than what the team estimated. The biggest reason is repeated system prompts, large tool definitions, and verbose schemas that get sent with every request.

Before applying optimization tricks, the first step is simple. Measure what is really being sent to the model. Once teams see the full token footprint of each request, the biggest cost leaks become obvious. So…

Audit Before You Optimize

Many teams begin with optimization tricks before they understand where the tokens are actually going. That approach rarely works. In production systems, costs usually concentrate in a few specific endpoints or features. If you do not isolate those first, you end up optimizing areas that barely affect the final bill.

Track Token Usage by Feature, Not Just Globally

Most teams monitor only their provider’s overall usage dashboard. That view shows total spend but reveals nothing about which feature caused it.

A better approach is to instrument your application so that every AI request records:

• Feature or endpoint name
• Input tokens
• Output tokens
• Model used
• Cost per request

This can be logged directly in your application layer or through middleware. Once implemented, you can see exactly which product surfaces generate the highest token consumption. In real systems, the difference between endpoints can be dramatic.

Apply the 80/20 Rule to Token Spend

After instrumentation, the pattern becomes clear. A small number of features usually account for the majority of token usage.

In many audits we run, the distribution looks like this:

• One conversational feature consumes about 40 percent of tokens
• One RAG workflow consumes another 30 percent
• A reporting or summarization feature consumes around 10 percent

Everything else barely moves the needle.

That means the fastest cost reduction often comes from optimizing just two or three workflows, not the entire application.

Tools That Help With Token Observability

Several tools make this process easier by capturing token usage and request traces.

Useful options include:

• Helicone for request logging and cost tracking across AI providers
• LangSmith for tracing LLM pipelines and debugging prompt chains
• The OpenAI Usage Dashboard for model level cost analysis
• Gemini’s token counter API to estimate usage before sending requests

These tools provide visibility into prompt size, response length, and latency. More importantly, they show how those values change as features evolve.

What We Often Discover During Audits

When teams first review detailed token data, the results are usually surprising.

In one client project, a retrieval system designed to answer internal documentation questions looked efficient during testing. Once employees started using it daily, the feature quietly consumed about seventy percent of the company’s AI API budget. The issue was not the model itself. It was the amount of context being attached to every request.

This pattern repeats often. Teams focus on prompt quality and forget to monitor the size of the surrounding context.

You cannot reduce token costs if you cannot see where they originate.

Metrics That Actually Matter

Once instrumentation is in place, a few metrics reveal the true health of your AI architecture:

• Cost per request to measure the financial impact of each feature
• Tokens per user session to understand real user behavior
• Output-to-input ratio to see whether responses are unnecessarily long
• Cache hit rate for repeated prompts or embeddings

These numbers provide a clear baseline. After that, optimization decisions become practical rather than speculative.

Prompt Engineering That Actually Saves Money

Prompt engineering advice often focuses on clarity and correctness. Cost efficiency rarely enters the conversation. In production systems, however, prompt structure directly affects token usage. Small changes in prompt design can reduce both input and output tokens without harming quality.

The Verbosity Trap in System Prompts

Many AI features begin with a simple system prompt. Over time, as new requirements appear, developers keep adding instructions. Safety rules, formatting guidelines, brand tone, fallback behaviors.

After several product iterations, the system prompt becomes a long block of text that runs on every request.

The problem is not the instructions themselves. The problem is repetition. When a prompt grows from 80 tokens to 500 tokens, that cost is paid on every single call. In high-traffic applications, the difference becomes significant.

A periodic review of system prompts often reveals outdated or redundant instructions that can be safely removed.

Instructions vs Examples

Few-shot prompting can improve accuracy by showing the model examples of the desired output. The tradeoff is token cost.

Examples consume far more tokens than plain instructions. In some tasks, the improvement justifies the expense. In others, a clear instruction produces nearly identical results.

The practical rule is simple. Use examples only when they meaningfully improve output quality. Otherwise they become an expensive habit.

The Politeness Tax

Many prompts contain conversational filler such as greetings, polite phrasing, or narrative instructions. While this feels natural, it adds tokens.

Research in prompt design has shown that polite language can add roughly a dozen extra tokens per request. On a single query the difference is negligible. Across millions of requests, it becomes measurable spend.

In production prompts, brevity usually performs just as well.

Structured Output Reduces Response Length

Another overlooked optimization is the format of the model’s response.

When responses are written in natural language, the model tends to produce longer explanations. If the output format is structured, such as JSON or a fixed schema, the response becomes shorter and easier to parse.

For tasks like classification, tagging, extraction, or scoring, structured outputs often reduce token usage while improving reliability.

Write Constraints Instead of Cleaning Output Later

A common mistake in prompts is allowing the model to generate unnecessary text and then removing it later in code.

For example, developers may ask the model to explain reasoning and then strip that reasoning from the response before returning it to users.

A better approach is to set clear constraints in the prompt. If the application only needs a label or structured field, the prompt should request only that output. Preventing extra text is cheaper than generating and discarding it.

A Simple Prompt Audit Your Team Should Run

Production prompts tend to accumulate complexity as features evolve. A quarterly audit helps keep them efficient.

Questions worth asking during that review include:

• Does the system prompt contain outdated instructions?
• Are examples still necessary for this task?
• Can the output format be structured instead of narrative?
• Are responses longer than the application actually needs?
• Are multiple prompts repeating the same instructions?

These small reviews often reveal hundreds of unnecessary tokens per request. In large scale applications, that difference translates directly into lower operating costs.

Caching — The Highest ROI Investment Most Teams Underuse

Caching reduces token costs more than almost any other technique. Yet many AI products rely on a single caching layer or skip it entirely. In practice, cost efficient AI systems use three distinct layers of caching, each solving a different type of repetition.

When implemented together, these layers remove a large portion of unnecessary model calls.

Layer 1: Provider Level Prefix and Context Caching

Modern AI providers already support prompt level caching. Many teams simply do not structure their prompts in a way that allows the system to benefit from it.

Some models automatically cache repeated prompt prefixes. For example, long prompts that exceed about a thousand tokens often become partially cached by the provider. When the same prefix appears again, the provider reuses previously processed tokens and charges less for them. This can reduce input costs significantly without any application changes.

Context caching also exists in systems like Gemini. In these setups, developers can reuse previously processed context such as long documents or reference material instead of sending them again with every request.

The most important rule is prompt structure.

Static information must appear first in the prompt. Dynamic information should appear last.

Correct structure	Incorrect structure
System instructions Reference documents Examples User input	User input System instructions Examples Documents

When static context appears first, the provider can cache the shared prefix across requests. When teams reverse the order, caching becomes ineffective.

In many production prompts, this ordering mistake alone prevents the system from benefiting from built in caching.

Layer 2: Semantic Caching at the Application Layer

Provider caching handles repeated prompts. Semantic caching handles repeated questions.

Many customer facing AI systems receive similar requests again and again. Support bots, documentation assistants, onboarding guides, and FAQ systems often see large numbers of near duplicate queries.

Semantic caching works by storing previous answers along with embeddings. When a new question arrives, the system checks whether a similar query already exists. If the similarity score passes a defined threshold, the cached answer is returned instead of calling the model again.

This approach works well when:

• Queries repeat frequently
• The underlying knowledge changes slowly
• Slight variations in phrasing lead to the same answer

Common tools include GPTCache, Redis based cache layers, or custom implementations built with embeddings and a vector database.

The most important configuration decision is the similarity threshold. If the threshold is too strict, the cache rarely activates. If it is too loose, incorrect responses may appear. Finding the right balance requires testing with real user queries.

In one customer support bot we built, nearly a third of incoming questions were semantically identical to previous ones. Adding semantic caching reduced the client’s AI bill by roughly half without changing the model or prompt.

Layer 3: Response Caching for Deterministic Tasks

Some AI tasks produce deterministic results. These outputs should be cached aggressively.

Typical examples include:

• Content classification
• Language detection
• Tag extraction
• Sentiment scoring
• Data formatting

If the same input always produces the same output, there is no reason to run the model repeatedly.

In these cases, a simple response cache keyed by input text works well. Each cached response can include a time to live depending on how frequently the underlying logic might change.

For example:

Classification results might remain valid for months.
Content summaries may expire sooner if source material updates frequently.

One common production bug appears when multiple identical requests arrive simultaneously before the cache is populated. Each request triggers its own model call, defeating the cache entirely.

The safest approach is to warm the cache before launch or implement request locking so only one model call runs for identical inputs.

When these three layers operate together, token consumption drops significantly without reducing model capability.

Model Routing — The Strategy Most Teams Implement Wrong

Another common cost problem comes from sending every request to the most capable model available. This approach simplifies development but wastes money in production.

Different tasks require different levels of reasoning. Efficient AI systems route requests to models that match the complexity of the task.

Understanding Model Tiers

Most model ecosystems now include several tiers.

Flagship models handle complex reasoning, multi step analysis, and ambiguous instructions. They are powerful but expensive.

Mid tier models balance capability and cost. They perform well for structured writing, summarization, moderate reasoning, and conversational tasks.

Mini or lightweight models are designed for speed and efficiency. They perform well for classification, extraction, tagging, and simple transformations.

In practice, many applications send straightforward tasks to flagship models even though smaller models perform the same job at a fraction of the cost.

Classifying Request Complexity

Effective routing begins with a simple classification step.

Questions worth asking include:

• Does the task require deep reasoning or creative synthesis
• Is the output structured and predictable
• Is the task primarily extraction or transformation
• Does accuracy depend on understanding subtle context

If the task involves structured extraction or simple formatting, a lightweight model often performs well. If it requires reasoning across multiple pieces of information, escalation to a stronger model makes sense.

Cascade Routing

A powerful strategy is cascade routing.

In this architecture, the system attempts the task with a smaller model first. If the response confidence falls below a threshold or fails validation checks, the request escalates to a larger model.

This approach works well when most requests are simple but occasional queries require deeper reasoning.

Instead of always paying flagship model prices, the system only uses those models when necessary.

The Routing Trap Many Teams Fall Into

Some teams attempt routing using simple heuristics. For example, they may check for certain keywords in the prompt and choose a model based on that rule.

This approach breaks quickly because language varies widely. The same task can be expressed in many different ways.

A lightweight classifier, trained to detect task type or complexity, produces far more reliable routing decisions.

Real Differences Between Mid Tier Models

Even within similar pricing tiers, model capabilities differ.

For example, some fast models are strong at extraction and structured tasks but weaker at reasoning across long documents. Others perform better at conversational context but struggle with precise formatting.

Understanding these differences helps teams decide where a model swap is safe.

A Practical Example From Production

In one document processing pipeline we reviewed, every request used a high end reasoning model. The pipeline’s primary task was extracting fields from structured documents.

After testing, most requests were moved to a faster mid tier model optimized for extraction tasks. Only complex cases escalated to the original model.

Roughly eighty percent of requests stayed on the cheaper model. The system maintained the same output quality while reducing monthly API costs by several thousand dollars.

The lesson was simple. The task required structured extraction, not deep reasoning. Once the architecture reflected that reality, the cost problem largely disappeared.

Conversation and Context Window Management

Many guides suggest trimming chat history to reduce token usage. That advice is directionally correct but incomplete. The real challenge is designing a context architecture that keeps conversations coherent without sending unnecessary tokens on every request.

The Compounding Cost of Conversations

In chat-based applications, token usage grows with every turn.

If the system sends the entire conversation history each time, the tenth message in a session can cost several times more than the first. Every previous message, system instruction, and assistant response must be read again by the model.

In naïve implementations, the cost curve looks like this, as I mentioned before:

Turn 1
System prompt + user input

Turn 5
System prompt + full conversation history

Turn 10
System prompt + full conversation history again

By the tenth turn, the request can contain thousands of tokens even if the latest user message is short. So, without careful management, conversations become the most expensive feature in an AI product.

Three Practical Context Management Patterns

Teams typically use one of three patterns to control conversation growth.

• Rolling window

The system keeps only the most recent messages in the context window. Older messages are removed once the conversation exceeds a defined length.

This approach is simple and works well for short task-oriented chats. The downside is that important information from earlier in the conversation may disappear.

• Summarization-based memory

Instead of keeping the entire history, the system periodically summarizes earlier messages and stores that summary as context.

• The conversation might look like this:

System prompt
Conversation summary
Last three messages
Current user input

The summary compresses earlier turns while preserving key information. This approach maintains context while dramatically reducing token usage.

• Selective memory retrieval

In more advanced systems, the conversation history is stored in a database with embeddings. When a new message arrives, the system retrieves only the most relevant past interactions and includes them in the prompt.

This technique works well for long-running conversations, such as support systems or coaching assistants.

When to Summarize and When to Truncate

Truncation works best for short conversations where earlier context becomes irrelevant. Summarization works better when past information still matters but does not need to appear verbatim.

A common pattern is to generate a short conversation summary every few turns. The summary replaces older messages and keeps the prompt compact.

In one client chatbot we built, adding a summarization step every few turns reduced token usage by roughly sixty percent while preserving conversation quality.

Agentic Systems Multiply the Cost Problem

Agent-based AI systems introduce an additional cost pattern.

Each time an agent calls a tool or runs a reasoning step, the entire context is typically sent again. A single user query may trigger multiple model calls, each containing the same prompt history.

Without strict limits, agent workflows can multiply token usage rapidly. Monitoring these pipelines carefully is essential for cost control.

Always Set Output Token Limits

Another simple but critical control is setting maximum output tokens.

Without this limit, models may generate long explanations or reasoning text that the application never uses. Setting explicit output limits ensures responses stay within the token budget required for the feature.

In production systems, controlling response length is as important as controlling prompt size.

RAG Pipeline Optimization

Retrieval augmented generation is widely used to give models access to external knowledge. While effective, poorly designed RAG systems often become one of the largest sources of token consumption.

The issue rarely comes from the model itself. It comes from how much context the system sends with each request.

Retrieval Inflation

Many RAG pipelines retrieve far more context than necessary.

A common configuration fetches the top ten chunks from a vector database and sends them directly to the model. In many cases, the correct answer appears within the first two or three chunks.

Sending all ten chunks inflates the prompt size without improving accuracy.

Reducing retrieval depth often produces similar results with far lower token usage.

Chunk Size Affects Token Costs

Chunking strategies are usually designed to improve retrieval accuracy. They also influence token usage.

Large chunks contain more information but also send more tokens to the model. Smaller chunks may require retrieving more items to provide adequate context.

Finding the right balance between chunk size and retrieval depth helps control prompt size while preserving relevance.

Rerank Before Sending Context to the Model

Another effective optimization is adding a reranking step before the final prompt.

Instead of sending all retrieved chunks to the model, a lightweight reranking model evaluates their relevance. Only the most relevant chunks are included in the final prompt.

This step reduces context size while often improving answer quality because irrelevant documents are removed.

Context Compression

In some pipelines, even the most relevant chunks remain too large. Context compression tools can reduce token size while preserving key information.

Tools such as context compression libraries summarize or filter documents before sending them to the language model. These techniques introduce extra processing steps, so they are most useful when the original context is extremely large. So…

Define a Token Budget for Context

One of the most effective controls is simply setting a fixed token budget for retrieved context.

For example, a pipeline might allow a maximum of two thousand tokens for external documents. Retrieved chunks are sorted by relevance score and added until the limit is reached. Remaining content is discarded or truncated.

This rule forces the system to prioritize the most useful information and prevents uncontrolled prompt growth.

When RAG systems operate within a clear token budget, they remain predictable, efficient, and far less expensive to run at scale.

Batch API, Async, and Off Peak Processing

Most optimization guides focus on prompts and models. Few discuss when the request actually needs to run. In many AI systems, a large portion of workloads do not require immediate responses. Moving those tasks to batch or asynchronous pipelines can reduce costs significantly.

Batch APIs for Large Scale Processing

Several providers now offer batch processing options designed for non real time workloads.

Batch APIs allow developers to submit large groups of requests that the provider processes asynchronously. Because these jobs run outside latency sensitive infrastructure, they are typically priced lower.

For example, batch processing pipelines can reduce costs substantially compared with standard real time requests. The tradeoff is simple. You wait longer for results, but you pay far less per token.

This approach works best when results are not required instantly.

When Batch Processing Makes Sense

Common use cases include:

• Generating large sets of reports
• Processing document archives
• Bulk content classification
• Nightly data enrichment pipelines
• Product catalog tagging or summarization

In these situations, users rarely need results within seconds. Running the tasks overnight or in scheduled batches produces the same outcome at a lower cost.

Async Workflows Inside Applications

Even within interactive applications, some AI tasks can move to asynchronous pipelines.

For example, a user may upload a document and receive a notification when analysis is complete. The processing step can run in a queue rather than inside the request cycle.

This design separates user experience from model execution and gives the system flexibility to process requests more efficiently.

A Practical Rule From Production Systems

One simple rule often leads to large savings.

If the feature does not require real time output, process it asynchronously or in batches.

In multiple client projects, identifying these workloads and moving them to batch pipelines reduced overall AI spending by roughly forty to fifty percent. The application behavior remained the same from the user’s perspective, but the infrastructure cost dropped dramatically.

Fine Tuning vs Prompt Engineering — The Economic Decision

Fine tuning often appears in technical discussions as a performance improvement technique. In production systems, it is also a financial decision. Teams must evaluate whether the cost of training and maintaining a custom model is justified by lower inference costs.

When Fine-Tuning Makes Financial Sense

Fine-tuning becomes attractive when three conditions exist.

First, the task is narrow and consistent. Examples include classification, structured extraction, or domain-specific formatting.

Second, the application runs at high volume. Savings from shorter prompts and smaller models accumulate quickly at scale.

Third, the output format remains stable over time.

In these scenarios, a fine-tuned model can perform the task with fewer instructions and less context than a general-purpose model.

Understanding the Break-Even Point

The economics usually look like this.

• Without fine-tuning:

A powerful model processes long prompts with instructions, examples, and formatting rules.

• With fine-tuning:

A smaller model performs the same task with minimal prompting because the behavior is embedded in the trained model.

The decision depends on whether the training investment plus ongoing maintenance costs are less than repeated high-cost inference with large prompts.

For applications processing millions of requests, the difference can be substantial.

The Operational Risk

Fine-tuned models introduce a maintenance burden.

When requirements change, the model may need retraining. If output formats evolve frequently or the underlying data shifts, maintaining the fine-tuned model can become expensive.

For rapidly evolving products, prompt engineering often remains the more flexible approach.

A Practical Way to Think About It

Fine tuning behaves like a capital investment. You spend upfront to reduce costs later.

Prompt engineering behaves like an operational expense. It costs more per request but requires little long term maintenance.

So, teams should choose based on the stage of their product. Early stage applications benefit from flexibility. Mature high volume systems benefit from efficiency.

Final Verdict — Governance, Monitoring, and Cost Culture

Technical optimizations matter, but sustainable cost control requires organizational discipline. AI costs grow quickly when teams treat token usage as an invisible infrastructure detail.

Successful teams treat token spending as a first-class engineering metric. So…

Define Token Budgets for Each Feature

During sprint planning, AI-powered features should include a target cost profile.

For example, a team might define that a support chatbot interaction must remain below a certain token threshold per session. Setting these limits early forces architectural decisions that keep costs predictable.

Without these constraints, token usage tends to expand over time as prompts grow and features evolve.

Monitor Costs Continuously

Usage dashboards help identify long term trends, but real protection comes from automated alerts.

Engineering teams should monitor:

• sudden increases in tokens per request
• spikes in total daily token consumption
• abnormal usage patterns from specific endpoints

Detecting these signals early prevents small configuration mistakes from turning into large monthly bills.

Most teams end up wiring these alerts themselves on top of provider dashboards, which only report at the account level. Dedicated AI cost management tooling attributes spend to individual features, models, and teams so a spike can be traced to its source before the invoice arrives.

Protect Systems From Abuse

AI endpoints can become targets for abuse or runaway usage.

Rate limiting at the user or session level protects the system from automated requests or accidental infinite loops inside application logic.

These safeguards ensure that a single misbehaving client cannot generate excessive token usage.

Make Cost Visibility Part of Engineering Culture

Database query performance, API latency, and infrastructure utilization are standard metrics in modern software development. Token cost deserves the same attention.

When teams monitor tokens per request and cost per feature alongside traditional performance metrics, optimization becomes a routine engineering practice rather than a reactive effort after the invoice arrives.

AI applications can scale efficiently, but only when cost awareness is built into both the architecture and the development process. This approach also helps reduce post-launch costs.