In one of my side projects, I was heavily using AI APIs to meet users’ text summarization and analysis needs. Initially, everything was fine, but as the user base grew and features expanded, API costs started to rise much faster than I anticipated. Seeing the monthly bill, I realized that just writing code wasn’t enough; active optimization of API usage was also necessary.
AI API costs can escalate quickly, especially through token usage and model selection. In this post, I’ll share 5 practical methods I’ve applied in my own projects that have proven effective in controlling and even significantly reducing these expenses. My goal is to provide you with concrete steps you can implement when facing similar challenges.
1. How to Achieve Token Optimization with Prompt Engineering?
AI API costs are typically determined by the number of tokens processed, and long prompts lead to unnecessary token consumption. This applies to both input and output prompts; the shorter and clearer you are, the less you pay. In my experience, most developers focus on getting the model to produce the desired output rather than on cost, which is a natural approach to start with.
However, to reduce costs in a production environment, making prompts more efficient is essential. For instance, in an AI assistant we used to help operators in a production ERP system, we initially provided very long and detailed instructions. Then I realized that much of these instructions consisted of general information the model already knew, and shorter prompts with just keywords provided the same efficiency.
Techniques for Shortening and Clarifying Prompts
There are several fundamental techniques for optimizing prompts. First, avoiding unnecessary words and convoluted phrases by getting straight to the point makes a big difference. For example, saying “Summarize the text:” instead of “Could you please summarize the following text for me?” even saves tokens. This helps the model focus on the desired task while also reducing the amount of data sent to the API.
Second, using zero-shot or fewer examples instead of few-shot examples can also reduce costs. If the model is already good at a specific task, just describing the task might be sufficient instead of providing lengthy examples. On the other hand, while few-shot examples can improve output quality for complex or niche tasks, it’s important to remember that this also increases costs. This is a quality-cost trade-off, and it’s more pragmatic to aim for sufficient quality at the lowest cost rather than always striving for the highest quality.
Third, the placement of instructions is important. Placing information the model should pay most attention to at the beginning or end of the prompt can mitigate the impact of unnecessary tokens in between. Additionally, if you require specific formats like JSON output, clearly stating this and even providing a sample JSON schema helps the model spend fewer “trial-and-error” tokens. This is a critical step, especially in automated workflows, as it ensures consistent output.
# Example: Bad prompt
prompt_bad = """
Hello AI assistant, I hope you are well. I would like to ask you to summarize the following long text for me, without being too detailed.
Please highlight only the main ideas and the most important points. Avoid unnecessary information in the summary and be as brief as possible.
Here is the text:
[Long text goes here]
"""
# Example: Optimized prompt
prompt_good = """
Summarize the text, highlighting main ideas and key points:
[Text goes here]
"""
# To check token count (e.g., with OpenAI's tiktoken library)
# import tiktoken
# enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
# print(f"Bad prompt token count: {len(enc.encode(prompt_bad))}")
# print(f"Good prompt token count: {len(enc.encode(prompt_good))}")2. Smart Model Selection: Not Every Task Needs the Same Model
When many people start using AI APIs, they often gravitate towards the most popular or most capable model. For example, the superior capabilities offered by GPT-4 can be very appealing. However, the costs of these models are significantly higher than those of smaller and faster models. In my own side projects, I initially tried to use the best model for every task, but I quickly realized this was unsustainable.
Different tasks require different model capabilities. Simple text summarization, classification, or tasks requiring quick responses can often be handled sufficiently by smaller and more affordable models. For instance, to quickly respond to user questions in a chat application, models like Gemini Flash or GPT-3.5 Turbo are much more cost-effective than GPT-4 or Gemini Ultra. When starting with lower-cost models, it’s important to carefully test their performance on complex tasks.
Task-Specific Model Selection and Fallback Strategies
The key is to analyze each AI task and select the most suitable and cheapest model for it. If a task requires critical accuracy or complex reasoning (e.g., legal text analysis or in-depth code generation), choosing a more capable and expensive model might be logical. In such scenarios, the high cost can be balanced by the value obtained.
However, using the same capable model for a simple customer support bot response or a short text translation is unnecessary spending. By choosing lighter models for such tasks, you can achieve significant reductions in your overall AI bill. In one of my projects, by switching to GPT-3.5 for most routine tasks, I managed to roughly halve the costs.
Another strategy is to implement fallback mechanisms. You can first try with a cheaper model, and if the output quality is insufficient or the model falls below a certain confidence threshold, you can automatically switch to a more capable and expensive model. This approach guarantees quality for critical tasks while optimizing costs. This hybrid approach is ideal for systems that handle requests of varying complexity.
# Example: Dynamic model selection
def get_ai_response(task_type, prompt):
if task_type == "simple_summary" or task_type == "quick_faq":
# More affordable model for simple summarization or FAQs
model = "gemini-flash-1.5" # or "gpt-3.5-turbo"
print(f"Simple operation performed with '{model}' model.")
# Actual API call and response processing should be here
# response = call_api(model, prompt)
# if not is_quality_sufficient(response): # Quality check
# model = "gemini-pro-1.5" # Fallback
# print(f"Fallback model '{model}' used.")
# response = call_api(model, prompt)
elif task_type == "complex_analysis" or task_type == "code_generation":
# More capable model for complex analysis or code generation
model = "gemini-ultra-1.5" # or "gpt-4-turbo"
print(f"Complex operation performed with '{model}' model.")
# response = call_api(model, prompt)
else:
# Default or general-purpose model
model = "gpt-3.5-turbo"
print(f"Default operation performed with '{model}' model.")
# response = call_api(model, prompt)
# return response
# Usage
get_ai_response("simple_summary", "The weather is nice today.")
get_ai_response("complex_analysis", "Analyze the financial report.")Specifically, using multi-provider platforms like OpenRouter to switch between different models not only increases flexibility but also opens new doors for cost optimization. Some providers (Groq, Cerebras) can even offer much faster and more cost-effective solutions for certain models, which is a great opportunity to find the right balance between performance and cost. We can put these providers behind a Proxy and use dynamic routing to direct requests to the most suitable model.
3. External Information Management with Retrieval-Augmented Generation (RAG)
One of the biggest cost components for AI models is the long texts that fill their context window. When users ask questions like “What is the sales trend for product X over the last five years?” in an ERP system, including all this information in the prompt every time can be very expensive. This is where RAG (Retrieval-Augmented Generation) comes in. RAG significantly reduces token consumption by retrieving the information the model needs from an external database and including only the relevant parts in the prompt.
In my own projects, especially in a production ERP, I worked with large amounts of internal company documents, product information, and historical sales data. Sending this data directly to an LLM was both costly and risky from a security perspective. With the RAG architecture, I stored this data in a vector database, retrieved the most relevant pieces for the user’s question, and sent them to the model along with the question text. This approach not only reduced token costs but also decreased the likelihood of the model “hallucinating.”
How Does the RAG Architecture Work and Save Tokens?
The RAG architecture primarily consists of two steps: Retrieval and Generation. In the first step, the user’s query is passed through an embedding model to convert it into a vector. This vector is then used to find the most relevant pieces of external information (documents, database records, etc.) that have been previously indexed and vectorized. In the second step, the retrieved relevant information pieces are sent to an LLM as a prompt along with the original query.
This process allows the LLM to work on a small, relevant context without needing to read the entire knowledge base from start to finish. Thus, both the prompt length is reduced, and more accurate, contextually relevant answers are obtained. For example, when querying the technical specifications of a product in a production ERP, sending only the technical document snippets for that specific product would be much more efficient than sending the entire product catalog.
graph TD; A["User Query"] --> B["Create Embedding"]; B --> C["Search Vector Database"]; C --> D["Select Most Relevant Chunks"]; D --> E["Construct Prompt (Query + Relevant Chunks)"]; E --> F["LLM API Call"]; F --> G["Response"];
4. Recycling Spending with Caching and Reuse Strategies
Do AI API calls always produce the same response? If the prompt is the same and the model is deterministic, yes. In this case, caching the response instead of repeating the same call every time can lead to significant cost savings. In one of my side projects, I was analyzing specific scenarios for financial calculators. Since these scenarios were frequently repeated, I significantly reduced API calls by caching the responses instead of going to the AI every time.
Caching strategies are very effective, especially for data or analyses that are frequently queried but change infrequently. For example, when you generate a general description of a product or an FAQ answer from an AI model, that answer will likely remain valid for a long time. In such cases, saving the response from the first call and serving it directly from the cache for subsequent identical requests eliminates costs and reduces latency.
When and How to Cache?
There are a few points to consider when caching. First, ensure that the responses you cache are deterministic and static. If the model’s response always varies (e.g., creative writing tasks), caching will not be beneficial. Second, it’s important to determine how long the cache will be valid (TTL - Time-To-Live). A very short TTL will reduce the benefit of caching, while a very long TTL might lead to outdated data being served.
Using a fast key-value store like Redis at the application layer, we can use the hashed prompts of incoming AI API requests as keys and store model responses as values. When a request comes in, we first check if the response for this prompt exists in the cache. If it does, the response is returned directly from the cache; otherwise, an API call is made, and the response is cached.
import hashlib
import json
import redis
# Redis connection (example)
# r = redis.Redis(host='localhost', port=6379, db=0)
def get_ai_response_with_cache(prompt, model_name, ttl=3600): # TTL 1 hour
prompt_hash = hashlib.sha256((prompt + model_name).encode('utf-8')).hexdigest()
# Check in cache
# cached_response = r.get(prompt_hash)
# if cached_response:
# print("Response retrieved from cache.")
# return json.loads(cached_response)
# If not in cache, make API call
print("API call made.")
# Actual AI API call should be here
# response = call_ai_api(prompt, model_name)
response = {"generated_text": f"This is a response generated by AI for '{prompt}'. (Model: {model_name})"}
# Cache the response
# r.setex(prompt_hash, ttl, json.dumps(response))
return response
# Usage
get_ai_response_with_cache("Summarize the text: AI costs.", "gpt-3.5-turbo")
get_ai_response_with_cache("Summarize the text: AI costs.", "gpt-3.5-turbo") # Second call should come from cacheCaching is an indispensable tool for reducing AI API costs, especially in read-heavy applications. However, complex cache invalidation strategies can sometimes become a problem in themselves. Therefore, when deciding where and how to implement caching, it’s important to consider your application’s needs and data change frequency.
5. Increasing Efficiency with Batch Processing and Asynchronous API Calls
Making AI API calls one by one can be both costly and time-consuming, especially when working with large datasets. Most AI providers offer batch processing capabilities that allow you to group multiple requests into a single API call. This often translates to lower token costs or more efficient resource utilization. In a data analysis project, when I needed to summarize hundreds of short texts, I saved costs and significantly reduced processing time by sending them in batches instead of individually.
Batch processing is ideal for scenarios requiring offline or near real-time processing. For example, batch APIs are great for reporting tasks done overnight or data classification tasks where users don’t expect immediate responses. With this approach, the connection cost and overhead to the API are paid once for a single batch request, rather than separately for each individual request.
Optimizing Processes with Asynchronous Processing
Along with batch processing, asynchronous API calls also play a significant role in cost and performance optimization. Many AI APIs offer asynchronous (non-blocking) call models for long-running operations. This allows the request to be processed in the background, and you receive feedback when the response is ready (via callback or polling), instead of sending the request and waiting for the response immediately.
Asynchronous calls improve overall system performance by preventing the blocking of your application’s main thread. They also become even more efficient when used in conjunction with batch processing. For example, using a message queue (Kafka, RabbitMQ), you can queue texts to be processed, have a separate worker service fetch texts in batches from this queue and send them to the AI API asynchronously. When responses arrive, they are again forwarded to the relevant places via another queue or database.
This approach is crucial for large-scale data processing and background tasks. In a production ERP system, I set up such an asynchronous batch system for tasks like automatic tagging of newly added products or summarizing stock movements. This allowed us to perform large-volume AI processing cost-effectively without impacting the user experience.
graph TD;
A["Application"] --> B["Message Queue (Data to Process)"];
B --> C["Batch Processor (Worker)"];
C --> D{"Collect Data & Create Batch"};
D --> E["AI API (Batch Call)"];
E -- Asynchronous Response --> F["Response Queue"];
F --> G["Result Processor"];
G --> H["Database/User"];
Conclusion
Optimizing AI API spending is an ongoing process that affects not only your wallet but also the overall efficiency of your systems. One of the most important things I’ve seen in my twenty years of technical experience is that cost optimization isn’t just about “cutting costs” but also about using resources more intelligently. From shortening prompts to choosing the right model, managing external information with RAG, to caching and batch processing, these five practical methods can help you significantly reduce your AI API costs.
Remember, every application has its unique requirements and constraints. When adapting these strategies to your own project, always consider the balance between quality and cost. Don’t be afraid to make mistakes; I also encountered unnecessarily high costs when integrating AI into a production ERP system early on, and through that, I learned these lessons. The important thing is to continuously measure, test, and learn. This way, you can both protect your budget and fully leverage the potential offered by artificial intelligence.