Token Budget

A token budget is the total number of tokens available for an AI interaction — covering both the tokens you send in (your prompt, context, and conversation history) and the tokens the model generates in response. Managing your token budget effectively is key to controlling costs and getting the best results.

How token budgets work

Every AI model has a maximum context window — say 128,000 tokens. This entire window must accommodate your input and the model's output. If your input uses 120,000 tokens, only 8,000 tokens remain for the response. If you set a maximum output length of 4,000 tokens, your input budget is 124,000 tokens.

AI providers charge per token — both input tokens and output tokens (with output tokens typically costing more). Your token budget therefore directly determines the cost of each interaction.

Components of the token budget

• System prompt: Instructions that define the model's behaviour. Can range from a few hundred to several thousand tokens.
• Conversation history: Previous messages in a multi-turn conversation. Grows with each turn.
• Retrieved context: Documents provided via RAG for grounding. Often the largest component.
• User message: The current query or instruction.
• Model response: The generated output, bounded by max_tokens setting.

Managing token budgets

• Prompt optimization: Write concise system prompts. Remove unnecessary instructions and examples.
• Context selection: Only retrieve and include the most relevant documents, not everything potentially related.
• Conversation management: Summarize or trim conversation history when it grows too long.
• Output limits: Set appropriate max_tokens to prevent unnecessarily long responses.
• Model selection: Use smaller, cheaper models for simple tasks and reserve larger models for complex ones.

Token budget and cost

Understanding token budgets is essential for cost management. A workflow that processes 100 documents at 10,000 tokens each, with a 2,000-token response for each, uses 1.2 million tokens. At typical API prices, this could cost anywhere from a few cents to several dollars depending on the model.

Common mistakes

Teams often over-stuff context with marginally relevant information, eating into the response budget and increasing cost without improving quality. Others set output limits too high, paying for verbose responses when concise ones would suffice.