Batch Inference

Batch inference is the practice of collecting multiple prediction requests and processing them together rather than handling each one individually. For workloads that do not require real-time responses, batch inference can reduce costs by 50% or more while dramatically increasing throughput.

Batch versus real-time inference

• Real-time (online) inference: Each request is processed immediately as it arrives. The user waits for the response. Latency is critical. Example: a chatbot responding to a customer.
• Batch inference: Requests are collected over a period and processed together. Results are available later — minutes, hours, or overnight. Example: classifying 100,000 support tickets for weekly reporting.

Why batch inference is cheaper

Several factors make batch processing more cost-effective:

• GPU utilisation: Real-time inference often wastes GPU capacity between requests. Batch processing fills the GPU continuously, extracting maximum value from expensive hardware.
• Scheduling flexibility: Batch jobs can run during off-peak hours when compute is cheaper.
• API discounts: Major AI providers offer significant discounts (typically 50%) for batch API access because it allows them to manage load more efficiently.
• Reduced overhead: Each API call has overhead (authentication, network latency, connection setup). Batch processing amortises this overhead across thousands of requests.

Common batch inference use cases

• Document processing: Classifying, summarising, or extracting information from large document collections.
• Data enrichment: Adding AI-generated labels, categories, or scores to existing database records.
• Content generation: Producing product descriptions, meta tags, or translations for an entire catalogue.
• Evaluation: Running AI model outputs through quality assessment at scale.
• Reporting: Generating weekly or monthly analytics that require AI processing of accumulated data.

Implementation approaches

• Provider batch APIs: OpenAI, Anthropic, and other providers offer dedicated batch processing endpoints with lower pricing and higher rate limits.
• Queue-based systems: Use message queues (SQS, RabbitMQ) to collect requests and process them in batches on a schedule.
• Data pipeline integration: Embed batch inference into existing data processing pipelines (Airflow, Dagster, dbt).
• Parallel processing: For self-hosted models, process multiple inputs simultaneously by increasing the batch size in the inference engine.

Sizing your batches

• Too small: You miss out on the efficiency gains. Processing 10 items at a time is barely more efficient than processing them individually.
• Too large: Memory constraints and error handling become difficult. If a batch of 100,000 items fails partway through, recovery is complex.
• Sweet spot: Typically hundreds to thousands of items per batch, depending on the task complexity and provider constraints.

Error handling in batch processing

Batch inference requires robust error handling because failures affect multiple items:

• Implement retry logic for transient failures
• Log failures individually so that specific problematic items can be reprocessed
• Design for partial success — do not discard an entire batch because one item failed
• Validate outputs before writing results to production systems