Natural Batching

What It Does

Natural Batching (originally called “Smart Batching” by Martin Thompson, renamed to avoid confusion) is a consumer-side batching strategy that begins forming a batch the moment the first item arrives in the queue and finalizes the batch when either the queue is empty or the batch reaches a configured maximum size. No timer is used, and the consumer never blocks waiting for a fixed minimum batch size.

The strategy exploits the observation that under real load, more work arrives while the current batch is being prepared — so the consumer can opportunistically collect additional items without waiting. Under no load, the first item is processed immediately with no latency penalty. This gives natural batching a latency profile that is strictly bounded by the batch processing time rather than any fixed timeout value.

Martin Thompson originally documented this as “Smart Batching” in 2011 in the context of the LMAX Disruptor ring buffer. The 2026 Caer Sanders article on martinfowler.com revived and renamed the concept in the context of AI inference serving.

Key Features

• Zero-timeout latency floor: Processing begins on the first item; no waiting for a timeout window to expire.
• Greedy queue drain: Consumer atomically observes and drains all items present at the moment processing starts.
• Maximum size cap: Prevents a single over-full batch from monopolizing the processing thread indefinitely.
• Throughput-latency co-optimization: Under load, larger batches amortize per-batch overhead; under low load, each item is processed immediately — no forced latency.
• Complements the single-writer principle: The single writer thread uses natural batching to process its message queue efficiently.
• GPU inference friendly: Amortizes kernel launch overhead across multiple inference requests without requiring a fixed wait window.

Use Cases

• AI inference serving: Group multiple client embedding or completion requests into a single GPU batch call, starting immediately rather than waiting for a fixed timeout or fixed batch size.
• Database write batching: Accumulate multiple INSERT/UPDATE statements into a single transaction when they arrive concurrently, without blocking single requests under low load.
• Event sourcing / log appends: Flush multiple appended events in a single fsync when concurrent writers produce them, improving I/O efficiency without adding artificial write latency.
• Financial matching engines: Process all pending orders that arrive while the previous match cycle executes, avoiding both per-order processing overhead and fixed tick-rate coupling.

Adoption Level Analysis

Small teams (<20 engineers): Fits whenever there is a batching opportunity — the implementation is simple (drain-queue loop with max-size guard) and the latency improvement is immediate. Useful even for batch database writes or HTTP API fanout.

Medium orgs (20–200 engineers): Fits for ML platform teams building inference servers, message processors, or event pipelines. Low implementation risk; the pattern is composable with existing async architectures.

Enterprise (200+ engineers): Fits across platform teams, especially for shared AI inference infrastructure and high-volume message buses. Well-understood in financial services engineering; increasingly relevant in AI serving infrastructure.

Alternatives

Evidence & Sources

• Martin Thompson — Smart Batching (2011)
• Smart Batching — DZone
• Smart Batching — Java Code Geeks
• Principles of Mechanical Sympathy — Caer Sanders, martinfowler.com (2026)

Notes & Caveats

• Latency model depends on batch overhead assumption. The 100–200µs versus 200–400µs comparison in the Sanders article is model-derived, not empirically measured on a real system. Actual numbers depend heavily on what the batch operation does (GPU inference kernel launch, disk fsync, network round trip).
• Queue must be non-blocking. Natural batching requires a lock-free or wait-free queue (e.g., LMAX Disruptor ring buffer) to avoid the writer and reader serializing on queue operations, defeating the purpose.
• Not suitable for strict SLA guarantee. Under extremely high load, if producers consistently outpace the consumer, batches may always hit max size, causing head-of-line blocking. Back-pressure (producer-side) or concurrency (multiple consumer threads on partitioned queues) must be combined with natural batching.
• GPU-specific caveat: For deep learning inference, batch size affects model accuracy for online learning and can require padding to a power-of-two tensor shape — natural batching’s variable batch size may force padding overhead that partly offsets the latency gain.