How to use this calculator
- Enter or choose rate limit.
- Enter or choose concurrent requests in flight.
- Enter or choose average latency per request.
- Enter or choose requests to process.
- The achievable throughput updates as you type. Use Copy result or Share to keep it.
Method
Concurrency ceiling = Concurrency × 60,000 ÷ Latency ms; Throughput = min(Rate limit, Concurrency ceiling); Concurrency needed = ⌈Rate limit × Latency ms ÷ 60,000⌉
- Latency ms
- average time from sending a request to receiving the full response
- Rate limit
- requests per minute allowed
Worked example
- 5 requests in flight, each taking 1 s: 5 × 60 = 300 per minute.
- That is below the 600/min limit, so concurrency is the bottleneck; 600 × 1 s ÷ 60 = 10 concurrent requests would reach the limit.
- 3,000 requests at 300/min take 10 minutes.
With 5 concurrent 1-second requests you reach 300 requests a minute, half the 600 limit; double the concurrency to use it all.
What the result means
If concurrency is the bottleneck, add parallel workers up to the needed figure. If the rate limit binds, more concurrency only causes 429 errors: batch requests, cache results, or ask for a higher limit.
Assumptions and limits
- Latency stays the same as concurrency rises.
- The rate limit is enforced smoothly over the minute.
- Token-based limits (such as tokens per minute on LLM APIs), burst limits and retries are not modelled.
Questions
What is Little’s law?
In a stable system, items in progress = arrival rate × time each spends in the system. Here: concurrent requests = requests per second × latency in seconds.
Should I run at exactly the limit?
Aim slightly below it, for example 80–90%, with back-off on 429 responses, because latency and limits fluctuate.