What should API rate limiting control?
API rate limiting controls how quickly a client can make requests within a defined policy. For an integration that calls another company’s API, start with one shared request budget for user actions, background work, polling, and retries. Coordinate every worker that consumes the same allowance. A separate limiter in each worker can still overload the shared quota.
The useful engineering decision is how the application behaves when demand exceeds that budget. Which work can wait? Which user action needs capacity now? When should retries stop? A queue can absorb a temporary spike, but it cannot make sustained demand fit an insufficient allowance.
This guide gives an engineering lead a proposed budget worksheet and a worked calculation for a hypothetical integration. The figures are assumptions, not a vendor plan, benchmark, or Hapy deployment result. Use the custom web app integration guide for the wider API access, data ownership, and sync plan.
Find the real scope of the provider’s limits
Record what each limit counts and who shares it before choosing a request rate. A quota may apply to an account, installation, user, endpoint, region, or resource category. A request count can also coexist with a concurrency or compute-cost limit.
GitHub’s REST API documentation illustrates why one headline allowance is incomplete. It distinguishes primary limits from secondary limits on behavior such as concurrent or expensive requests. Some authentication paths share a budget. Check the current rules for the actual account and endpoint rather than copying another application’s number.
Capture these fields in the integration contract:
- Scope and unit: which identity shares the limit, and whether it counts requests, points, concurrent work, or another unit.
- Time and burst behavior: the window, refill rule, and permitted short burst, if documented.
- Counting rules: whether pagination, failures, conditional requests, or batch operations consume allowance.
- Feedback: status codes, response headers, reset-time format, and provider-specific error details.
- Recovery and ownership: the retry ceiling, queue-age limit, escalation owner, and approved capacity-change path.
Distinguish a provider’s target from a strict ceiling. AWS API Gateway documents throttles and quotas as best-effort targets, not guaranteed maximum request counts. Do not use such a target as proof of a hard spending cap or an exact safety boundary.
Allocate capacity before adding workers
Reserve capacity for the work the business must keep usable, then give the remaining allowance to background jobs. Make that allocation at the same scope as the provider’s limit. It is an internal policy, not a request for extra provider capacity.
Assume a hypothetical provider permits 120 equal-cost requests per minute. The team chooses to schedule no more than 100. The remaining 20 are a planning margin, not a guarantee against provider throttling. All callers use the same account, and the example excludes other provider limits.
| Use | Requests per minute | Proposed treatment |
|---|---|---|
| Interactive user actions | 25 | Reserve capacity; show a clear pending state if it is exhausted |
| Scheduled polling | 15 | Spread calls through the window rather than starting them together |
| Retries | 10 | Include retry attempts in the same total and stop at the agreed ceiling |
| New background work | 50 | Dispatch only when the shared budget permits |
| Total scheduled allowance | 100 | Keep all workers within this internal allocation |
The background allocation is 100 − 25 − 15 − 10 = 50 requests per minute. If one record needs five API calls, it supports at most 10 records per minute under these assumptions. Processing 600 records therefore needs at least 60 minutes with no new arrivals, failures, or other constraints.
Pagination changes that estimate. A job that reads a record, fetches several pages of related data, and writes an update does not cost one request merely because the operator sees one task. Count calls or provider-defined points per completed business operation.
Check whether the queue can recover
A queue drains only when its completion capacity exceeds incoming work. In the example, 12 new records per minute exceed the 10-record background capacity. Adding queue storage postpones the failure; it does not resolve the capacity gap.
Even demand below capacity can produce a slow recovery. At eight new records per minute, only two records per minute remain for clearing old work. A backlog of 300 records then needs at least 150 minutes to drain, with the same simplifying assumptions. Compare that delay with the business freshness requirement before approving the design.
If the result is too slow, reduce calls per operation, reduce polling frequency where the business permits it, move eligible work to events or supported batches, or obtain an approved higher allowance. Do not assume batching or caching changes the provider’s accounting; check its documentation.
Enforce one budget across concurrent callers
Use a shared coordinator or an equivalent atomic allocation mechanism when several workers consume one provider quota. Per-worker counters do not enforce an account-wide total. Three workers each allowed 50 requests per minute can attempt 150 against the example’s 100-request internal budget.
The scheduling mechanism should also fit the provider’s timing rules:
- A fixed-window counter limits work within discrete windows but can concentrate requests near a boundary.
- A sliding-window approach accounts for recent activity as time advances.
- A token bucket refills permits over time and allows a configured burst.
- A concurrency limiter controls in-flight requests, not total requests over time.
Microsoft’s rate-limiting guidance distinguishes time-based limiters from concurrency limits. Choose both when the provider constrains both. The name of an algorithm does not prove that its configuration matches the provider’s policy.
Give queued work an expiry or maximum age. Separate urgent and bulk work when needed, but define how lower-priority jobs eventually receive capacity. Otherwise a permanently busy interactive lane can leave reconciliation unfinished.
Handle rate-limit responses without creating more load
When the provider signals a limit, pause the affected work according to its contract. A 429 Too Many Requests response commonly signals rate limiting. Check the documented error details because providers can use other codes too. Do not immediately retry from every worker.
The Retry-After header can contain a delay in seconds or an HTTP date. Parse the documented format; do not mistake an epoch reset timestamp for a number of seconds to sleep. A malformed or missing header needs a conservative, bounded fallback policy, not a tight retry loop.
For example, GitHub’s recovery guidance uses Retry-After when present, its primary reset information when remaining allowance is zero, and a separate fallback for secondary limits. Its rules are provider-specific. Do not apply that timing to every API.
Set both a maximum number of attempts and a maximum elapsed retry age. Confirm whether the SDK already retries so that a job runner does not multiply those attempts with another retry layer. When the budget or deadline is exhausted, preserve the operation’s state and send it to the defined recovery path.
Pacing and duplicate prevention solve different problems. A delayed write can still execute twice if the first result was lost. Keep the same operation identity where supported, and use the idempotency-key retry contract to define safe replay rather than generating a new action on each attempt.
Verify the operating behavior before launch
Test the shared budget under representative concurrent work, not only one successful API call. Use a sandbox or controlled simulation. Do not flood a live third-party API to discover its limit.
For this proposed design, require evidence for these cases:
- Start all workers together. Confirm their combined dispatch stays within the internal allocation and respects the provider’s shorter-window and concurrency rules.
- Fill the bulk queue, then perform a user action. Confirm the reserved path remains usable or reports a truthful pending state.
- Return a rate-limit error with a valid delay, an HTTP date, and no usable timing header. Confirm the documented pause or bounded fallback in each case.
- Restart a worker while work is waiting. Confirm it does not forget the shared pause or replay an uncertain write as a new action.
- Sustain demand above completion capacity. Confirm queue-age alerts identify the delay and an owner acts before work becomes unusable.
Monitor dispatch volume at the quota scope, rate-limit responses, retry attempts, completion rate, and the age of the oldest waiting operation. Report a completed sync only when the downstream result is confirmed. An empty local queue is not sufficient if failed work disappeared from it.
Bring the completed budget and these recovery cases to the build review. Hapy offers business systems automation for teams that need help designing that operating contract. Hapy has a commercial interest in the service.
