Web development

API rate limiting: design and cost

A rate-limit check costs 0.222 µs and 121 bytes per key: the design work sits in the key and the 429, not in micro-optimisation.

Key points

  • A million fixed-window checks cost 221.9 ms, which is 0.222 µs per check; a token bucket costs 271.7 ms.
  • Each key takes 121 bytes of heap: a million keys weigh about 121 MB.
  • Cleaning a million expired keys blocks the loop for 244.3 ms.
  • RFC 6585, April 2012, defines the 429 code; RFC 9110, June 2022, defines Retry-After.

A rate limit is a contract with three parts: a key, a window and a cost per request. The check itself is cheap, 0.222 µs per call in memory; the design work sits in what you count, where you keep the counter and what you return when the limit trips.

Choose the key before the number

The key decides who pays for the limit. For an authenticated client it is the user or API key identifier; for an anonymous one it is the IP address, possibly combined with the route. A per-IP limit punishes users behind a shared proxy, a per-user limit does not stop a distributed attack. Pick the key that matches the resource you are protecting: the database, the payment provider, the email queue.

Three algorithms, three trade-offs

A fixed window counts requests in equal intervals: it is the simplest and it allows a double burst across the boundary, because the counter restarts at zero. A token bucket refills at a constant rate and allows a burst up to capacity: it costs one state per key and two arithmetic operations. A sliding window weights previous requests and is more precise, but it has to store timestamps. The three implementations measured here land in the same order of magnitude.

The cost per check

At 0.222 µs per check, a single core can do on the order of millions of checks per second: the bottleneck is not the arithmetic, it is memory and coordination between instances. With 100,000 active keys the footprint is about 12 MB; with a million it reaches 121 MB. If the counter lives in one process, every instance counts for itself and the effective limit is the instance count multiplied by the declared limit.

What to return when the limit trips

RFC 6585, April 2012, defines the 429 Too Many Requests code; RFC 9110, HTTP Semantics, June 2022, describes Retry-After, which tells the client when to try again. Also return how many tokens remain and what the limit is, in documented headers, so the client can slow down before hitting the wall. Do not use 403: the client is authenticated and the problem is temporary.

for i in $(seq 1 8); do curl -s -o /dev/null -D - "https://api.example.com/v1/orders" | grep -iE "^(HTTP|retry-after|ratelimit)"; done

Cleanup is part of the design

Accumulated keys have to be removed, and the scan is not free: 244.3 ms for a million expired keys, all at once, on the event loop. Split cleanup into small chunks, run it on an interval, and use a structure ordered by expiry when keys are many. Or let keys expire in the datastore with a TTL and pay the cost where it does not stop requests.

The limit: this is not DDoS defence

An application rate limit protects a service from a misconfigured client or moderate abuse; it does not stop a volumetric attack, which is stopped earlier, in the network. And with several instances an in-memory counter is not a global limit: either move the state to a shared datastore, or accept that the real limit is higher. Both choices must be written down, because the number promised in the documentation has to match the one enforced.

Where to start

    Frequently asked questions

    429 or 403 when the limit trips?

    429. The client is authenticated and allowed, but has exceeded a temporary quota; 403 would say permission is missing, and the client would stop retrying. Add Retry-After to say when to come back.

    Where do I keep the counters?

    In memory if you run one instance, in a shared datastore as soon as you run two. With local state each instance applies its own limit, so the effective limit is the sum, not the documented number.

    What limit should I choose?

    The one the service survives, measured. Load the system to a known capacity, watch where latency grows and set the limit below that point. A number copied from another API says nothing about your system.

    Does a rate limit stop a DDoS?

    No. It stops a client that overdoes it and moderate abuse; a volumetric attack is absorbed in the network or at the CDN. An application rate limit is the last layer, not the first.

    How do I handle a legitimate burst?

    With a token bucket: the bucket allows a burst up to capacity and then enforces the average rate. A fixed window lets twice the limit through across the boundary, which is usually not what you want.