Rate Limiting at the Edge — Token Buckets, the 429 Contract, and Protecting the Upstream Budget

A limiter is not a wall. It is a promise about how much you can afford to say yes to.

§ IFrame

Last Sunday's lesson pushed data outward: server-sent events and WebSockets streaming live market prices to the browser, with backpressure holding the frontend to a speed the render loop could survive. That lesson answered how to stop the client from drowning. It left the mirror question open. When ten thousand clients each open a stream and each asks for more, what stops the backend from drowning?

The answer is a limiter at the edge, and the edge is the right place for it. A request that a limiter refuses at the edge costs the origin nothing. It never touches the application server, never opens a database connection, never spends a token against a metered upstream API. A request that gets through and then fails deep in the stack has already spent everything it was going to spend. The limiter earns its keep by refusing early, cheaply, and honestly.

This is pure reliability work, no domain refraction, which is what Sunday is for. Rate limiting is the same shape whether the traffic is trade orders, agent tool-calls, or login attempts. Name the shape once and it travels across every scale.

§ IIFoundations — The Three Questions Every Limiter Answers

Before any algorithm, a limiter answers three questions. Get these wrong and the algorithm cannot save you.

Who is the bucket? A rate limit counts against a key. The key can be a client IP, an API token, a user account, or the pair of an account and a specific route. The choice decides who shares a limit with whom. Key on IP and a whole office behind one NAT gateway shares a single bucket, so one heavy user starves the rest. Key on the API token and each caller carries its own allowance, which is what a metered API almost always wants. Nginx builds this key explicitly: a limit_req_zone is defined over a variable such as $binary_remote_addr, and that variable is the identity of the bucket. Choose the key before you choose the rate.

What are you protecting? A limiter defends a finite resource behind it. Name that resource. It might be CPU on the origin, connection slots on a database, or the case that governs the whole trio today: a quota of tokens on a paid upstream API that bills per call. Call this finite thing the Upstream Budget. The limiter's whole job is to keep the arriving demand under what the Upstream Budget can pay for. A limiter that protects nothing specific is theater; it throttles traffic to a number nobody chose for a reason nobody remembers.

What happens to the refused request? This is the question most limiters answer badly. A refused request can be dropped silently, delayed until the bucket refills, or rejected with an honest status the client can act on. The honest rejection is a contract, and it has a name in the HTTP standard: 429 Too Many Requests, carried with a Retry-After header telling the client exactly how long to wait. A limiter that drops silently teaches clients to hammer harder. A limiter that answers 429 with Retry-After teaches clients to back off on a schedule. The status code is not decoration. It is the instruction the client obeys.

§ IIIMechanism — The Bucket That Refills

Two algorithms dominate, and they are duals of the same idea.

The leaky bucket models the limiter as a bucket with a hole. Requests pour in at the top; the bucket drains at a fixed rate through the hole. If the inflow exceeds the drain and the bucket fills, the overflow spills, and spilled requests are refused. The output is perfectly smooth because the hole never widens. This is exactly Nginx's limit_req model, which the manual describes as smoothing request processing to a steady rate and rejecting what arrives faster than the configured drain.

The token bucket inverts the picture. A bucket holds tokens up to some capacity. Tokens are added at a fixed refill rate. Every arriving request must take one token to proceed; if the bucket is empty, the request is refused. The refill rate sets the sustained throughput. The bucket capacity sets something the leaky bucket cannot express on its own: the burst.

Burst is the difference that matters in production. Real traffic is not smooth. A user loads a page and fires eight requests in the same instant, then goes quiet for a minute. A pure smooth limiter rejects seven of those eight even though the user's average rate is trivially low. The token bucket, holding a capacity above one, lets the burst of eight drain the accumulated tokens at once, then refills over the quiet minute. Nginx models this with the burst parameter on limit_req, and pairs it with nodelay when the burst should be served immediately rather than queued. Set the sustained rate to what the Upstream Budget can pay for over time; set the burst to how much slack the budget can absorb in a spike. Two numbers, two distinct decisions.

The token bucket also composes cleanly across layers, which the Nginx manual demonstrates by allowing multiple limitations to apply at once: a per-IP request-rate limit and a separate per-connection transfer-rate limit stacked on the same traffic. The edge can hold a coarse global bucket while a per-token bucket sits behind it. Each bucket protects a different Upstream Budget.

§ IVWorked Example — Two Buckets Guarding a Metered API

Take the case the whole trio shares. An origin service proxies calls to a paid upstream API that bills per request and enforces its own hard rate ceiling. Cross that ceiling and the upstream returns 429 to you, and if you keep pushing, it can suspend the account. The Upstream Budget here is literal money and a literal quota. The edge limiter's job is to guarantee the origin never sends the upstream more than it can pay for, even under a stampede.

Two buckets, stacked. The first is coarse and global, keyed on client IP, sitting at the very edge, with a generous rate whose only purpose is to blunt a crude flood before it costs anything downstream. In Nginx this is a limit_req_zone on $binary_remote_addr with a high rate and a modest burst. It is the doorman, not the accountant.

limit_req_zone $binary_remote_addr zone=edge:10m rate=50r/s;
limit_req_zone $http_x_api_token   zone=tenant:10m rate=1r/s;

server {
  location /api/ {
    limit_req zone=edge burst=100 nodelay;
    limit_req zone=tenant burst=5;
    limit_req_status 429;
    proxy_pass http://upstream_metered;
  }
}

The second bucket is precise, keyed on the API token, and its rate is derived directly from the Upstream Budget. If the upstream allows 600 calls per minute across the whole account and the account serves ten tenants, each tenant's token gets a bucket refilling at roughly one token per second with a small burst. Now the arithmetic is guaranteed: the sum of every tenant bucket's sustained rate sits under the upstream ceiling by construction. The edge cannot physically forward more than the upstream can accept, because the tokens to do so do not exist.

The SRE discipline names the deeper principle. Preventing server overload is not about surviving the flood after it arrives; it is about ensuring the arriving load is bounded before it can arrive. A limiter keyed to the budget makes overload arithmetically impossible rather than merely survivable. That is the difference between a system that degrades gracefully and one that only hopes to.

One caution the manual is explicit about. A limiter's counter is state, and state that is per-node does not hold across a fleet. Run three edge nodes each enforcing "one token per second" independently and the true rate the upstream sees is three per second. The bucket must count against shared state, a single counter store all nodes read and write, or the guarantee is a fiction. Today's Dev lesson takes exactly this problem into Node and Express and moves the counter to Redis to make it true across the fleet.

§ VConnection to Prior Lessons

The 07-05 lesson held the frontend to a speed the render loop could survive; backpressure was the client protecting itself from a fast producer. Rate limiting is the same instinct pointed the other direction: the server protecting itself from fast consumers. Backpressure and rate limiting are one principle, bound the flow to what the slow side can afford, applied at the two ends of the same wire. Name it once: the slow side sets the pace, and something at the boundary enforces it.

The 06-28 API-contract lesson built structured, predictable responses across the frontend-backend boundary. The 429-with-Retry-After reply is that discipline extended to the failure path. A limiter that refuses without a status code and a wait-time has broken the API contract precisely where the client most needs it to hold, because a client that cannot read its refusal cannot obey it.

The 06-21 zero-downtime lesson flipped a readiness flag to 503 to tell the load balancer to stop routing during a drain. The 429 is the same move at request granularity rather than instance granularity: a machine-readable "not now, try again in N seconds" the caller is built to respect. Both are honest refusals that keep the system inside its budget.

§ VIConnection to Today's Dev Lesson

The Node and Express lesson implements this edge limiter as middleware and confronts the shared-state problem head-on. It builds the token bucket in the request pipeline, sets the Retry-After header from the same arithmetic used above, then moves the counter out of process memory and into Redis so a horizontally scaled fleet enforces one true rate instead of one rate per node. The Ops contract is the what and why; the Dev lesson is the how it becomes real when there is more than one server.

§ VIIClosing

A rate limiter is a promise about how much you can afford to say yes to, made cheaply and kept honestly. The promise rests on three decisions made before any code: who the bucket counts against, what finite resource it protects, and what the refused request is told. Get those right and the algorithm is almost incidental, a token bucket with a sustained rate set to the Upstream Budget and a burst set to the slack that budget can absorb. Get them wrong and no algorithm rescues you, because you are throttling the wrong identity to protect nothing in particular and telling the client nothing it can use.

Open your edge config. Find the limit. Ask it the three questions: who is the bucket, what does it protect, what does the refused caller hear. If any answer is "I'm not sure," the limit is a number someone typed once and never defended. Give it a reason or take it out.

Paired lessons → Dev Polyglot-Dev/Web/2026-07-12-rate-limiting-middleware-in-node-and-express-... · Cert Cert-Prep/Anthropic/2026-07-12-rate-limits-and-cost-governance-in-the-claude-agent-harness-...