Rate-Limiting Middleware in Node and Express — The Token Bucket, the Retry-After Header, and Moving the Counter to Redis

The limiter that works on one server is the limiter that lies on three.

§ IFrame

Today's Ops lesson gave the limiter its contract: a token bucket keyed to an identity, a sustained rate set to the Upstream Budget, a burst for the slack the budget can absorb, and a 429 with Retry-After on refusal. That contract is language-agnostic. Writing it in Node and Express is where the runtime teaches the one thing the contract left implicit, and it is the thing that breaks in production.

Express makes the limiter look trivial. A limiter is middleware, and middleware is the thing Express does best. Ten lines put a working token bucket in front of a route. Those ten lines pass every local test and every staging test, and then they fail silently the moment the service scales past one instance, because the bucket lives in the memory of a single process and every process keeps its own.

This lesson builds the limiter in the order that exposes the trap. First the in-process bucket, correct on one node. Then the demonstration that it multiplies its own rate by the node count. Then the move to Redis, where the counter becomes shared and the arithmetic becomes true again.

§ IILanguage Idiom — Middleware, the Pipeline, and Where the Bucket Lives

Express rests on one idea the Middleware chapter of Web Development with Node and Express states plainly: a middleware function receives the request, the response, and a next callback, and it either ends the response or calls next to pass control down the pipeline. Order is everything. Middleware runs in the order it is registered, and a limiter that must refuse before any work happens has to sit near the top, above the route handlers and above anything that opens a database connection or spends against the Upstream Budget.

A limiter is the purest kind of middleware, because it exercises the fork the chapter describes: inspect the request, decide, and either call next to allow it or end the response with 429 to refuse it. It never renders anything. It is a gate.

The in-process token bucket is a map from identity to a small record: how many tokens remain and when the bucket was last refilled. On each request the middleware computes how many tokens have accrued since the last refill, adds them up to the capacity, and checks whether at least one remains.

function tokenBucket({ capacity, refillPerSec }) {
  const buckets = new Map();
  return function limiter(req, res, next) {
    const key = req.get('x-api-token') || req.ip;
    const now = Date.now();
    const b = buckets.get(key) || { tokens: capacity, last: now };
    const elapsed = (now - b.last) / 1000;
    b.tokens = Math.min(capacity, b.tokens + elapsed * refillPerSec);
    b.last = now;
    if (b.tokens >= 1) {
      b.tokens -= 1;
      buckets.set(key, b);
      return next();
    }
    buckets.set(key, b);
    const waitSec = Math.ceil((1 - b.tokens) / refillPerSec);
    res.set('Retry-After', String(waitSec));
    res.status(429).json({ error: 'rate_limited', retry_after: waitSec });
  };
}

The refill is computed lazily rather than on a timer. There is no interval ticking tokens into every bucket; instead each request asks how much time has passed and credits that much. This is the Node-idiomatic move. The single-threaded event loop has no spare thread to run a refill clock, and it does not need one. Time is a number you subtract on arrival. The Retry-After value falls out of the same arithmetic: the tokens still missing, divided by the refill rate, rounded up to whole seconds. The client is told precisely when the bucket will have a token for it.

§ IIICode Worked Example — Why One Node Lies, and Redis Tells the Truth

Register the middleware and it works. On one node.

const app = express();
app.use('/api', tokenBucket({ capacity: 5, refillPerSec: 1 }));
app.get('/api/quote', handler);

Now scale to three nodes behind a load balancer, which is the entire point of a stateless Node service. The buckets map is a plain JavaScript object in each process's heap. Three processes, three maps, three independent counts. A tenant limited to one request per second now gets three, one per node, because the load balancer sprays their requests across all three and each node believes it is the only one counting. The limiter did not fail loudly. It quietly enforced three times the rate it promised, and the Upstream Budget the Ops lesson worked so hard to protect is overspent by exactly the fleet size.

The fix is to take the counter out of the process and put it somewhere all processes share. Redis is the ordinary choice, and the reason is not speed. It is atomicity. The bucket update is a read-modify-write: read the token count, subtract one, write it back. If two requests on two nodes read the same count before either writes, both think a token was available and both proceed, and the limit is breached under exactly the concurrency it exists to handle. The read-modify-write must be atomic, meaning no other request can interleave between the read and the write.

Redis gives atomicity through a Lua script that runs on the server as a single indivisible step. The script does the whole token-bucket calculation — refill by elapsed time, check for a token, decrement if present — and returns whether the request is allowed and how long to wait if not. Because the script is one atomic operation, no two nodes can both see the same last token.

const BUCKET = `
local key      = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local state    = redis.call('HMGET', key, 'tokens', 'last')
local tokens   = tonumber(state[1]) or capacity
local last     = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + (now - last) / 1000 * refill)
if tokens >= 1 then
  tokens = tokens - 1
  redis.call('HMSET', key, 'tokens', tokens, 'last', now)
  redis.call('EXPIRE', key, 3600)
  return {1, 0}
else
  local wait = math.ceil((1 - tokens) / refill)
  redis.call('HMSET', key, 'tokens', tokens, 'last', now)
  redis.call('EXPIRE', key, 3600)
  return {0, wait}
end`;

The Express middleware becomes a thin wrapper. It builds the key, calls the script through the Redis client with the bucket parameters and the current time, and reads back the allow flag and the wait. On allow it calls next; on refuse it sets Retry-After and answers 429. The bucket arithmetic is identical to the in-process version. Only its home changed, from one heap that lies to one store that tells the truth to every node at once.

The EXPIRE on the key matters more than it looks. A limiter keyed on API token or IP accumulates one Redis key per distinct caller, and callers are unbounded. Without expiry the key space grows without limit and Redis eventually runs out of memory, which converts a rate limiter into an outage. Expiring idle buckets after an hour bounds the key space to the active caller set. The limiter that protects the Upstream Budget must not become the thing that exhausts its own store.

§ IVConnection to Today's Ops Lesson

The Ops lesson named the shared-state caution in one sentence: a per-node counter multiplies the true rate by the node count, so the bucket must count against shared state or the guarantee is a fiction. This lesson is that sentence made concrete and then repaired. The in-process Map is the fiction; three nodes enforce three times the promised rate. The Redis Lua script is the repair; one atomic read-modify-write across the fleet restores the arithmetic the Ops lesson depends on. The Retry-After value is computed identically in both lessons, because the client-facing contract does not change when the counter moves. The client neither knows nor cares whether the bucket lives in a heap or in Redis. It reads its wait and returns on schedule.

§ VPrior-Lesson Reach

The 06-21 graceful-shutdown lesson taught that middleware order decides behavior and that Express layers on the core http server. The limiter is that ordering discipline applied to admission rather than shutdown: it must sit above the handlers so its refusal costs nothing, exactly as the readiness flag had to be readable before the drain began. The 06-28 Express-endpoints lesson built the request-inspection pattern the limiter reuses — reading a header, deciding, ending or forwarding. The 07-01 dependency-trust lesson is the caution against reaching for a limiter package without reading it: a rate limiter runs on every request to your service, so an unaudited limiter dependency with a malicious install script sits at the most privileged spot in the pipeline. Build the bucket from primitives you understand, or vet the package as if it guarded the door, because it does.

§ VIClosing

Express makes the limiter look like ten lines, and on one node it is. The runtime's lesson is that a stateless service is a promise of more than one node, and the moment the second node exists, the in-process counter enforces a rate the fleet-wide caller never agreed to. The move to Redis is not an optimization. It is the difference between a limiter that keeps its promise and one that silently multiplies its rate by however many machines you happen to be running. The atomic read-modify-write is the whole game, and the key expiry is what keeps the limiter from becoming the outage.

Find the rate limiter in your service. Check where its counter lives. If it is a variable in the process, count your instances and multiply — that is the rate you are actually enforcing, not the number in the config. Move the counter before you add the third node, not after the upstream suspends your account.

Paired lessons → Ops 01-Earth-DevOps/.../2026-07-12-rate-limiting-at-the-edge-... · Cert Cert-Prep/Anthropic/2026-07-12-rate-limits-and-cost-governance-...