Ruby's Memoization and a TTL-Aware Cache Layer — Conditional Assignment, Hash Default Blocks, and the Staleness Guard for Inference Caching

Day / pairMon — Backend Stack / Ruby refracting DevOps + Cognition
Lesson classDev — Ruby (Backend Stack)
Shipped2026-06-22T05:30Z — MD + HTML in-cycle

Ruby's Memoization and a TTL-Aware Cache Layer — Conditional Assignment, Hash Default Blocks, and the Staleness Guard for Inference Caching

§I — Frame

Today's Ops lesson named three caching primitives: the prompt cache that reuses computation, the semantic cache that reuses answers, and the staleness guard that decides when a cached answer has gone wrong. Three primitives, one rule: cache the computation freely, cache the answer carefully, never cache anything you cannot expire.

Ruby has opinions about all three, and they are unusually direct. Most languages express "compute this once, then reuse it" with a helper, a decorator, or a library. Ruby expresses it with a two-character operator that reads like punctuation. The language treats cache-or-compute as a first-class idiom rather than a pattern to import. That directness is a gift and a trap: the gift is that the simplest cache in Ruby is almost invisible; the trap is that the invisible cache has no expiry, and a cache with no expiry is the exact thing the Ops lesson warned against.

This lesson builds the cache up in three moves. First, ||=, the fetch-or-compute that fits in one operator. Second, the Hash default block, the populate-on-miss that turns a hash into a memo table. Third, a Data.define entry carrying its own expiry, the staleness guard that turns a memory leak waiting to happen into a cache you can actually run in production.

§II — Language idiom: ||= is fetch-or-compute

The conditional assignment operator ||= is Ruby's prompt cache in miniature. The expression @answer ||= compute reads as "if @answer is already set, use it; otherwise compute it, store it, and use that." It expands to @answer || (@answer = compute). The right-hand side runs once, on the first call, and never again while the value stays truthy.

This is exactly the fetch-or-compute shape the prompt cache implements: check for the stored result, return it on a hit, compute and store on a miss. Ruby collapses the whole branch into one operator. A method that wraps an expensive call in ||= pays the cost on its first invocation and returns the stored value on every call after, for the lifetime of the object that holds the instance variable.

class SystemPrompt
  def encoded
    @encoded ||= expensive_encode(@text)
  end
end

The discipline lives in two cautions. The first is the falsy-value trap: ||= recomputes whenever the stored value is nil or false, so a cache whose legitimate answer is false will recompute forever and never hit. The second is lifetime: the instance variable lives as long as the object does, and an object that lives as long as the process is a cache that never frees its memory. ||= is the right tool when the value is genuinely immutable for the object's life — an encoded system prompt that does not change between deploys. It is the wrong tool for anything that should expire.

§III — Language idiom: the Hash default block is populate-on-miss

A single ||= caches one value. A fleet caches many, one entry per distinct prompt, per distinct query. Ruby's Hash constructed with a block is the memo table built for exactly this.

Hash.new { |hash, key| hash[key] = compute(key) } creates a hash whose block runs on a missing key, computes the value, stores it under the key, and returns it. The next access to the same key skips the block and returns the stored value. The hash is now a cache keyed by whatever you look up, populating itself on every miss.

prompt_cache = Hash.new do |cache, prefix|
  cache[prefix] = expensive_encode(prefix)
end

prompt_cache["You are a documentation agent..."]

Flanagan's treatment of methods and procs is the ground here: the block passed to Hash.new is a Proc that closes over its surrounding scope, so the compute function can reference whatever the cache needs without being handed it on every call. The block is not a default value. It is a default behavior, run lazily, once per novel key. The distinction matters because Hash.new(expensive_encode(x)) evaluates the argument once at construction and hands the same object to every miss, while Hash.new { ... } runs the block per miss. For a cache, only the block form is correct.

This is the semantic cache's populate-on-miss, minus the embedding search. The real semantic cache would key on a similarity match rather than an exact key, but the populate-on-miss control flow is identical: look up, return on hit, compute-and-store on miss. The Hash default block gives you that flow for free, and it scales to as many entries as the keyspace holds, which is also its problem. An unbounded memo table is an unbounded memory commitment, and the next move fixes it.

§IV — Code worked example: the TTL-aware cache with a staleness guard

The Ops lesson was blunt: never cache anything you cannot expire. The bare Hash cache cannot expire anything. Every key it sees, it holds forever. To run it in production it needs the staleness guard, an expiry per entry and an eviction path.

Start with the entry. Ruby's Data.define builds an immutable value object, the right shape for a cache record that carries a value and the moment it goes stale.

Entry = Data.define(:value, :expires_at) do
  def fresh?(now = Time.now)
    now < expires_at
  end
end

Now the cache wraps a hash of entries, checks freshness on read, and computes-and-stores on a miss or a stale hit. Newman's invalidation rule is the spine: the hard part is not storing the value, it is knowing when to stop trusting it. The cache below trusts an entry only while fresh? holds.

class TTLCache
  def initialize(ttl_seconds)
    @ttl = ttl_seconds
    @store = {}
  end

  def fetch(key)
    entry = @store[key]
    return entry.value if entry&.fresh?

    value = yield
    @store[key] = Entry.new(value:, expires_at: Time.now + @ttl)
    value
  end
end

The fetch method is the fetch-or-compute of ||= raised to a keyed, expiring cache. It reads the entry, returns the value only if an entry exists and is still fresh, and otherwise runs the supplied block, stores the result with a fresh expiry, and returns it. A stale hit is treated as a miss: the old value is overwritten rather than served. This is the time-to-live floor the Ops lesson named.

Event-driven invalidation is the second eviction path, and it is one method.

class TTLCache
  def invalidate(key)
    @store.delete(key)
  end

  def flush
    @store.clear
  end
end

When the documentation source updates, the fleet calls invalidate on the affected key. When a model rollback fires, the 06-15 un-promotion path, the fleet calls flush, because the pinned model did not produce the cached answers and none of them can be trusted. The cache exposes both the slow clock (TTL) and the fast switch (event invalidation), which is precisely the two-control discipline the Ops staleness guard demanded.

Using it ties the trio together:

cache = TTLCache.new(6 * 3600)

answer = cache.fetch(query_key) do
  call_model(query)
end

Six-hour TTL, compute on miss or stale, one block that names the expensive work. The same control flow as ||=, now with an expiry it can state and an invalidation it can fire.

§V — Connection to today's Ops lesson

The Ops lesson's three primitives map one-to-one onto Ruby's idioms. The prompt cache, identical output paid in memory, is ||= on an immutable value: the encoded prefix that never changes for the object's life. The semantic cache, whole calls skipped on a populate-on-miss, is the Hash default block, with the exact-key lookup standing in for the embedding match. The staleness guard, the discipline that makes both safe, is the TTLCache with its fresh? check, its TTL floor, and its flush path wired to the rollback runbook.

The refraction also exposes where Ruby's directness misleads. The language makes the unsafe cache, ||= with no expiry, Hash.new with no bound, the easy one to write. The safe cache takes the extra structure of the Data.define entry and the freshness check. Ruby will let you ship the memory leak in two characters. The discipline is to spend the extra ten lines that let you expire what you cache.

§VI — Prior-lesson reach

The 06-12 Ruby lesson built strategy-attribution rollups on Enumerable, Comparable, and BigDecimal, opening the Ruby track on lazy pipelines and money-safe arithmetic. This lesson stays in the same register, small, composable objects doing one job, and adds the language's memoization family. The Data.define entry here is the same value-object discipline the 06-12 lesson used for its attribution records, now carrying an expiry instead of a return figure. Two Ruby lessons, one posture: immutable records, lazy computation, and the standard library doing the structural work before any gem is reached for.

§VII — Closing discipline

Ruby gives you the cache in one operator and the leak in the same breath. ||= is fetch-or-compute. The Hash default block is populate-on-miss. Neither expires anything, and a cache that cannot expire is the failure the Ops lesson named first. The TTLCache is the ten lines that buy back safety: an entry that knows when it goes stale, a TTL floor, and a flush wired to rollback.

Write the two-character cache when the value is truly immutable. Write the ten-line cache the moment it is not. Read the difference before production reads it for you.

Related

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura