Pub/Sub subscriptions — the 31-day deletion clock

A subscription nobody reads is a subscription Google will delete.

<!-- hal:authoritative:yaml -->

A subscription nobody reads is a subscription Google will delete. Find out which ones are on the clock before the month-end job does.

§I. Frame

Unless topic retention is turned on, a Pub/Sub topic keeps no backlog of its own. Each subscription holds its own backlog, and every subscription on a topic gets its own copy of each message. Kleppmann calls this pattern fan-out: each message is delivered to all of the consumers, so independent readers can "tune in" to the same broadcast without affecting each other (DDIA ch11, printed p.430).

That independence has a price. A subscription is a resource with a life of its own, and the Pub/Sub subscription-properties page says how that life ends: "Subscriptions without any subscriber activity or changes made to the subscription properties expire." The default expiration period is 31 days. Open connections, active pulls, successful pushes, or any property update restart the clock.

A nightly consumer never notices. A quarterly reconciliation job does. Its subscription goes quiet after the close, Pub/Sub deletes it on day 31, and on day 90 the job finds NOT_FOUND. Recreating the subscription does not bring the messages back. The delete_subscription docs state it plainly: a new subscription with the same name "has no association with the old subscription."

The problem for today: read every subscription in a project, report which ones are on a deletion clock, which ones lost their topic, and whether a replay could recover acknowledged messages.

§II. The Two Clocks

The Two Clocks (named technique). Every subscription runs two timers, and they measure different things.

1. Retention clock, per message. message_retention_duration counts from publish time. Default 7 days, minimum 10 minutes, maximum 31 days. Past it, Pub/Sub may discard the message whether or not anyone acked it.
2. Deletion clock, per subscription. expiration_policy.ttl counts from the last subscriber activity or property change. Past it, the whole subscription goes.

The two are linked by one rule from the properties page: the expiration period "must be at least as long as the message retention duration."

The deletion clock has three states in the API, and only one of them looks like a value:

So an absent field means "31 days" and an empty policy means "never." A census that treats both as "no policy" gets the most important row wrong.

From gcloud (550.0.0 --help on the lab Mac):

gcloud pubsub subscriptions update recon-quarterly \
  --expiration-period=never \
  --message-retention-duration=7d

--expiration-period accepts INTEGER[UNIT] or the special value never.

§III. Orphans and detached subscriptions

Deleting a topic does not delete its subscriptions. The API marks them instead: the topic field "will be _deleted-topic_ if the topic has been deleted." Such a subscription receives no new messages, yet it still exists and still sits on its own deletion clock.

A detached subscription is a different state reached on purpose. Per the model docs, detached subscriptions "don't receive messages from their topic and don't retain any backlog," and pulls return FAILED_PRECONDITION.

Three outcomes follow for an operator:

1. If a subscription shows _deleted-topic_, then its reader is either gone or broken. Delete it or find the owner.
2. If a subscription is detached, then someone chose that. Record who before you touch it.
3. If a live subscription sits on a deletion clock and its reader runs less often than the ttl, then set --expiration-period=never or shorten the gap.

§IV. The census, in one cargo script

pubsub-expiry-census.rs sits in this bundle. The core is a three-way match on the policy:

enum Expiry {
    Default31d,     // expiration_policy unset: server applies ttl = 31 days
    Never,          // expiration_policy set, ttl unset
    AfterDays(i64), // expiration_policy.ttl set explicitly
}

fn expiry(s: &Subscription) -> Expiry {
    match &s.expiration_policy {
        None => Expiry::Default31d,
        Some(p) => match &p.ttl {
            None => Expiry::Never,
            Some(ttl) => Expiry::AfterDays(ttl.seconds() / DAY),
        },
    }
}

The listing loop uses the official Google Cloud Rust client:

let client = SubscriptionAdmin::builder().build().await?;
let mut subs = client
    .list_subscriptions()
    .set_project(format!("projects/{project}"))
    .by_item();
while let Some(s) = subs.next().await.transpose()? {
    // one row per subscription
}

Four mechanics:

• **by_item() needs a trait in scope.** It comes from google_cloud_gax::paginator::ItemPaginator, imported as _. Without that line the method does not exist, which is why the manifest names google-cloud-gax next to google-cloud-pubsub.
• Credentials come from Application Default Credentials. The builder reads ADC unless you call with_credentials(). Grant the caller roles/pubsub.viewer and nothing more.
• The replay column reads two fields. A seek to a past time can restore acked messages only if retain_acked_messages is true or the topic keeps its own retention, which the subscription reports as topic_message_retention_duration.
• Crate name history. google-cloud-pubsub used to be a community crate; its author donated the name to Google, and the old code lives on as gcloud-pubsub. Pin "1" to get the official client.

What was checked, and what was not. A scratch crate built from the frontmatter manifest and body passed cargo check and cargo clippy -- -D warnings on stable cargo 1.98.1 (box) and 1.96.0 (the lab Mac), resolving google-cloud-pubsub 1.5.0, google-cloud-gax 1.15.0 and tokio 1.53.1. The -Zscript entry ran on the lab Mac under nightly cargo 1.101.0 with a placeholder project id. It built, found the lab Mac's ADC file, and stopped at token refresh with invalid_grant, because the stored user credential is stale. No Pub/Sub call completed. A project with one quiet batch subscription and one orphan would read like this (illustrative):

recon-quarterly expiry=31d(default) retention=7d replay=unacked-only dlq=no
orders-audit expiry=never retention=7d replay=acked+unacked dlq=yes
legacy-billing expiry=31d(default) retention=7d replay=unacked-only dlq=no ORPHAN
subscriptions=3 on_deletion_clock=2 orphaned=1 detached=0

§V. Wrap it with Nix

pubsub-expiry-census.nix:

{ pkgs ? import <nixpkgs> { } }:
pkgs.writers.writeBashBin "pubsub-expiry-census" ''
  exec cargo +nightly -Zscript ${./pubsub-expiry-census.rs} "$@"
''

Built on the lab Mac, it produced /nix/store/ywh2z5ig…-pubsub-expiry-census, a two-line bash script that execs cargo on the store copy of the .rs file. The wrapper pins the script, not the toolchain: cargo +nightly still resolves through rustup on the host.

§VI. What not to do

1. Reading a missing expiration_policy as "never expires." It means 31 days.
2. Recreating a deleted subscription and expecting its backlog back.
3. Deleting a topic to retire a pipeline and leaving its subscriptions behind as orphans.
4. Setting an expiration period shorter than the retention duration. The properties page requires expiry to be at least as long as retention.
5. Giving the census roles/pubsub.editor because the viewer role "might not be enough."

§VII. Close instruction

Run the census against a project you own. For each row on a deletion clock, write down how often its reader runs. Any subscription whose reader runs less than once per ttl gets --expiration-period=never today, and every ORPHAN row gets an owner or a delete by Friday.

Related

• Dev: mpsc fan-out, one thread per subscription (same trio)
• Cert: PCA Pub/Sub delivery, ordering, seek (same trio)
• Prior Cloud Ops: StackSet census in cargo script
• Prior Cloud Ops: SQS DLQ and redrive census
• — ch11 Messaging systems, fan-out and acknowledgements