Azure Service Bus queues — max delivery and the dead-letter side queue

A message that fails ten times is sitting in a side queue with a reason code.

<!-- hal:authoritative:yaml -->

A message that fails ten times is not missing. It is sitting in a side queue with a reason code. Count those before you raise the delivery limit.

§I. Frame

Azure Service Bus is the broker that keeps a message until a receiver settles it. Bootcamp az-900 README: it moves data asynchronously between senders and receivers, with queues for competing consumers and topics plus subscriptions for fan-out. Kleppmann's acknowledgement rule still applies (DDIA ch11, printed p.431): until the broker hears an ack, it may redeliver.

Service Bus adds a second queue that SQS callers often confuse with a redrive policy. Every queue (and every subscription) has a dead-letter sub-queue. Messages land there when delivery count exceeds maxDeliveryCount, when TTL expires and dead-lettering on expiration is on, or when a subscription filter throws. The active queue looks empty. The work is still in the namespace.

The problem for today: list every Service Bus namespace in a subscription, walk every queue, and report the ones whose delivery ceiling, dead-letter depth, session flag, or expire-to-DLQ setting needs an owner.

§II. The Side Queue

The Side Queue (named technique). Treat the dead-letter sub-queue as a first-class queue that shares the entity name and keeps its own depth.

Four ARM fields on SbQueueProperties decide the story:

Peek-lock vs receive-and-delete is a client receive mode. It does not appear on the queue resource. The census therefore cannot invent a "mode" column from ARM. It can report lockDuration (default PT1M, max five minutes) because that is a queue property that bounds how long a peek-lock hold may last.

From the dead-letter docs: after maxDeliveryCount failed receives, Service Bus moves the message to the dead-letter sub-queue and stamps it with a reason. Recreating the queue does not move those messages back. You read them from $DeadLetterQueue (or the SDK dead-letter receiver) and decide.

§III. Shape the census

Inputs: AZURE_SUBSCRIPTION_ID plus whatever DefaultAzureCredential can resolve (env service principal, managed identity, or az CLI cache). Role needed: Reader on the namespaces, or a custom role that can list Microsoft.ServiceBus/namespaces and .../queues.

Algorithm:

1. namespaces.list(subscription_id) across the whole subscription.
2. Parse resourceGroups/{rg} out of each namespace id.
3. queues.list_by_namespace(rg, ns, subscription_id) per namespace.
4. Flag a queue when any of: max_delivery_count != 10, dead_letter_message_count > 0, requires_session == true, or dead_lettering_on_message_expiration == false.

Quiet queues with default delivery, empty DLQ, no sessions, and expire-to-DLQ on stay silent. The print is the exception list, then a one-line summary.

§IV. cargo script + Azure Rust SDK

servicebus-queue-census.rs (cargo script frontmatter abbreviated):

use azure_identity::{DefaultAzureCredential, TokenCredentialOptions};
use azure_mgmt_servicebus::Client;
use futures::stream::StreamExt;

let credential: Arc<dyn TokenCredential> =
    Arc::new(DefaultAzureCredential::create(TokenCredentialOptions::default())?);
let endpoint: Url = "https://management.azure.com".parse()?;
let scopes = vec!["https://management.azure.com/.default".to_string()];
let client = Client::new(endpoint, credential, scopes, azure_core::ClientOptions::default());

let mut pages = client.namespaces_client().list(&sub).into_stream();
// ... for each namespace, queues_client().list_by_namespace(&rg, &name, &sub)

Four mechanics:

• **Client::new takes an ARM endpoint URL, an Arc<dyn TokenCredential>, scopes, and ClientOptions.** The 0.21 generated clients do not hide the management plane URL the way some older helpers did.
• **DefaultAzureCredential::create is fallible.** Call it. There is no DefaultAzureCredential::default() in 0.21.
• **Namespace id carries the resource group.** list is subscription-scoped; queue list needs the RG. Parse it from the ARM id rather than guessing.
• **page.value is a Vec, not an Option.** Empty page means empty vec. Stream with into_stream() and page until the continuation ends.

What was checked, and what was not. A scratch crate built from the frontmatter manifest and body passed cargo check on the lab Mac stable cargo 1.96.0, resolving azure_mgmt_servicebus 0.21.0, azure_identity 0.21.0, azure_core 0.21, tokio 1.53.1. The lab Mac has no Azure credentials (AZURE_CLIENT_ID unset, no ~/.azure, az account absent). Live ARM listing was not run. A namespace with one hot DLQ and one session queue would read like this (illustrative):

rg-orders/sb-prod/checkout	active=12	lock=PT1M	dlq=47
rg-orders/sb-prod/fulfillment	active=0	lock=PT2M	sessions
rg-billing/sb-batch/nightly	active=3	lock=PT1M	max_delivery=5,expire_dlq=off
namespaces=2 queues=5 flagged=3 dlq_hot=1 sessions=1 expire_dlq_off=1

§V. Wrap it with Nix

servicebus-queue-census.nix:

{ pkgs ? import <nixpkgs> { } }:
pkgs.writers.writeBashBin "servicebus-queue-census" ''
  exec cargo +nightly -Zscript ${./servicebus-queue-census.rs} "$@"
''

The wrapper pins the script path into the store. The toolchain still comes from rustup on the host (cargo +nightly).

§VI. What not to do

1. Treating an empty active queue as "no backlog" without reading deadLetterMessageCount.
2. Raising maxDeliveryCount to silence DLQ growth instead of fixing the consumer.
3. Pointing a non-session receiver at a requiresSession queue and calling the API broken.
4. Expecting ARM to report peek-lock vs receive-and-delete. That choice lives in the client.
5. Granting the census Contributor because Reader "might not list queues."

§VII. Close instruction

Run the census against a subscription you own once credentials exist. For every dlq= row, open the dead-letter sub-queue and record the reason code. For every expire_dlq=off row with a short TTL, either turn expiration dead-lettering on or document that silent drop is intentional. For every sessions row, name the process that owns the session lock.

Related

• Dev: trait objects for queue delivery policies (same trio)
• Cert: AZ-900 three messaging shapes (same trio)
• Prior Cloud Ops: Pub/Sub deletion clock census
• Prior Cloud Ops: SQS DLQ and redrive census
• — ch11 acknowledgements and redelivery