# Solution 04 — Retries and the dead-letter topic

*Attempt the exercise first — this file is the post-mortem.*

## What the verified run showed

Timeline from the recorded run (times from the app log, seconds after the
burst was sent; the same evidence is summarized in
`kafka-lab/docs/runs/2026-11-21-retryable-topic-dlt.md`):

**Blocking demo (`lab.spring.orders` → `.DLT`) — the partition pins:**

```
10:42:20.445  shipping: cannot process [boom] — throwing      (attempt 1)
10:42:20.964  shipping: cannot process [boom] — throwing      (attempt 2, +0.5s)
10:42:21.479  shipping: cannot process [boom] — throwing      (attempt 3, +0.5s)
10:42:21.987  DEAD LETTER: [boom] — quarantine it, alert someone
10:42:21.994  shipping: order-17 … — processed, acking         ← the queue drains NOW
```

Between the first `throwing` and the `DEAD LETTER`, the healthy orders on
that partition waited ~1.5 s — per poison record. That is the
head-of-line block, measured.

**Non-blocking demo (`lab.spring.retries` → `-retry-0/1/2` → `-dlt`) — the
partition never waits:**

```
10:42:18.941  retries: [boom] failed on lab.spring.retries — republished…
10:42:20.129  retries: [boom] failed on lab.spring.retries-retry-0
10:42:20.136  retries: [retry-order-04 …] processed on lab.spring.retries — acking
   … (all 9 healthy records consumed while the poison is mid-walk)
10:42:22.141  retries: [boom] failed on lab.spring.retries-retry-1   (+2s)
10:42:26.150  retries: [boom] failed on lab.spring.retries-retry-2   (+4s)
10:42:26.311  DEAD LETTER (non-blocking): [boom] from lab.spring.retries-dlt
```

Group ledger (the one-command proof):

```
spring.retries  lab.spring.retries  (all partitions)  LAG 0
```

DLT headers (the incident report):

```
kafka_original-topic:lab.spring.retries
kafka_dlt-original-consumer-group:spring.retries
retry_topic-attempts:…
kafka_exception-message:Listener failed; transient failure: boom
key: customer-4   value: boom
```

## The understanding checks, answered

1. **Who stalls, quantified.** Blocking: healthy orders behind a poison
   record waited ~1.5 s each (three attempts, 500 ms apart —
   `FixedBackOff(500L, 2)` in `OrderListener.java`). Non-blocking: the
   healthy records were consumed ~1.2 s after the poison *first* failed,
   while it was still on `retry-0` — the partition's wait was **zero**.
   Same failure, ~1.5 s vs ~0 s of collateral delay.
2. `.DLT` (uppercase) is the topic *we* created in
   `lab.AdminLesson` for the blocking demo's recoverer, which appends
   `".DLT"` by code. `-dlt` (lowercase) is spring-kafka's **default suffix**
   for `@RetryableTopic`'s auto-created DLT — the framework created the
   whole retry ladder itself (`autoCreateTopics` is on by default). Who
   created the topic decides its name.
3. Analytics is its own group with its own container and its own (default,
   lenient) error handling; shipping's error handler never touches it, and
   its offsets advance independently. Error handling is per-container —
   Exercise 03's group isolation, applied to failures.
4. The SLO in code: *"a dependency outage is retried for ~7 seconds
   (1s + 2s + 4s); anything still failing after that is escalated to a
   human-readable queue (the DLT)."* Backoff + attempts is a recovery-time
   budget, chosen deliberately.
5. With original topic/partition/offset in the headers, the operational
   move after fixing the producer is **replay**: re-produce the DLT
   record's bytes to its original topic and let the (idempotent) consumer
   process it again. The DLT is a work queue, not a trash can.

## The design trade, in one table

| | Blocking (`DefaultErrorHandler`) | Non-blocking (`@RetryableTopic`) |
|---|---|---|
| Healthy records behind a failure | wait out the backoff | keep flowing |
| Retry state lives in | consumer memory (in-place) | **Kafka topics** (visible, replayable, monitorable) |
| Cost | zero new topics | retry + DLT topics to own and monitor |
| Use when | ordering within the partition is sacred; failures are rare | the partition must stay live; failures are expected |

## What this buys you in production

When someone proposes "just retry in the listener," you now have measured
evidence for the conversation: the retry itself is fine — *where the
waiting happens* is the architecture. Side-topic retries make the waiting
inspectable (`kafka-consumer-groups.sh` on a retry topic is a lag chart of
your dependency's health), and the DLT header set is the incident report
format your on-call can standardize on.
