# Exercise 04 — Retries and the dead-letter topic: two ways to fail

**VERIFIED** 2026-11-21 — the Spring app run and every observation below
come from a recorded execution (see `../VERIFICATION.md` §4, and the raw log
kept in `kafka-lab/docs/runs/2026-11-21-retryable-topic-dlt-app.log`).

**Time:** ~60 min · **Prerequisites:** Exercises 01–03; JDK 25 + cluster up.

## The situation

Your order listener calls a flaky dependency. The naive fix — catch and
retry in place — works, but while record 401 waits out its backoff, records
402–480 sit **behind it on the same partition**. One bad dependency slows
every order. There are two standard fixes in Spring Kafka, and this lab has
**both, side by side, in one app**:

- **Blocking retries** (`DefaultErrorHandler` + `DeadLetterPublishingRecoverer`):
  retry in place, then dead-letter — `OrderListener.java`.
- **Non-blocking retries** (`@RetryableTopic`): republish the failure to a
  side topic, let the partition keep flowing — `RetryableOrderListener.java`.

You will run both and time them against each other.

## Predict first

1. Blocking demo: a poison record retries 3 times, 500 ms apart. During
   those ~1.5 s, what happens to the healthy records on the same partition?
2. Non-blocking demo: the poison record is republished to `…-retry-0`. While
   it waits there, what is the main topic's consumer lag doing?
3. Both demos: the analytics consumer (a second group on the same topic)
   sees each poison record how many times?
4. A retry re-delivers a record. What property must the handler have for
   that to be safe (Exercise 03's last answer)?

## The experiment (working directory: `kafka-lab`)

**0. Read the reference before running it** — five minutes of reading saves
an hour of guessing:

```bash
less spring-lab/src/main/java/lab/spring/OrderListener.java          # blocking
less spring-lab/src/main/java/lab/spring/RetryableOrderListener.java # non-blocking
less spring-lab/src/main/java/lab/spring/OrderGenerator.java          # what gets sent
```

Note: attempts=4 counts the *original* delivery (1 + 3 retries), backoff
1s → 2s → 4s — waited on the **side topics**.

**1. Run the app** (starts on a free port, prints its URL; stop with
`Ctrl-C` when the logs go quiet — the producer sends a fixed burst of 40
records total and then idles):

```bash
scripts/run-spring.sh
```

Watch the interleaving timestamps as both demos play out (~10 seconds).
Key lines to watch for:

```
shipping: cannot process [boom] — throwing          ← blocking attempt
shipping: order-08 … — processed, acking             ← healthy records resume AFTER the boom
retries: [boom] failed on lab.spring.retries — republished to a side topic…
retries: [retry-order-04 …] processed on lab.spring.retries — acking   ← partition keeps flowing
retries: [boom] failed on lab.spring.retries-retry-0 …
…-retry-1 … -retry-2 …
DEAD LETTER (non-blocking): [boom] from lab.spring.retries-dlt
```

**2. List the topics the two strategies created** — the shape is the
lesson:

```bash
scripts/k kafka-topics.sh --list | grep lab.spring
# lab.spring.orders, lab.spring.orders.DLT            ← blocking demo
# lab.spring.retries, lab.spring.retries-retry-0/-retry-1/-retry-2,
# lab.spring.retries-dlt                             ← non-blocking demo
```

**3. The lag evidence** — the non-blocking demo's whole claim in one
command:

```bash
scripts/k kafka-consumer-groups.sh --describe --group spring.retries
```

**4. The morgue, with its incident report.** The DLT record keeps the
original topic/partition/offset and the exception in its **headers**:

```bash
scripts/k kafka-console-consumer.sh --topic lab.spring.retries-dlt \
  --from-beginning --max-messages 1 \
  --property print.headers=true --property print.key=true
```

## Expected observations

- Blocking demo: the two `[boom]` records each print
  `cannot process … throwing` **three times** (500 ms apart), *then*
  `DEAD LETTER: [boom]`. The healthy orders on the same partition queue
  behind the retry: watch `shipping: order-…` lines pause during the
  retries and resume after. That pause is the head-of-line block.
- Non-blocking demo: `[boom]` fails on the main topic and **the healthy
  records behind it are consumed immediately** (verified run: all consumed
  while the poison was still walking the side topics). The record surfaces
  on `lab.spring.retries-retry-0`, then `-retry-1` (2 s later), then
  `-retry-2` (4 s later), then `lab.spring.retries-dlt`.
- Step 3: `LAG 0` on every `lab.spring.retries` partition for group
  `spring.retries` — the partition never waited for the poison.
- Step 4: headers include `kafka_original-topic`, the consumer group, the
  attempt count, and `kafka_exception-message` naming the failure — an
  incident report a human can read.
- One framework ERROR with a stacktrace appears at the DLT handoff
  (logged by spring-kafka's recoverer). Expected narration, not a failure.

## Understanding checks

1. From your own log lines: which record was consumed *later*, a healthy
   order behind the blocking boom, or the same-position record in the
   non-blocking demo? Quantify with timestamps.
2. Why does the blocking demo use `.DLT` (uppercase) and the non-blocking
   one `-dlt`? (Hint: who created each topic?)
3. The analytics group never throws. Why did it never stall during either
   demo — whose error handling applies to a group?
4. `attempts = 4` with backoff 1s/2s/4s: write the SLO this encodes in one
   sentence ("a dependency outage is retried for ~7 s; beyond that, a
   human").
5. DLT headers name the original topic and offset. What operational move
   does that enable once the producer bug is fixed?

## Hints

- For check 1: grep your console output — the app prints the topic the
  record failed *on*; line up the two demos' clocks.
- For check 3: error handling is per-container, and containers are
  per-group. Revisit Exercise 03's `--members` output.
- Before looking at the solution for check 5, ask: if you re-produce a DLT
  record's bytes to its original topic, what must be true of the consumer?

## Troubleshooting

- App fails to start / producers time out — `../TROUBLESHOOTING.md` §1
  (broker address) and §4 (JDK).
- No `[boom]` lines — the app is quiet because a previous run's offsets are
  committed; a fresh group id or the reset warning in `SETUP.md` replays
  it. Prefer: just re-run `scripts/run-spring.sh` — the generator produces
  a fresh burst each start.
- `Ctrl-C` leaves the port bound — `../TROUBLESHOOTING.md` §6.

## Verification

You are done when your own run shows all three:

1. The blocking demo's three `cannot process [boom]` attempts with the
   healthy-order pause between the first attempt and the DLT.
2. `LAG 0` for `spring.retries` while the poison was still in flight (or
   the log interleaving that proves the same thing).
3. The `lab.spring.retries-dlt` record with `kafka_original-topic` and the
   exception message in its headers.

## Solution (open after your attempt)

→ [`../solutions/04-retries-and-dead-letter-topic.md`](../solutions/04-retries-and-dead-letter-topic.md)
