# Solution 05 — Capstone: the order investigation

*Attempt the report first — this file is the model post-mortem to compare
against, written from the verified run
(`kafka-lab/docs/runs/2026-11-21-retryable-topic-dlt-app.log`).*

## Model incident report — "the morning the orders stalled"

### 1. Timeline (from the app log)

```
18.418  generator: 10 records (1 poisoned) → lab.spring.retries
        generator: 30 orders (2 poisoned) → lab.spring.orders        [same run]
18.770–20.169  ALL 9 healthy retries-records consumed; analytics logs
               everything; shipping consumes/acks its healthy orders
20.445  shipping: [boom] attempt 1 (blocking path)
20.964  attempt 2 · 21.479  attempt 3
21.987  DEAD LETTER: [boom]            ← blocking poison parked (1 of 2)
21.994  shipping resumes: order-17, order-20 …
   …    second boom: same 3-attempt walk, same stall-then-park
22.141  retries: [boom] on -retry-0 · 26.150 on -retry-2
26.311  DEAD LETTER (non-blocking): [boom] from lab.spring.retries-dlt
```

### 2. The delay

Two healthy-order pauses, each ~1.5 s, one per blocking poison record
(three attempts at `FixedBackOff(500, 2)`). The delay mechanism is the
**head-of-line block**: a blocking retry pins the partition while it
waits. The non-blocking path delayed nothing — its healthy records were
consumed 1.2 s after the poison failed, with group `spring.retries` at
`LAG 0` the whole time. *Same app, same morning, two failure
architectures.*

### 3. The duplicates question

Possible? **Yes — by design.** The pipeline is at-least-once: offsets are
committed manually (`ack-mode: manual_immediate`, `enable-auto-commit:
false` — `application.yml`), so any crash between "handler finished" and
"ack committed" re-delivers (Exercise 03's window). Retries re-deliver by
definition. The pipeline *accepts* this rather than paying for
exactly-once machinery because every handler is written to be
**idempotent** — the property that turns at-least-once redelivery into
exactly-once *effect*. What would NOT be safe: a non-idempotent handler
(charge-a-card) behind a retry policy.

### 4. Analytics' claim

True, and provable: `spring.analytics` group describe shows
`CURRENT-OFFSET = LOG-END-OFFSET` on every partition of
`lab.spring.orders` — it advanced past all 30 records, booms included
(it logs and acks everything by design, its own group isolation from
shipping's error handling). **But** the offsets prove *advancement*, not
*correctness* — an offset says "the handler returned normally", not "the
business outcome was right". The claim "we saw everything" is verified;
"we processed everything correctly" is a different claim needing
different evidence (downstream state).

### 5. The morgues

- `lab.spring.orders.DLT`: 2 records (`boom`, keyed) — blocking path,
  headers carry `kafka_dlt-original-topic:lab.spring.orders`, original
  partition/offset, and `kafka_dlt-exception-message: … poison payload:
  boom` with the full stacktrace. On-call can read *what* failed, *where
  it came from*, and *why* — before touching a consumer.
- `lab.spring.retries-dlt`: 1 record — non-blocking path, headers carry
  the original topic, consumer group, per-hop attempt counts, and the
  exception class/cause/message (`transient failure: boom`).
- Cross-check: `kafka-console-consumer … lab.spring.orders | grep -c
  '^order-'` → **28**. Nothing lost; the morgue only ever held the three
  poisoned records.

### 6. Recommendations

**Would make:** treat retry/DLT topics as first-class monitoring — alert
on `spring.retries`-style *retry topic lag*, because it is a direct chart
of dependency health (the run's 7-second walk is visible as lag on
`-retry-0` for exactly that long). Cheap, and it converts the design's
visibility into an operational signal.

**Would NOT make:** "retry harder / more attempts." Exercise 04 measured
what more blocking retries buy: longer partition stalls (every extra
attempt × backoff is healthy-record delay). And dropping poison records
without a DLT is the other wrong answer — it loses the producer-bug
evidence that the headers preserve. The DLT is the *right* amount of
persistence: keep the bytes, keep the provenance, page a human.

### Grading your own report

- Cited **your own** outputs (timestamps/counts), not this file's.
- Named both pause mechanisms and the one non-pausing path.
- Argued duplicates from the commit window, landing on idempotency.
- Distinguished offset-advance from business correctness for analytics.
- One change with its trade named; one rejected change with its trap
  cited.

If your report does all five, you have the skill this course promised:
you investigate before you configure.
