# Exercise 05 — Capstone: the order investigation

**VERIFIED** 2026-11-21 — every command below was run; the app log, DLT
consumption, and group describes are recorded in `../VERIFICATION.md` §5
(raw log: `kafka-lab/docs/runs/2026-11-21-retryable-topic-dlt-app.log`).

**Time:** ~60 min · **Prerequisites:** all four exercises; the vocabulary is
assumed, not re-taught.

## The situation

You are handed a Spring order pipeline and an incident report:

> *"Two `boom` records crashed order processing this morning. Some orders
> look delayed, some look duplicated(?), the dead-letter topic has
> something in it, and analytics claims it saw everything fine. Explain
> exactly what happened — with evidence, not opinions — and tell us what to
> change."*

Nothing is broken for you to fix. Your job is the *investigation*: run the
pipeline, gather the evidence yourself, and write the incident explanation.
The solution file is a model report — compare it to yours after.

## Your mission

Produce a short written report answering, each **citing a command output or
log line you gathered**:

1. **Timeline** — what happened, in order, with timestamps.
2. **The delay** — which healthy records were delayed, by how much, and
   the *mechanism* (name it).
3. **The duplicates question** — could any consumer have processed any
   record twice? What in the configuration/pipeline makes that possible or
   impossible, and what one-word property must every handler have?
4. **Analytics' claim** — "we saw everything": is it true? Cite the group
   offsets.
5. **The morgue** — what exactly is in each DLT, and what do the headers
   tell an on-call engineer?
6. **The recommendation** — one change you would make and why; one change
   you would *not* make (and the trap it avoids).

## The experiment (working directory: `kafka-lab`)

**1. Gather the before-state** (fresh cluster: every group below shows no
committed offsets or `LAG` equal to the log end):

```bash
scripts/k kafka-topics.sh --list | grep lab.spring
scripts/k kafka-consumer-groups.sh --describe --state --group spring.shipping
scripts/k kafka-consumer-groups.sh --describe --state --group spring.analytics
```

**2. Run the incident** (the same fixed 40-record burst as Exercise 04:
28 orders + 2 poison through the blocking path, 9 records + 1 poison
through the non-blocking path):

```bash
scripts/run-spring.sh
```

Capture the console output — it *is* your primary evidence. Save it:

```bash
scripts/run-spring.sh 2>&1 | tee incident-run.log
```

**3. The groups, after** — the ledger of every consumer in the story:

```bash
scripts/k kafka-consumer-groups.sh --describe --group spring.shipping
scripts/k kafka-consumer-groups.sh --describe --group spring.analytics
scripts/k kafka-consumer-groups.sh --describe --group spring.retries
```

**4. Both morgues, with their incident reports:**

```bash
scripts/k kafka-console-consumer.sh --topic lab.spring.orders.DLT \
  --from-beginning --max-messages 2 \
  --property print.headers=true --property print.key=true

scripts/k kafka-console-consumer.sh --topic lab.spring.retries-dlt \
  --from-beginning --max-messages 1 \
  --property print.headers=true --property print.key=true
```

**5. Cross-check the healthy records** — everything that was *not*
poisoned, from the broker's point of view:

```bash
scripts/k kafka-console-consumer.sh --topic lab.spring.orders \
  --from-beginning --timeout-ms 5000 | grep -c '^order-'
# expected: 28 (order-01 … order-28)
```

## Expected observations (the evidence set you should end up holding)

- From the log: the two blocking `boom`s (3 attempts, 500 ms apart, healthy
  orders queueing behind each), the non-blocking `boom` walking
  `retries-retry-0/-retry-1/-retry-2` to `-dlt` while healthy records
  consumed immediately, and analytics logging every record exactly once.
- Group describes: all groups at `LAG 0` after the burst; the per-partition
  `CURRENT-OFFSET`s show who consumed what.
- `lab.spring.orders.DLT` holds exactly the 2 blocking poison records,
  headers naming the original topic/partition/offset and
  `kafka_dlt-exception-message: … poison payload: boom`.
- `lab.spring.retries-dlt` holds exactly 1 record with
  `kafka_original-topic`, attempt counts, and the exception class, cause,
  and message.
- Step 5 returns **28** — nothing was lost; the morgue only ever held the
  three poisoned records.

## Understanding checks (the report graders)

1. In your timeline, which two mechanisms *paused* work — and which one
   did *not* pause the partition it failed on?
2. "Duplicates": from the config you read in Exercise 04
   (`enable.auto.commit: false`, manual acks, rebalances), argue both sides
   — where could a duplicate arise, and why is the pipeline *designed* to
   accept that (at-least-once) rather than promise exactly-once?
3. Analytics' offsets prove its claim — but do they prove it processed
   *correctly*, or only that it *advanced*? (The distinction is the
   difference between an offset and a business outcome.)
4. Your recommendation must name its trade. "Just add retries" and "just
   drop poison records" are both wrong for reasons this course has already
   shown you — cite the exercises.

## Hints

- Build the timeline from the log's timestamps first, then *attach* each
  command output to a line — evidence hangs on the timeline, not the other
  way round.
- For the delay question, the log prints the topic each failure happened
  *on*; the pause only ever affects one demo.
- For the duplicates question, remember who commits offsets (Exercise 03)
  and what `ack-mode: manual_immediate` means for the window between
  "processed" and "recorded as processed".
- A recommendation that costs nothing is usually wrong: name what your
  change trades away (latency? complexity? ordering? human time?).

## Troubleshooting

Same as Exercise 04 (`../TROUBLESHOOTING.md` §1, §4, §6), plus:

- A previous run's records make the DLT show *more* than expected — the
  topics retain history (Exercise 01!); match your DLT counts to your
  run's timestamps, or use `--max-messages` to bound the read.

## Verification

Your report is complete when every one of its six answers points at a
specific log line or command output from **your** run, and the counts
reconcile: 28 healthy orders consumed, 2 poison in the blocking DLT, 1 in
the non-blocking DLT, 0 lag across all groups.

## Solution (open after your attempt)

→ [`../solutions/05-capstone-order-investigation.md`](../solutions/05-capstone-order-investigation.md)
