# VERIFICATION — what we ran, what we saw, when

This is the evidence ledger behind every **VERIFIED** stamp in the
exercises. Nothing in this course is marked verified without a line here.

**Verification environment (all runs):**

| Piece | Version / value |
|---|---|
| Date | 2026-11-21 |
| Kafka | 3 × `apache/kafka:3.9.0` (KRaft, `kafka-lab/docker/docker-compose.yml`) |
| CLI | Kafka 3.9.0 tools, inside the kafka-1 container (`scripts/k`) |
| Java | OpenJDK 25.0.1, Maven 3.9.9 via `./mvnw` |
| spring-lab | Spring Boot 3.5.7, spring-kafka 3.3.10 |
| OS | macOS workstation; Docker on a Linux host |

The raw console log of the Spring app run is kept with the lab repo at
`kafka-lab/docs/runs/2026-11-21-retryable-topic-dlt-app.log`; the excerpts
below are copied from that session's captures so this pack stands alone.

## §1 — Exercise 01 (first message) — VERIFIED

Commands as printed in the exercise, against a fresh `course.notes` topic.

Observed:

```
read 1 (fresh group, latest)         → (no output)
read 2 (same group, --from-beginning) → (no output)
read 3 (fresh group, --from-beginning) → hello / world / kafka

GROUP          TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
course.readers course.notes 0          3               3               0
```

Reads 1 and 2 printing nothing is the *load-bearing observation*: the
first read's group committed an offset at exit, and committed offsets beat
`--from-beginning`.

## §2 — Exercise 02 (keys and partitions) — VERIFIED

Observed placement (`--property print.partition=true --property
print.key=true`):

```
Partition:0 : customer-alice : order-1
Partition:0 : customer-alice : order-3
Partition:0 : customer-carol : order-4     ← collision with alice
Partition:0 : customer-alice : order-5
Partition:2 : customer-bob   : order-2
```

Unkeyed burst of 12 (`course.groups`): **all 12 on Partition:2** — the
Kafka 3.3+ sticky partitioner for unkeyed batches (KIP-794), confirmed on
3.9.0.

Java path — `scripts/run.sh lab.ProducerLesson` completed all sections;
section 02's predicted (client-side `murmur2(key) % 6`) and observed
placements matched record-for-record (`amy → partition 0 @ offset 147`,
`bill → partition 3 @ offset 353`, colliding `dmitri → partition 3`).

## §3 — Exercise 03 (consumer groups) — VERIFIED

Two console consumers in group `course.workers` (client ids worker-a/-b),
12 records on the topic:

```
--state:  Stable   #MEMBERS 2          (range strategy)
--members: worker-a 3 partitions · worker-b 3 partitions
after killing worker-b: Stable #MEMBERS 1
plain describe: survivor …worker-a… holds all 6 partitions, LAG 0 each
record counts: worker-a printed 12, worker-b printed 0
```

The 12/0 split is real evidence, not a failure: all records sat on the
sticky Partition:2, which range assignment gave to A (fair assignment,
skewed data — the exercise turns this into a lesson).

Java path — `scripts/run.sh lab.ConsumerLesson` completed, including
section 03's live rebalance (three scripted consumers, one closes,
survivors print `revoked→assigned` and pick up the leaver's partitions and
committed offsets).

## §4 — Exercise 04 (retries + DLT) — VERIFIED

`scripts/run-spring.sh` — full recorded run (raw log path above).

Blocking demo (`OrderListener`, `lab.spring.orders` → `.DLT`):

```
10:42:20.445  shipping: cannot process [boom] — throwing      (attempt 1)
10:42:20.964  … (attempt 2)   10:42:21.479  … (attempt 3)     ← 500 ms apart
10:42:21.987  DEAD LETTER: [boom]
```

Healthy orders on the same partition resumed at `10:42:21.994` — the
~1.5 s head-of-line pause, measured.

Non-blocking demo (`RetryableOrderListener`, `lab.spring.retries` →
`-retry-0/-retry-1/-retry-2` → `-dlt`):

```
10:42:18.941  [boom] failed on lab.spring.retries — republished
10:42:20.129  [boom] failed on lab.spring.retries-retry-0
10:42:20.136… all 9 healthy records consumed on the main topic
10:42:22.141  -retry-1      10:42:26.150  -retry-2
10:42:26.311  DEAD LETTER (non-blocking): [boom] from lab.spring.retries-dlt
```

Group ledger after: `spring.retries` at **LAG 0** on every partition while
the poison was mid-walk. Topics listed: `lab.spring.retries`,
`-retry-0/-retry-1/-retry-2`, `-dlt` (auto-created by spring-kafka).

DLT record headers (consumed with `--property print.headers=true`):

```
kafka_original-topic:lab.spring.retries
kafka_dlt-original-consumer-group:spring.retries
retry_topic-attempts:…   kafka_exception-cause-fqcn:java.lang.IllegalArgumentException
kafka_exception-message:Listener failed; transient failure: boom   key: customer-4
```

One expected framework ERROR with stacktrace at the DLT handoff
(DeadLetterPublishingRecovererFactory) — narration, not a failure.

## §5 — Exercise 05 (capstone) — VERIFIED

Same recorded run plus the after-state commands:

```
lab.spring.orders.DLT   → 2 records (the two blocking booms), headers:
  kafka_dlt-original-topic:lab.spring.orders,
  kafka_dlt-exception-message: … poison payload: boom
lab.spring.retries-dlt  → 1 record (non-blocking boom, headers as §4)
healthy orders consumed  → 28 of 28 (grep -c '^order-' on lab.spring.orders)
spring.shipping / spring.analytics / spring.retries → LAG 0 after the burst
```

All six report questions in the solution are answerable from these
captures plus the raw log's timeline.

## What was NOT verified (and why)

- **Windows / WSL2 path** — the lab's scripts are bash; untested on
  Windows. PREREQUISITES.md says so.
- **The optional standalone single-broker container** (one `docker run
  apache/kafka:3.9.0` without the lab repo): NOT exercised in our
  environment — port 9092 was occupied by the lab cluster's broker at
  verification time. Every course exercise runs on the lab's 3-broker
  compose, which **is** verified end to end; if you run the standalone
  variant, expect identical CLI behavior minus the cluster features
  (Exercise 02's RF-3 topics need ≥3 brokers — on a single broker use
  `--replication-factor 1`).
- **Long-running stability / load** — deliberately out of scope; every
  exercise is bounded to ≤ 100 records.

## Re-verification recipe (for anyone)

Setup → exercises in order → each exercise's Verification section names
the exact command whose output proves it. A fresh cluster (SETUP.md's
reset warning) makes every group behave as brand-new.
