Why Kafka instead of a normal work queue?#
Kafka retains an ordered sequence of records per partition. Consumers track their own offsets; one group advancing does not globally delete a record for every other group. That model fits event history, replay, CDC distribution, and multiple derived consumers better than a queue whose primary abstraction is pending work.
Core architecture#
A cluster contains brokers. Topics split into partitions. Each partition has an ordered log, leader, replicas, and offsets. Producers choose partitions directly or through keys; consumers in a group divide assigned partitions.
Keys determine ordering and parallelism#
partition_key = customer_id can keep one customer's events in one partition, preserving partition-scoped order. It can also make a celebrity customer hot. With eight partitions, one consumer group normally cannot use more than eight active consumers for that topic's partition work at once, subject to assignment and topology.
Four partitions constrain group parallelism
Add or remove group members to see partition ownership rebalance.
| Partition | Owner |
|---|---|
| P0 | Consumer A |
| P1 | Consumer B |
| P2 | Consumer A |
| P3 | Consumer B |
Cross-link partition keys, consumer groups, and ordering; Kafka implements those concepts rather than replacing them.
Offsets, retention, replay, and compaction#
Database → CDC → Kafka
├→ Search indexer
├→ Analytics
└→ Warehouse loaderEach consumer can resume or replay within retained history. Retention can be policy-driven by time or size. Log compaction is a distinct keyed policy that preserves at least the latest value for keys according to Kafka's compaction semantics; it does not mean “keep every historical record forever.”
Replication and exactly-once nuance#
Each partition's leader replicates to followers. Producer acknowledgements, replication factor, in-sync replica state, and broker failure determine durability; avoid an absolute “Kafka never loses data.” Idempotent producers and Kafka transactions can coordinate records and offsets for supported Kafka processing boundaries.
Operational interview concerns#
| Concern | Design consequence |
|---|---|
| Partition count | Bounds group parallelism and influences metadata, files, and reassignment work. |
| Consumer lag | Measures distance from current history and recovery time pressure. |
| Hot partition | A skewed key can overload one leader and its consumers. |
| Rebalancing | Ownership moves; consumers must handle pauses and safe offset progress. |
| Large messages | Pressure network, memory, storage, and processing; objects may belong elsewhere. |
| Schema evolution | Independent producers and consumers need compatible contracts. |
Kafka vs queue-oriented systems#
| Requirement | Kafka tendency | Queue tendency |
|---|---|---|
| Retained history and replay | Central capability | Often not central |
| Independent consumer groups | Natural offset ownership | Separate queues/subscriptions may be needed |
| Simple background job dispatch | Often more machinery than required | Natural fit |
| Delayed jobs and straightforward retries | Application patterns required | Often first-class operational model |
- Saying Kafka guarantees exactly once for arbitrary external effects.
- Claiming ordering across an entire multi-partition topic.
- Adding Kafka for one simple background worker.
- Ignoring lag, retention, rebalances, hot keys, and schema changes.
- Calling partitions only a scaling detail when they also define ordering.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Why is replay a requirement?
- Which key defines order and can it become hot?
- How are offsets committed relative to side effects?
- What happens during broker failure or group rebalancing?
- Could a managed queue satisfy the simpler requirement?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Apache Kafka introductionApache Kafka Documentation
- Kafka design and delivery semanticsApache Kafka Documentation