Kafka transports history; Flink computes over it#
A simple consumer can parse events and write results. Flink becomes compelling when computation holds state across events, reasons about when events occurred, evaluates windows, and restores distributed processing state after failure. Kafka is a common source, not a mandatory partner.
Sources, stateful operators, sinks, and checkpoints#
Keyed state places all events for one logical key on the appropriate operator instance. Checkpoints capture sufficient operator state and source positions to resume after failure. End-to-end guarantees still depend on the source, sink connector, and side-effect protocol.
Event time corrects arrival-time distortion#
A delayed purchase
event occurred: 12:00:01
event arrived: 12:00:20
Processing-time logic may place it in the wrong window.
Event-time logic uses the occurrence timestamp and lateness policy.A watermark estimates how far event time has progressed so operators can produce results despite out-of-order arrivals. It does not prove that no earlier event will arrive. Define allowed lateness and what happens to late events.
Windows bound continuous computation#
| Window | Mental model | Example |
|---|---|---|
| Tumbling | Fixed, non-overlapping buckets | Purchases per store every five minutes |
| Sliding | Overlapping windows evaluated repeatedly | Error rate over the last ten minutes each minute |
| Session | Activity grouped until an inactivity gap | User browsing session |
Checkpoints restore computation, not arbitrary external reality#
On failure, Flink can restore operator state and source positions from a consistent checkpoint and replay input. A sink must participate with appropriate transactional or idempotent semantics for an end-to-end claim. A call to an external payment API remains an application correctness boundary.
Flink vs a simple consumer#
| Simple consumer may be enough | Flink becomes more compelling |
|---|---|
| Stateless mapping or validation | Large keyed state across events |
| Straightforward idempotent database writes | Event-time windows and late-data policy |
| Small volume and simple recovery | Distributed checkpointed computation |
| Ordinary request/background logic | Continuous fraud, metrics, ETL, enrichment, sessionization |
- Adding Flink because Kafka is present.
- Using processing time when product semantics require event time.
- Claiming a watermark proves completeness.
- Claiming checkpoints make arbitrary external effects exactly once.
- Choosing Flink for a stateless low-volume consumer.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- What state must survive across events?
- Which timestamp defines the product result?
- How late can data arrive and what happens then?
- How are checkpointed source positions coordinated with the sink?
- Why is a normal consumer insufficient?
Technology selection reference#
| Need | Technology worth considering |
|---|---|
| Shared in-memory cache or typed transient state | Redis |
| Retained event stream and replay | Kafka |
| Relational transactions | PostgreSQL |
| Managed key-oriented distributed store | DynamoDB |
| Distributed full-text search | Elasticsearch / OpenSearch |
| Large object bytes | S3 / object storage |
| Distributed wide-column data | Cassandra |
| Flexible broker routing | RabbitMQ |
| Managed AWS work queue | SQS |
| Distributed coordination metadata | ZooKeeper / etcd |
| Stateful event-time stream computation | Flink |
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Stateful stream processingApache Flink Documentation
- Streaming analytics: event time and watermarksApache Flink Documentation