Start with the decision#
Keep configuration, placement, lifecycle, and policy decisions off the latency-critical serving path; data-plane nodes should operate from validated cached state during bounded control-plane disruption.
Mechanism and state flow#
- Clients mutate desired state through an authenticated control API.
- The control plane validates, versions, audits, and distributes configuration.
- Data-plane nodes acknowledge applied versions and serve traffic locally.
- Health signals feed reconciliation without requiring the controller on every request.
- Safe defaults, expiry, and rollback bound stale configuration.
Concrete example#
Feature flag service
The control plane stores targeting rules and audits changes. Edge SDK relays cache a signed version and evaluate locally. A control outage blocks new configuration but does not add a dependency to every application request.
Architecture consequence: Serving availability and latency are decoupled from configuration churn, with explicit staleness and emergency-disable rules.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Local evaluation | Serving latency and control-plane isolation dominate. | Stale versions and distribution complexity. |
| Remote evaluation | Central policy must be immediately authoritative. | Control availability and network latency enter the hot path. |
| Hybrid | Most rules can be cached but sensitive decisions need a server. | Two paths and more consistency reasoning. |
Failure diagnosis#
- Failure
- A bad configuration version reaches every data node.
- Impact
- The independently available data plane consistently serves the wrong behavior.
- Detection
- Applied-version telemetry, canary outcome metrics, config validation, and audit correlation.
- Mitigation
- Stage distribution, keep last-known-good state, sign versions, provide rapid rollback, and constrain blast radius.
- Trade-off
- Progressive rollout slows global convergence but limits failures.
Design checkpoint#
Work the scenario before revealing the checklist
- Label each API as control or data plane.
- Choose a configuration version and acknowledgement model.
- Define behavior during control-plane loss and after cache expiry.
- Design canary, rollback, and audit paths.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Can the data plane start if the controller is down?
- How stale may configuration become?
- Which emergency control must bypass normal rollout?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Kubernetes cluster architectureKubernetes
- Envoy architecture overviewEnvoy
Sources reviewed 2026-09-04. Validate product limits, legal obligations, and operational assumptions for the system you are designing.