Mental model#
Observability is the ability to ask useful questions about a running system from its outputs. Start with user journeys and decisions: a signal earns its ingestion, retention, and on-call cost only when it helps detect, explain, or prevent a meaningful failure.
Mechanism and state flow#
- Name the user outcome, owner, dependency boundary, and acceptable failure.
- Emit structured logs, bounded-cardinality metrics, and trace context at that boundary.
- Route signals through a controlled collection pipeline with redaction, sampling, retention, and access policy.
- Connect symptom alerts to dashboards, traces, logs, and a runbook owned by a team.
- Review unused telemetry and expensive dimensions; reduce cost without removing the signals needed to operate the service.
Concrete example#
Checkout latency rises only for one payment path
A latency SLI detects the user-visible symptom. A trace identifies the slow payment dependency, a metric shows connection-pool saturation, and correlated logs provide the bounded error detail. No single signal answers every question.
Architecture consequence: Preserve correlation across the checkout boundary and budget telemetry volume by diagnostic value, not by collecting every field forever.
Scaling and operational trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Broad low-cost metrics | You need continuous symptom detection and trends. | Labels must remain bounded and aggregate detail can hide individual requests. |
| Sampled traces | Cross-service latency and causality matter. | Sampling can miss rare cases and full fidelity is expensive. |
| Targeted structured logs | Operators need event detail and audit evidence. | Sensitive data, indexing, and retention require explicit controls. |
Failure diagnosis#
- Failure
- The telemetry pipeline becomes slow during an outage.
- Impact
- The team loses diagnostic evidence when it is most valuable, while synchronous exporters may worsen the service.
- Detection
- Exporter queue age, dropped-signal counters, and collector health diverge from application traffic.
- Mitigation
- Keep export off the request critical path, bound buffers, shed lower-value telemetry, and retain local health signals.
- Trade-off
- Bounded pipelines deliberately lose some diagnostic detail under pressure.
Checklist exercise#
Work the scenario before revealing your answer
- Choose one user journey and write its latency, availability, and correctness questions.
- Assign the cheapest signal that can answer each question.
- Mark every high-cardinality or sensitive attribute and decide whether to drop, hash, aggregate, or restrict it.
- Name the owning team, retention period, runbook, and monthly cost review.
Level calibration#
| Level | Useful evidence |
|---|---|
| SDE I–II | Instrument one request path and distinguish logs, metrics, and traces. |
| Senior | Design symptom-led alerts, correlation, failure-safe export, and retention boundaries. |
| Staff | Set cross-team semantic conventions, ownership, privacy controls, and a sustainable telemetry cost model. |
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Which signal wakes a person, and why?
- What happens when the collector is unavailable?
- How do you stop tenant IDs or request IDs from exploding metric cardinality?
- Who owns the dashboard, runbook, retention, and deletion policy?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- OpenTelemetry specification overviewOpenTelemetry
- SignalsOpenTelemetry
- Monitoring distributed systemsGoogle SRE
Sources reviewed 2026-09-04. Product-specific controls and threat assumptions still require current verification.