Mental model#
Metrics compress repeated observations into numeric time series. They are excellent for rates, totals, gauges, and distributions, but every label combination creates another series and therefore another unit of memory, storage, and query work.
Mechanism and state flow#
- Choose a user or resource question and the counter, gauge, or histogram that answers it.
- Define units, aggregation, label vocabulary, and reset behavior before emitting data.
- Keep labels bounded; move request IDs, user IDs, and raw error text to traces or logs.
- Aggregate over windows and preserve distributions for latency rather than averaging away the tail.
- Use exemplars or trace links to move from a symptom to a representative request.
Concrete example#
Measure API latency without a series per user
Record a duration histogram labeled by stable route template, method, region, and outcome class. Do not label it with user ID or raw URL. A trace exemplar can retain request-level context.
Architecture consequence: Bounded labels keep the metric operable while distributions preserve tail behavior needed for an SLI.
Scaling and operational trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| More labels | A bounded dimension changes diagnosis or ownership. | Cardinality multiplies ingestion, storage, and query cost. |
| Finer buckets | Decision thresholds need more precision. | More series and still only approximate the distribution. |
| Pre-aggregation | High-volume dashboards need predictable queries. | Lost dimensions cannot be recovered later. |
Failure diagnosis#
- Failure
- A customer ID is added as a metric label.
- Impact
- Active series count grows with customers, destabilizing ingestion and making queries expensive.
- Detection
- Series-creation rate and cardinality by label rise sharply after deploy.
- Mitigation
- Drop the unbounded label, aggregate by a bounded tenant tier, and use a trace or controlled log for individual investigation.
- Trade-off
- Operators lose direct per-customer metric filtering.
Checklist exercise#
Work the scenario before revealing your answer
- Define one latency histogram and one error counter for an API.
- List every proposed label and prove its value set is bounded.
- Specify which user-visible question each metric answers and the trace/log pivot for deeper diagnosis.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Why is average latency insufficient?
- Which labels can grow without a fixed bound?
- How does metric aggregation affect multi-region diagnosis?
- What will you stop collecting if telemetry cost doubles?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- SignalsOpenTelemetry
- Monitoring distributed systemsGoogle SRE
Sources reviewed 2026-09-04. Product-specific controls and threat assumptions still require current verification.