Mental model#
A trace represents one operation as a tree or graph of spans across process boundaries. Trace context supplies correlation; span timing and attributes explain where time and errors accumulated. A trace is evidence about a request, not a complete service-health measure.
Mechanism and state flow#
- Create or accept trace context at the trusted entry boundary.
- Start spans around meaningful client, server, queue, and storage operations.
- Propagate context explicitly across supported carriers and validate untrusted baggage.
- Record status and bounded attributes; avoid payloads, credentials, and uncontrolled identifiers.
- Sample with a stated policy and correlate selected spans to metrics and logs.
Concrete example#
Find the slow hop
A checkout trace spends 40 ms in the API, 60 ms in the order service, and 900 ms waiting on payment. The trace directs investigation to the payment dependency; service metrics determine whether the request is representative.
Architecture consequence: Propagate context across every remote boundary and pair traces with aggregate service signals before changing capacity.
Scaling and operational trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Head sampling | A cheap decision can be made when the request starts. | The sampler cannot know the eventual latency or error. |
| Tail sampling | Errors and slow traces must be preferentially retained. | Collectors buffer trace state and require more capacity. |
| Full tracing | Traffic is low and diagnostic value exceeds cost. | Storage, network, and sensitive attribute exposure grow with volume. |
Failure diagnosis#
- Failure
- A queue consumer starts a new unrelated trace.
- Impact
- The request-to-async-work path is broken and latency attribution stops at the broker.
- Detection
- Traces end at publish spans while consumers show orphan roots for the same operation.
- Mitigation
- Propagate or link the supported context in message metadata and test the producer/consumer boundary.
- Trade-off
- Context increases message metadata and long delays may make parent-child semantics misleading, so span links can be preferable.
Checklist exercise#
Work the scenario before revealing your answer
- Draw spans for one synchronous call and one queued continuation.
- Mark where context enters, propagates, is validated, and may be sampled.
- Choose five safe span attributes and one attribute that must remain out of telemetry.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Where can trace context be lost?
- How do retries appear in the trace?
- When would you use a link instead of a child span?
- How does sampling bias an incident investigation?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- SignalsOpenTelemetry
- Context propagationOpenTelemetry
Sources reviewed 2026-09-04. Product-specific controls and threat assumptions still require current verification.