Tracing: What It Is, Principles, and Data Collection

What Is Tracing in Monitoring

Tracing is a method of distributed observation where each incoming request is tracked through all services and components of the system. Unlike metrics, which show aggregate values (average response time, error count), tracing preserves the full context of a single specific request.

Each processing step — a database call, an HTTP request to another microservice, execution of a background task — is recorded as a separate unit with a timestamp, status, and attributes. According to Google Dapper (original publication in 2010), tracing allows localizing delays in distributed systems down to a single call.

Tracing is especially important for mobile applications where the backend consists of dozens of microservices. A user action — such as logging into an account — can go through an API Gateway, authentication service, database, and Push service. Without tracing, determining which component is slowing down the response is nearly impossible.

Spans and Traces: Basic Data Structure

The basic unit of tracing is a span. Each span represents one logical operation: an HTTP request, SQL query, gRPC call, JSON serialization. A span contains a unique identifier, parent identifier, operation name, start time, duration, status, and a set of attributes.

Span Hierarchy in a Trace

All spans related to a single root request are grouped into a trace. The root span represents the entry point — an HTTP request from a mobile client to the API. Child spans form a tree, where each span references its parent through the parent_span_id field.

The duration of a trace equals the sum of the durations of unique time segments of all spans. If two child spans execute in parallel, their time is not summed — this is critical for correctly analyzing delays caused by parallel microservice calls.

Span Attributes and Events

Each span can contain attributes — key-value pairs with meta-information: request URL, user ID, API version, host name. Attributes are used for filtering and grouping traces. In addition to attributes, spans support events — timestamps with text descriptions, such as "cache miss" or "connection retry".

How Distributed Tracing Works

Distributed tracing solves the problem of linking spans that are created in different processes and on different machines. The mechanism is based on context propagation: when service A calls service B, a header containing the current trace ID and parent span ID is added to the outgoing request.

Standard context propagation protocols include W3C Trace Context (traceparent and tracestate headers) and Zipkin B3 (X-B3-TraceId, X-B3-SpanId headers). W3C Trace Context was adopted as a standard by the W3C consortium in 2021 and is supported by all major telemetry providers.

Upon receiving the request, service B extracts the trace_id from the header and creates a child span with that trace_id. Thus, after the request completes, all spans from different services are combined into a single trace at the collector side. This requires every service to be instrumented with the same tracing library.

Context Propagation in Microservices

In mobile development, context propagation covers not only the backend but also client-server interaction. A mobile application can send a trace_id in the header of each API request, allowing a client action to be linked with server-side processing. The OpenTelemetry SDK for iOS and Android supports automatic creation and propagation of trace context through HTTP clients.

import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.Tracer
import io.opentelemetry.context.Context

class TracingInterceptor : Interceptor {
    private val tracer: Tracer = OpenTelemetry.getTracer("mobile-app")

    override fun intercept(chain: Interceptor.Chain): Response {
        val span = tracer.spanBuilder("HTTP POST /api/login")
            .setParent(Context.current())
            .startSpan()

        return chain.proceed(chain.request())
            .also { span.end() }
    }
}

The presented Kotlin interceptor creates a span for each HTTP request to the server. The parent context is propagated from the calling code through Context.current(), allowing client-side tracing to be linked with server-side tracing.

Implementing Tracing with OpenTelemetry

OpenTelemetry is the de facto standard for collecting trace data. It provides a unified API for generating spans, automatic instrumentation of popular libraries, and a flexible mechanism for exporting data to various backends: Jaeger, Zipkin, Grafana Tempo, Datadog, New Relic.

Automatic Instrumentation

OpenTelemetry supports automatic span creation for popular frameworks: Spring Boot, Ktor, Flask, Express, gRPC. A developer only needs to add a dependency to the project, and the library automatically intercepts incoming and outgoing requests. Auto-instrumentation for Java uses a javaagent that modifies bytecode on the fly without changing source code.

For mobile platforms, OpenTelemetry provides Swift SDK and Kotlin SDK. They automatically create spans for network requests (URLSession, OkHttp), database operations (CoreData, Room), and background tasks. Developers can add custom spans for business logic.

Data Export

Collected spans are sent to a collector via the OTLP (OpenTelemetry Protocol). The collector can buffer, filter, and forward data to one or more storage systems. According to OpenTelemetry documentation, the typical latency from span generation to its display in a dashboard is 2–5 seconds when using gRPC export.

import OpenTelemetryApi
import OpenTelemetrySdk
import URLSessionInstrumentation

let instrumentation = URLSessionInstrumentation()
instrumentation.enable()

let tracer = OpenTelemetry.instance.tracerFactory
    .get("mobile-monitoring")

let span = tracer.spanBuilder("fetch-user-profile")
    .setAttribute(key: "user.id", value: userId)
    .startSpan()
span.end()

The Swift code activates automatic instrumentation of the networking layer and creates a custom span for the user profile retrieval operation. The user.id attribute allows filtering traces by a specific user later.

Trace Sampling Strategies

In high-load systems, tracing every request is impossible — it creates unacceptable load on storage and network. Sampling solves this problem by saving only a subset of traces. The choice of strategy directly affects data completeness and infrastructure cost.

Head-Based Sampling

The decision to save a trace is made at its creation — in the root span. The simplest and most common approach: a fixed percentage of requests (e.g., 5%) is saved, the rest are discarded. The drawback is that rare errors cannot be guaranteed to be captured. The Probability sampler in OpenTelemetry supports probability settings from 0.0 to 1.0.

Tail-Based Sampling

The decision is deferred until all spans of the trace are complete. An analyzer evaluates whether the trace contains errors, exceeded time limits, or interesting attributes, and only then saves it. This approach requires buffering all spans in the collector, which increases memory consumption. According to Grafana Labs, tail-based sampling is 40–60% more efficient in terms of "cost per useful data" in systems with rare but critical errors.

Tracing vs. Logging

Logging records individual events with severity levels (info, warn, error) but does not link them into the context of a single request. Tracing, on the other hand, creates a structured tree of operations belonging to one end-to-end request. In practice, these two approaches are not mutually exclusive but complement each other.

Logs are effective for detailed analysis of a specific error: a developer sees the exact message, stack trace, and variable values. Tracing answers the question "why is the request taking 5 seconds" — it shows which microservice or call took the most time. According to Honeycomb (2024), teams using tracing together with logging find the root cause of incidents 2.3 times faster.

The modern approach — observability — combines tracing, metrics, and logs into a single system. OpenTelemetry supports correlation between these three signals: each span can contain links to related logs, and metrics can be tagged with trace_id to drill down to specific traces.