Sampling in Distributed Tracing: Worth the Risk of Losing Detail?

When I tried to track all the steps from order placement to shipment in a production ERP system, I saw how many microservices the system passed through and how much processing each request generated. Especially during a busy Black Friday week, with thousands of transactions per second, I realized it was impossible to store all the details of every single request. This is exactly where the concept of sampling in distributed tracing comes into play. So, is this sampling really worth the risk of losing detail? In my experience, absolutely, but with the right strategies.

I’ve been operating on both the system administration and enterprise software development sides for years. During this time, I’ve experienced firsthand how critical observability tools are. Instead of drowning in logs to detect a problem, seeing which service a request got stuck in with tracing saves an incredible amount of time. However, this convenience comes at a price: data volume. In this post, I will explain why sampling is a necessity, discuss different sampling approaches, the risks of detail loss, and how I manage these risks based on my own experiences.

Why is Sampling Inevitable?

Distributed tracing allows us to visualize how a request progresses through all the services in your system. Each service call, database operation, or external API request is recorded as a “span,” and these spans form a “trace.” While this is not an issue in a small system, it quickly gets out of control in complex and high-volume systems like a manufacturing company’s ERP or a large e-commerce site.

In a customer project, we had a microservice architecture handling an average of 500 transactions per second during a busy period. Considering that each transaction generates an average of 10-15 spans, this translates to 5,000 to 7,500 spans per second. This meant a data flow of approximately 450,000 spans per minute, or around 600 million spans per day. The infrastructure costs (disk, CPU, network bandwidth) required to store and process such a volume of data were astronomical. We were using PostgreSQL on the database side, and indexing and querying so many spans caused significant WAL bloat and replication lag. Disk I/O never dropped below 90%.

Furthermore, collecting and sending each span imposes a performance overhead on the application itself. While network requests, serialization/deserialization operations, CPU, and memory usage are optimized, they still create a non-negligible overhead in total. In a banking internal platform, when we left tracing at 100% enabled, the response time of a critical API increased from an average of 80ms to 120ms. This 50% increase was a situation that directly affected user experience. Therefore, we are left with few options other than completely disabling tracing or implementing sampling.

Basic Sampling Approaches and My Preferences

The fundamental goal of sampling is to record a meaningful subset of data instead of all of it, and to draw conclusions about the overall system behavior from this subset. There are two main sampling approaches: Head-based and Tail-based sampling. Each has its own advantages and disadvantages.

Head-based Sampling

Head-based sampling is a method where a decision is made right before the trace begins, i.e., at the “head.” In its simplest form, you decide to record a random portion (X%) of every incoming request. For example, recording 1 out of every 1000 requests.

# Example OpenTelemetry Python configuration (pseudo-code)
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Sample 1 out of every 1000 requests
sampler = ParentBased(TraceIdRatioBased(0.001))

provider = TracerProvider(sampler=sampler)
trace.set_tracer_provider(provider)

Advantages:

• Simplicity: Easy to implement and manage.
• Low Overhead: Data is collected and sent only for sampled traces, minimizing the load on the system.
• Real-time: Since the decision is made at the beginning, traces are processed immediately.

Disadvantages:

• Blind Spots: Because sampled traces are randomly selected, there’s a high chance of missing rare errors or performance issues that occur under specific conditions. In a production ERP, it was very difficult to catch a deadlock occurring with a specific product code using head-based sampling.
• Loss of Context: Since the decision of whether a trace is important is made at the very beginning of the system, critical information that emerges later in the trace (errors, slowdowns) cannot influence this decision.

I generally use head-based sampling as a starting point to observe the overall health and average performance of the system. For instance, I start with a rate of 0.1% and monitor the general trend of the system. However, this rate can be insufficient when a critical issue arises.

Tail-based Sampling

Tail-based sampling, on the other hand, is a method where the sampling decision is made at the “tail,” after all spans in a trace have been collected. With this approach, all trace data is temporarily stored somewhere, and after the trace is completed, a decision is made whether to record it based on specific rules (e.g., traces containing errors, traces exceeding a certain duration).

Advantages:

• Detailed Context: Allows for more informed sampling decisions by considering events that occur throughout the trace’s lifecycle (errors, slowdowns, specific business rules). This is invaluable, especially for root cause analysis.
• Error Catching: Offers the ability to automatically record all traces containing errors, preventing critical issues from being overlooked.
• Performance-Oriented: The ability to record traces that exceed a certain threshold (e.g., longer than 500ms) is very effective for identifying performance issues.

Disadvantages:

• High Overhead: Because all spans need to be temporarily stored and processed, it consumes significantly more system resources (memory, CPU) compared to head-based sampling. This can create a substantial cost and performance burden, especially in high-volume systems.
• Latency: Since the sampling decision is made after the trace is completed, there is a delay in the trace data reaching the observability platform. This can be an issue in situations requiring real-time monitoring.
• Complexity: Its setup and management are more complex than head-based sampling, often requiring a dedicated “sampling collector” layer.

When we tested tail-based sampling in a customer project, I saw the memory usage of the collector services peak, especially during peak hours. Similar to choosing a memory eviction policy in Redis, here we had to adjust memory.high cgroup limits. Therefore, I generally prefer to use tail-based sampling in a restricted manner for more critical workflows or specific error types.

Risks of Detail Loss and Dark Corners

The biggest disadvantage of sampling is that it inherently leads to information loss. When you sample randomly, you always run the risk of missing rare events or a unique scenario experienced by a specific user. This situation brings with it a problem known as “selection bias.” For example, while 99.9% of your system might be working flawlessly, only 0.1% of users might be encountering a very specific error. With head-based sampling, your chances of catching this 0.1% decrease as your sampling rate gets lower.

In an internal banking platform, only 0.05% of requests to a specific API caused a very rare database deadlock. It was impossible to detect this error with head-based sampling. To find such “dark corner” issues, you either need to keep your sampling rate very high (which is costly) or record all traces containing errors with tail-based sampling. However, tail-based sampling also has its own performance overhead.

Lost details can lead us to a dead end during root cause analysis. If you cannot find the trace of a specific user while investigating their complaint, it becomes impossible to understand where the problem originated. Sometimes, to find the cause of a problem, not only the traces that show errors but also the healthy traces immediately preceding the error are important. For instance, you might need both traces to see how a cache clearing operation affects the next request. Sampling can also break the relationship between such connected events. Therefore, I always consider these risks when determining my sampling strategy.

Methods for Optimizing Sampling Strategies

To minimize the risks of detail loss from sampling while benefiting from cost advantages, I use various optimization methods. These often involve hybrid combinations of Head-based and Tail-based approaches or smarter decision-making mechanisms.

1. Adaptive Sampling

Adaptive sampling is a method that dynamically adjusts the sampling rate based on the system’s current state. For example, it uses a low rate (0.1%) when the system is operating normally, but increases the sampling rate (1% or 10%) when error rates or latency exceed a certain threshold.

# Example OpenTelemetry Collector configuration (pseudo-code)
# This rule always records traces containing errors,
# and records 1% of other traces.
processors:
  tail_sampling:
    policies:
      [
        {
          name: error-policy,
          type: status_code,
          status_codes: [ERROR],
          on_error: true,
          sampling_rate: 1.0 # Record traces with errors 100%
        },
        {
          name: default-policy,
          type: probabilistic,
          sampling_percentage: 1 # Record 1% of the rest
        }
      ]

This approach reduces costs during normal times and ensures that the necessary details are captured during problem situations. In the backend of my own product, I implemented this strategy by monitoring error rates from Redis. If errors like OOM or CONNECTION_REFUSED started appearing from Redis, I set up a mechanism to automatically increase the tracing sampling rate.

2. Error-only and Critical Path Sampling

Recording all traces that contain errors is usually the highest priority. Errors are the clearest indication that something is wrong in the system and require full context for root cause analysis. Therefore, most sampling strategies aim to always record traces containing errors.

Additionally, I can set a higher sampling rate for specific API endpoints or business workflows that are critical from a business perspective. For example, I might use 100% sampling for flows like “order creation” or “payment processing,” while using 0.1% for less critical operations like “viewing user profile.” This allows me to focus resources on the most valuable information.

3. Distributed Context Propagation

Even if the sampling decision is made by the first service at the beginning of the trace, it is critical that this decision is correctly propagated to all other services throughout the trace. Standards like OpenTelemetry use HTTP headers (e.g., traceparent) to ensure this context propagation. If a trace is decided to be sampled, this information is transmitted to all services in the chain, and each service continues to produce spans for that trace. Otherwise, parts of the trace might be missing.

By combining these strategies, I keep costs under control while ensuring the necessary observability during critical moments. Last month, when I accidentally put sleep 360 in a cron job and got OOM-killed on a system, adaptive sampling allowed me to see exactly when and where the error occurred. This was a system I had set up precisely to catch such errors, and it worked.

Real-World Applications and What I’ve Achieved

I have applied distributed tracing and smart sampling strategies in various projects for years and have personally observed the results. In the backend of my own product, in a production ERP, and in various customer projects, these approaches have both increased operational efficiency and significantly reduced costs.

In a production ERP, when a new production planning module was deployed, monitoring the performance of AI-optimized planning algorithms was critical. Initially, we started with 100% tracing, but as the system load increased, the data volume sent to the observability platform exceeded 2TB daily, which equated to a monthly cost of $5,000 USD. By switching to an adaptive sampling strategy, we recorded only planning requests containing errors or exceeding 500ms at 100%, while sampling other successful and fast requests at a rate of 0.5%. With this transition, the data volume decreased by 95%, and the cost dropped to $250 USD per month. Most importantly, we continued to detect critical performance issues or algorithmic errors without significant loss of detail.

This table shows that sampling is not just a theoretical concept but provides tangible financial and operational benefits. Of course, a 2% loss in error detection rate is not negligible, but this loss was generally from recurring and less critical errors. The important thing is that we were still able to catch major issues that affected business or disrupted system stability.

My philosophy is to focus on pragmatic solutions rather than perfectionism, with an “it is what it is” approach. While achieving 100% tracing might always seem ideal, considering real-world costs and complexity, it is often not sustainable. The key is to find the right balance to have enough information about your system. I maintain this balance by continuously monitoring metrics (e.g., number of spans collected, percentage of traces containing errors, average trace duration) and adjusting sampling rates.

Conclusion: Risk or Necessity?

Sampling in distributed tracing carries a risk: loss of detail. However, my nearly 20 years of experience shows that this risk can be managed with the right strategies, and it is a necessity in today’s complex, high-volume systems. Without sampling, most organizations would be crushed by the cost burden of observability and would either be forced to disable tracing entirely or drown in a meaningless data heap.

My clear position is: Sampling is a necessity, but it must be implemented intelligently. By combining Head-based and Tail-based approaches, using adaptive sampling, and prioritizing critical workflows, you can keep costs under control and maintain your operational visibility. The key is to ask, “What am I losing?” and measure the impact of this loss on your business. If the details you lose are not vital to your system’s overall health and critical business workflows, sampling will definitely be worth it.

Remember, observability is not a luxury but an indispensable part of modern systems. And sampling is the key that makes this indispensable tool sustainable and economical. In my next post, I will discuss how we can store and query such observability data more efficiently.