In metric collection systems, cardinality is a critical concept for balancing performance and cost. I have prepared a 4-step guide on how this balance is established in the real world.
In this post, I will explain what metric cardinality is, why it matters, and how we can find the right balance in our systems based on my own experiences. We won’t just stick to theoretical knowledge; we will address this topic with concrete examples and steps.
What is Metric Cardinality and Why Should We Care?
Metric cardinality is the number of unique label combinations of the metrics we use in our monitoring systems. Simply put, the more different labels we use to define a metric, the higher its cardinality becomes. For example, when monitoring a server’s CPU usage, adding labels like instance, job, region, and az increases cardinality.
This has a direct impact on storage space, query performance, and costs. High cardinality requires more disk space, causes queries to run slower, and leads to higher costs in cloud environments.
The mindset of “let’s label everything” when collecting metrics might provide more visibility initially, but in the long run, it can make our systems unmanageable. Therefore, we must definitely consider cardinality when defining our metric collection strategies.
Step 1: Analyze Your Existing Metrics and Labels
The first step is to understand which metrics you are collecting and what labels you have assigned to them. This analysis will help you identify unnecessary or over-labeled metrics. Most monitoring systems offer a list of current metrics and labels. In Prometheus, you can access this data using commands like promtool tsdb analyze or through visualization tools like Grafana.
During this analysis, you should determine which labels provide truly distinctive information and which are just repetitive or static values. For example, adding a label like deployment_version: v1.2.3 to every metric needlessly increases cardinality if your entire system is running the same version. Instead, it might make more sense to access this information in a different way.
This analysis will reveal the “cardinality monsters” in your system. Recognizing these monsters is the first step to defeating them.
Step 2: Clean Up Non-Distinctive Labels
You should remove the unnecessary or repetitive labels you identified during the analysis from your system. This usually requires updating the configurations of your metric collectors (agents). For example, for Prometheus, you can use relabel_configs or metric_relabel_configs directives to remove unwanted labels.
It is important to be careful during this cleanup process. Accidentally removing an important label can limit your monitoring capabilities. Therefore, it is best to apply changes in a test environment first and carefully observe their effects. As an example, in a microservices architecture, a label like service_name is critical. However, more dynamic and always-unique labels like pod_name can increase cardinality and can be removed if they are not used in general queries.
The cleanup done in this step will directly provide storage space savings and query performance improvements.
Step 3: Adjust the Metric Collection Level
In some cases, adjusting the collection level of metrics can also help keep cardinality under control. For example, for less critical systems or situations requiring less detailed monitoring, you can collect metrics with fewer labels or with less frequent sampling. Many monitoring tools allow you to adjust the sampling rate for specific metrics.
However, this approach also has trade-offs. Lower sampling rates can make it harder to detect transient issues. Therefore, when adjusting the collection level, you need to carefully evaluate your monitoring needs and potential risks.
The goal in this step is to ensure that the collection level of each metric is at the level of detail truly needed.
Step 4: Use Static Values Instead of Dynamic Labels
Using static values instead of dynamic labels in metrics is one of the most effective ways to manage cardinality. For example, instead of a unique ID for each pod or container, it is better to use static labels that only indicate the environment (production, staging) or the service (auth-service, user-service).
This is especially important in cases where the values of labels change constantly. If you are assigning a constantly changing value to a metric, this metric is probably not the right way to access the desired information. For such information, using different mechanisms like logging or distributed tracing might be more appropriate.
By following these steps, you can effectively manage your metric cardinality, optimize your systems’ performance, and reduce costs.
In conclusion, metric cardinality is an important but often overlooked aspect of system monitoring. With the right analysis, cleanup, and labeling strategies, it is possible to build both more efficient and more cost-effective monitoring systems. By applying these 4 steps, you can fully leverage the power of metrics in your systems.