Scaling the Collector

When planning your observability pipeline with the OpenTelemetry Collector, you should consider ways to scale the pipeline as your telemetry collection increases.

The following sections will guide you through the planning phase discussing which components to scale, how to determine when it’s time to scale up, and how to execute the plan.

What to Scale

While the OpenTelemetry Collector handles all telemetry signal types in a single binary, the reality is that each type may have different scaling needs and might require different scaling strategies. Start by looking at your workload to determine which signal type is expected to have the biggest share of the load and which formats are expected to be received by the Collector. For instance, scaling a scraping cluster differs significantly from scaling log receivers. Think also about how elastic the workload is: do you have peaks at specific times of the day, or is the load similar across all 24 hours? Once you gather that information, you will understand what needs to be scaled.

For example, suppose you have hundreds of Prometheus endpoints to be scraped, a terabyte of logs coming from fluentd instances every minute, and some application metrics and traces arriving in OTLP format from your newest microservices. In that scenario, you’ll want an architecture that can scale each signal individually: scaling the Prometheus receivers requires coordination among the scrapers to decide which scraper goes to which endpoint. In contrast, we can horizontally scale the stateless log receivers on demand. Having the OTLP receiver for metrics and traces in a third cluster of Collectors would allow us to isolate failures and iterate faster without fear of restarting a busy pipeline. Given that the OTLP receiver enables the ingestion of all telemetry types, we can keep the application metrics and traces on the same instance, scaling them horizontally when needed.

When to Scale

Once again, we should understand our workload to decide when it’s time to scale up or down, but a few metrics emitted by the Collector can give you good hints on when to take action.

One helpful hint the Collector can give you when the memory_limiter processor is part of the pipeline is the metric otelcol_processor_refused_spans . This processor allows you to restrict the amount of memory the Collector can use. While the Collector may consume a bit more than the maximum amount configured in this processor, new data will eventually be blocked from passing through the pipeline by the memory_limiter, which will record the fact in this metric. The same metric exists for all other telemetry data types. If data is being refused from entering the pipeline too often, you’ll probably want to scale up your Collector cluster. You can scale down once the memory consumption across the nodes is significantly lower than the limit set in this processor.

Another set of metrics to keep in sight are the ones related to the queue sizes for exporters: otelcol_exporter_queue_capacity and otelcol_exporter_queue_size. The Collector will queue data in memory while waiting for a worker to become available to send the data. If there aren’t enough workers or the backend is too slow, data starts piling up in the queue. Once the queue has hit its capacity (otelcol_exporter_queue_size > otelcol_exporter_queue_capacity) it rejects data (otelcol_exporter_enqueue_failed_spans). Adding more workers will often make the Collector export more data, which might not necessarily be what you want (see When NOT to scale).

It’s also worth getting familiar with the components that you intend to use, as different components might produce other metrics. For instance, the load-balancing exporter will record timing information about the export operations, exposing this as part of the histogram otelcol_loadbalancer_backend_latency. You can extract this information to determine whether all backends are taking a similar amount of time to process requests: single backends being slow might indicate problems external to the Collector.

For receivers doing scraping, such as the Prometheus receiver, the scraping should be scaled, or sharded, once the time it takes to finish scraping all targets often becomes critically close to the scrape interval. When that happens, it’s time to add more scrapers, usually new instances of the Collector.

When NOT to scale

Perhaps as important as knowing when to scale is to understand which signs indicate that a scaling operation won’t bring any benefits. One example is when a telemetry database can’t keep up with the load: adding Collectors to the cluster won’t help without scaling up the database. Similarly, when the network connection between the Collector and the backend is saturated, adding more Collectors might cause a harmful side effect.

Again, one way to catch this situation is by looking at the metrics otelcol_exporter_queue_size and otelcol_exporter_queue_capacity. If you keep having the queue size close to the queue capacity, it’s a sign that exporting data is slower than receiving data. You can try to increase the queue size, which will cause the Collector to consume more memory, but it will also give some room for the backend to breathe without permanently dropping telemetry data. But if you keep increasing the queue capacity and the queue size keeps rising at the same proportion, it’s indicative that you might want to look outside of the Collector. It’s also important to note that adding more workers here would not be helpful: you’ll only be putting more pressure on a system already suffering from a high load.

Another sign that the backend might be having problems is an increase in the otelcol_exporter_send_failed_spans metric: this indicates that sending data to the backend failed permanently. Scaling up the Collector will likely only worsen the situation when this is consistently happening.

How to Scale

At this point, we know which parts of our pipeline needs scaling. Regarding scaling, we have three types of components: stateless, scrapers, and stateful.

Most Collector components are stateless. Even if they hold some state in memory, it isn’t relevant for scaling purposes.

Scrapers, like the Prometheus receiver, are configured to obtain telemetry data from external locations. The receiver will then scrape target by target, putting data into the pipeline.

Components like the tail sampling processor cannot be easily scaled, as they keep some relevant state in memory for their business. Those components require some careful consideration before being scaled up.

Scaling Stateless Collectors

The good news is that most of the time, scaling the Collector is easy, as it’s just a matter of adding new replicas and using an off-the-shelf load balancer. When gRPC is used to receive the data, we recommend using a load-balancer that understands gRPC. Otherwise, clients will always hit the same backing Collector.

You should still consider splitting your collection pipeline with reliability in mind. For instance, when your workloads run on Kubernetes, you might want to use DaemonSets to have a Collector on the same physical node as your workloads and a remote central Collector responsible for pre-processing the data before sending the data to the storage. When the number of nodes is low and the number of pods is high, Sidecars might make more sense, as you’ll get a better load balancing for the gRPC connections among Collector layers without needing a gRPC-specific load balancer. Using a Sidecar also makes sense to avoid bringing down a crucial component for all pods in a node when one DaemonSet pod fails.

The sidecar pattern consists in adding a container into the workload pod. The OpenTelemetry Operator can automatically add that for you. To accomplish that, you’ll need an OpenTelemetry Collector CR and you’ll need to annotate your PodSpec or Pod telling the operator to inject a sidecar:

---
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar-for-my-workload
spec:
  mode: sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    processors:

    exporters:
      # Note: Prior to v0.86.0 use the `logging` instead of `debug`.
      debug:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [debug]    
---
apiVersion: v1
kind: Pod
metadata:
  name: my-microservice
  annotations:
    sidecar.opentelemetry.io/inject: 'true'
spec:
  containers:
    - name: my-microservice
      image: my-org/my-microservice:v0.0.0
      ports:
        - containerPort: 8080
          protocol: TCP

In case you prefer to bypass the operator and add a sidecar manually, here’s an example:

apiVersion: v1
kind: Pod
metadata:
  name: my-microservice
spec:
  containers:
    - name: my-microservice
      image: my-org/my-microservice:v0.0.0
      ports:
        - containerPort: 8080
          protocol: TCP
    - name: sidecar
      image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.69.0
      ports:
        - containerPort: 8888
          name: metrics
          protocol: TCP
        - containerPort: 4317
          name: otlp-grpc
          protocol: TCP
      args:
        - --config=/conf/collector.yaml
      volumeMounts:
        - mountPath: /conf
          name: sidecar-conf
  volumes:
    - name: sidecar-conf
      configMap:
        name: sidecar-for-my-workload
        items:
          - key: collector.yaml
            path: collector.yaml

Scaling the Scrapers

Some receivers are actively obtaining telemetry data to place in the pipeline, like the hostmetrics and prometheus receivers. While getting host metrics isn’t something we’d typically scale up, we might need to split the job of scraping thousands of endpoints for the Prometheus receiver. And we can’t simply add more instances with the same configuration, as each Collector would try to scrape the same endpoints as every other Collector in the cluster, causing even more problems, like out-of-order samples.

The solution is to shard the endpoints by Collector instances so that if we add another replica of the Collector, each one will act on a different set of endpoints.

One way of doing that is by having one configuration file for each Collector so that each Collector would discover only the relevant endpoints for that Collector. For instance, each Collector could be responsible for one Kubernetes namespace or specific labels on the workloads.

Another way of scaling the Prometheus receiver is to use the Target Allocator: it’s an extra binary that can be deployed as part of the OpenTelemetry Operator and will distribute Prometheus scrape targets for a given configuration across the cluster of Collectors. You can use a Custom Resource (CR) like the following to make use of the Target Allocator:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]

    exporters:
      # Note: Prior to v0.86.0 use the `logging` instead of `debug`.
      debug:

    service:
      pipelines:
        traces:
          receivers: [prometheus]
          processors: []
          exporters: [debug]    

After the reconciliation, the OpenTelemetry Operator will convert the Collector’s configuration into the following:

exporters:
   # Note: Prior to v0.86.0 use the `logging` instead of `debug`.
   debug: null
 receivers:
   prometheus:
     config:
       global:
         scrape_interval: 1m
         scrape_timeout: 10s
         evaluation_interval: 1m
       scrape_configs:
       - job_name: otel-collector
         honor_timestamps: true
         scrape_interval: 10s
         scrape_timeout: 10s
         metrics_path: /metrics
         scheme: http
         follow_redirects: true
         http_sd_configs:
         - follow_redirects: false
           url: http://collector-with-ta-targetallocator:80/jobs/otel-collector/targets?collector_id=$POD_NAME
service:
   pipelines:
     traces:
       exporters:
       - debug
       processors: []
       receivers:
       - prometheus

Note how the Operator added a global section and a new http_sd_configs to the otel-collector scrape config, pointing to a Target Allocator instance it provisioned. Now, to scale the collectors, change the “replicas” attribute of the CR and the Target Allocator will distribute the load accordingly by providing a custom http_sd_config per collector instance (pod).

Scaling Stateful Collectors

Certain components might hold data in memory, yielding different results when scaled up. It is the case for the tail-sampling processor, which holds spans in memory for a given period, evaluating the sampling decision only when the trace is considered complete. Scaling a Collector cluster by adding more replicas means that different collectors will receive spans for a given trace, causing each collector to evaluate whether that trace should be sampled, potentially coming to different answers. This behavior results in traces missing spans, misrepresenting what happened in that transaction.

A similar situation happens when using the span-to-metrics processor to generate service metrics. When different collectors receive data related to the same service, aggregations based on the service name will be inaccurate.

To overcome this, you can deploy a layer of Collectors containing the load-balancing exporter in front of your Collectors doing the tail-sampling or the span-to-metrics processing. The load-balancing exporter will hash the trace ID or the service name consistently and determine which collector backend should receive spans for that trace. You can configure the load-balancing exporter to use the list of hosts behind a given DNS A entry, such as a Kubernetes headless service. When the deployment backing that service is scaled up or down, the load-balancing exporter will eventually see the updated list of hosts. Alternatively, you can specify a list of static hosts to be used by the load-balancing exporter. You can scale up the layer of Collectors configured with the load-balancing exporter by increasing the number of replicas. Note that each Collector will potentially run the DNS query at different times, causing a difference in the cluster view for a few moments. We recommend lowering the interval value so that the cluster view is different only for a short period in highly-elastic environments.

Here’s an example configuration using a DNS A record (Kubernetes service otelcol on the observability namespace) as the input for the backend information:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:

exporters:
  loadbalancing:
    protocol:
      otlp:
    resolver:
      dns:
        hostname: otelcol.observability.svc.cluster.local

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - loadbalancing