Kubernetes Instrumentation¶

Subscription required

This section describes functionality which requires an active IAPM subscription. Start your subscription by choosing the plan right for you.

Deploy OpenTelemetry on Kubernetes to automatically instrument your workloads and collect cluster-wide telemetry. This guide covers the OpenTelemetry Operator, the OTel Collector as a DaemonSet or sidecar, and Helm-based configuration.

Back to Instrument Overview

Overview¶

There are two main strategies for Kubernetes instrumentation:

Strategy	Best For	How It Works
OTel Operator auto-injection	Injecting instrumentation into existing pods without code changes	The Operator injects an init container and sidecar that auto-instrument your app
OTel Collector DaemonSet/Sidecar	Centralizing telemetry collection and routing	A Collector instance receives, processes, and exports telemetry from your pods

Most production deployments use both: the Operator for auto-instrumentation and a Collector DaemonSet for centralized export.

OpenTelemetry Operator Auto-Instrumentation¶

The OpenTelemetry Operator can automatically inject instrumentation into your pods.

Install the Operator¶

# Install cert-manager (required by the Operator)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

# Wait for cert-manager to be ready
kubectl wait --for=condition=Available deployment/cert-manager-webhook -n cert-manager --timeout=120s

# Install the OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Create an Instrumentation Resource¶

Define an Instrumentation custom resource that tells the Operator how to configure auto-instrumentation:

# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: iapm-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: https://otlp.iapm.app
  propagators:
    - tracecontext
    - baggage
  env:
    - name: OTEL_EXPORTER_OTLP_HEADERS
      valueFrom:
        secretKeyRef:
          name: iapm-api-key
          key: api-key
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:latest

Pin image versions in production

The examples above use :latest tags for brevity. In production, always pin to a specific version (e.g., :1.5.0) to ensure reproducible deployments.

Create the API Key Secret¶

kubectl create secret generic iapm-api-key \
  --from-literal=api-key="API-Key=YOUR-API-KEY"

Apply the Instrumentation Resource¶

kubectl apply -f instrumentation.yaml

Annotate Your Pods¶

Add an annotation to your pod spec to enable auto-instrumentation for your language:

.NETJavaPythonNode.jsGo

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-dotnet: "true"

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-go: "true"

Example Deployment with auto-instrumentation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_SERVICE_NAME
              value: my-app

OTel Collector as DaemonSet¶

Deploy a Collector DaemonSet so every node has a local Collector instance. Your application pods export to the local Collector, which batches and forwards telemetry to IAPM.

# otel-collector-daemonset.yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: iapm-collector
  namespace: otel-system
spec:
  mode: daemonset
  env:
    - name: IAPM_API_KEY
      valueFrom:
        secretKeyRef:
          name: iapm-api-key
          key: api-key
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      batch:
        timeout: 5s
        send_batch_size: 1024
      memory_limiter:
        check_interval: 5s
        limit_mib: 512
        spike_limit_mib: 128
      k8sattributes:
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.pod.name
            - k8s.node.name

    exporters:
      otlp/iapm:
        endpoint: otlp.iapm.app:443
        headers:
          API-Key: ${env:IAPM_API_KEY}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/iapm]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/iapm]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/iapm]

Point Applications to the Local Collector¶

Configure your application pods to export to the node-local Collector via the status.hostIP:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(NODE_IP):4317"
  - name: OTEL_SERVICE_NAME
    value: my-app

OTel Collector as Sidecar¶

For workloads that need a dedicated Collector instance per pod:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: iapm-sidecar
spec:
  mode: sidecar
  env:
    - name: IAPM_API_KEY
      valueFrom:
        secretKeyRef:
          name: iapm-api-key
          key: api-key
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      batch:
        timeout: 5s

    exporters:
      otlp/iapm:
        endpoint: otlp.iapm.app:443
        headers:
          API-Key: ${env:IAPM_API_KEY}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/iapm]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/iapm]

Annotate your pod to inject the sidecar:

metadata:
  annotations:
    sidecar.opentelemetry.io/inject: "true"

Helm Chart Configuration¶

Use the official OpenTelemetry Helm charts for a managed installation:

Add the Helm Repository¶

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Install the Collector¶

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace otel-system \
  --create-namespace \
  --values collector-values.yaml

collector-values.yaml:

mode: daemonset

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 5s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 5s
      limit_mib: 512

  exporters:
    otlp/iapm:
      endpoint: otlp.iapm.app:443
      headers:
        API-Key: YOUR-API-KEY

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/iapm]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/iapm]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/iapm]

Protect your API key

For production, use a Kubernetes Secret instead of hardcoding the API key in the Helm values. Reference the secret using envFrom or env in the Collector's pod spec.

Install the Operator via Helm¶

helm install otel-operator open-telemetry/opentelemetry-operator \
  --namespace otel-system \
  --create-namespace \
  --set admissionWebhooks.certManager.enabled=true

ConfigMap for OTLP Endpoint¶

Create a shared ConfigMap that all application pods can reference:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-config
  namespace: default
data:
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://iapm-collector-opentelemetry-collector.otel-system.svc.cluster.local:4317"
  OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"

Reference it in your Deployment:

spec:
  containers:
    - name: my-app
      envFrom:
        - configMapRef:
            name: otel-config
      env:
        - name: OTEL_SERVICE_NAME
          value: my-app
        - name: OTEL_EXPORTER_OTLP_HEADERS
          valueFrom:
            secretKeyRef:
              name: iapm-api-key
              key: api-key

Verify It's Working¶

Deploy your instrumented workloads
Check that the Collector pods are running: kubectl get pods -n otel-system
Generate some traffic to your application
Open portal.iapm.app and select your Grid
Click Enter - you should see your services and traces within a few minutes

Check Collector logs

kubectl logs -l app.kubernetes.io/name=opentelemetry-collector -n otel-system

Troubleshooting¶

Collector pods not starting¶

Check pod events: kubectl describe pod <pod-name> -n otel-system.
Ensure the memory limits in the Collector config do not exceed the pod resource limits.
Verify the iapm-api-key secret exists in the correct namespace.

No data from auto-instrumented pods¶

Verify the Instrumentation resource exists: kubectl get instrumentation.
Check that the pod annotation matches the language (e.g., inject-java, not inject-jvm).
Restart the pod after adding annotations - the injection happens at pod creation time.
Check the init container logs: kubectl logs <pod-name> -c opentelemetry-auto-instrumentation.

Applications cannot reach the Collector¶

When using a DaemonSet, ensure your application uses status.hostIP and port 4317.
When using a sidecar, the Collector is at localhost:4317.
Check network policies that might block traffic on port 4317.

High resource usage on Collector¶

Increase memory_limiter limits or add more restrictive batch processor settings.
Consider switching from DaemonSet to a Deployment with HPA if the load is uneven across nodes.

Use sampling to reduce trace volume:

processors:
  probabilistic_sampler:
    sampling_percentage: 25