Kubernetes Health Setup

Populate the Kubernetes dashboard using either OTEL-native collectors or a Prometheus scraping setup (kube-state-metrics + cAdvisor).

Back to Health View
Option A — OTEL-Native Collectors

Full OTEL reference architecture with kubeletstats, k8s_cluster, and hostmetrics receivers. Emits k8s.* metric names.

  • Requires deploying an OTEL Collector Deployment + DaemonSet
  • Helm-based, works with otel/opentelemetry-collector-contrib
  • Metric names: k8s.node.condition_ready, k8s.pod.status_ready, etc.
Option B — Prometheus Scraping (kube-state-metrics + cAdvisor)

Simpler setup using a single OTEL collector with a Prometheus receiver that scrapes existing Kubernetes exporters.

  • Works with kube-state-metrics, kubelet /metrics/cadvisor, node-exporter
  • Metric names: kube_node_status_condition, kube_pod_status_phase, etc.
  • Auto-detected — no configuration change needed in SOBS
What You Need
  • Kubernetes cluster (1.20+)
  • kubectl access to the cluster
  • Helm 3.x installed
  • SOBS running with OTLP HTTP receiver enabled (port 4318)
What You Get
  • Node status & readiness
  • Pod phases & restart counts
  • Deployment desired/ready replicas
  • Namespace-level resource aggregates
Option A — OTEL-Native Collector Setup
  1. Add the OpenTelemetry Helm repository:
    helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
    helm repo update
  2. Install cluster-wide collector (Deployment):
    Handles Kubernetes API metadata, events, and cluster-level metrics.
    helm install sobs-k8s-deployment \
      open-telemetry/opentelemetry-collector \
      -f k8s/otel-k8s-deployment.yaml
  3. Install node-level collector (DaemonSet):
    Runs on every node to collect kubelet metrics, host metrics, and container logs.
    helm install sobs-k8s-daemonset \
      open-telemetry/opentelemetry-collector \
      -f k8s/otel-k8s-daemonset.yaml
  4. Verify collectors are running:
    kubectl get pods -o wide | grep -i otel
  5. Enable in SOBS:
    • Navigate to Settings → Kubernetes
    • Check the "Enable Kubernetes Health View" checkbox
    • Save settings
    • Data will appear in Kubernetes Health dashboard as metrics flow in
Option B — Prometheus Scraping Setup (Simpler)

If you already have kube-state-metrics and kubelet/cAdvisor running in your cluster, a single OTEL collector with a prometheus receiver can scrape them and forward to SOBS. SOBS auto-detects Prometheus-format metrics and adjusts queries accordingly — no extra configuration needed.

  1. Install kube-state-metrics (if not already present):
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
    helm install kube-state-metrics prometheus-community/kube-state-metrics -n kube-system
  2. Deploy a single OTEL collector with Prometheus receiver:
    kubectl apply -f k8s/otel-prometheus-receiver.yaml

    See the reference manifest below for a minimal working configuration.

  3. Enable in SOBS:
    • Navigate to Settings → Kubernetes
    • Check "Enable Kubernetes Health View" and save
    • SOBS will auto-detect Prometheus metric names and populate the dashboard
SOBS checks for OTEL-native metrics first. If none are found, it automatically falls back to querying Prometheus-style kube_* metric names. No configuration flag is needed.
Collector Architecture
OTEL-Native: Cluster-Wide Deployment (Single Pod)

Runs once in the cluster (in the OpenTelemetry Collector namespace).

  • kubernetesAttributes: Enriches all signals with pod/node/cluster context
  • kubernetesEvents: Collects Kubernetes API events (pod created, deployment updated, etc.)
  • clusterMetrics: Gathers cluster-level state (nodes, namespaces, deployments)
Prometheus: Single Collector with Prometheus Receiver

Scrapes existing Kubernetes exporters and forwards to SOBS via OTLP.

  • kube-state-metrics: Deployment/pod/node/namespace state & counts
  • kubelet /metrics/cadvisor: Container CPU, memory, network usage
  • node-exporter: Node-level OS metrics (optional)
Data Flow (Prometheus path):
kube-state-metrics + cAdvisor → OTEL Prometheus receiver → OTLP HTTP (port 4318) → SOBS ClickHouse
Dashboard Data Mapping
Dashboard Section OTEL-Native Metric Prometheus Equivalent
Nodes — ready k8s.node.condition_ready kube_node_status_condition{condition="Ready",status="true"}
Nodes — memory k8s.node.memory.usage kube_node_status_allocatable{resource="memory"}
Pods — phase k8s.pod.phase attribute kube_pod_status_phase{phase="Running"} (value=1)
Pods — ready k8s.pod.status_ready kube_pod_status_ready{condition="true"}
Pods — memory k8s.pod.memory.usage container_memory_working_set_bytes
Pods — restarts k8s.container.restart_count kube_pod_container_status_restarts_total
Deployments k8s.deployment.desired / ready kube_deployment_spec_replicas / kube_deployment_status_replicas_ready
Namespaces k8s.namespace.name attribute kube_namespace_status_phase
Troubleshooting
Collectors not starting?

Check logs and resource requests:

kubectl logs -l app.kubernetes.io/name=opentelemetry-collector
No data appearing in SOBS?

Verify OTLP endpoint is reachable from cluster:

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl -v http://sobs:4318/v1/metrics
Metrics arriving but dashboard empty?

Verify the metric names present in SOBS via SQL:

SELECT DISTINCT MetricName FROM default.otel_metrics_gauge WHERE MetricName LIKE 'kube_%' LIMIT 20

If you see kube_* names, SOBS should auto-detect the Prometheus format.

Wrong SOBS endpoint?

Update the Helm values endpoint field and upgrade:

helm upgrade sobs-k8s-deployment ... --set-string endpoint=http://your-sobs-ip:4318
Reference Manifests 

mode: deployment
image:
  repository: otel/opentelemetry-collector-contrib
  tag: 0.123.0

presets:
  kubernetesAttributes:
    enabled: true
    extractAllPodLabels: true
    extractAllPodAnnotations: true
  kubernetesEvents:
    enabled: true
  clusterMetrics:
    enabled: true

exporters:
  otlphttp/sobs:
    endpoint: http://sobs:4318
    compression: gzip

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/sobs]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/sobs]

mode: daemonset
image:
  repository: otel/opentelemetry-collector-contrib
  tag: 0.123.0

presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
    extractAllPodLabels: true
    extractAllPodAnnotations: true
  kubeletMetrics:
    enabled: true

config:
  receivers:
    kubeletstats:
      auth_type: serviceAccount
      endpoint: "https://${K8S_NODE_NAME}:10250"
      insecure_skip_verify: true

rbac:
  create: true

serviceAccount:
  create: true

clusterRole:
  create: true
  rules:
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]

exporters:
  otlphttp/sobs:
    endpoint: http://sobs:4318
    compression: gzip

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/sobs]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/sobs]

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: kube-state-metrics
              static_configs:
                - targets: ['kube-state-metrics.kube-system.svc:8080']
            - job_name: kubelet-cadvisor
              scheme: https
              tls_config:
                insecure_skip_verify: true
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
                - role: node
              relabel_configs:
                - replacement: /metrics/cadvisor
                  target_label: __metrics_path__

    exporters:
      otlphttp/sobs:
        endpoint: http://sobs:4318
        compression: gzip

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [otlphttp/sobs]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: otel-collector
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.123.0
          args: ["--config=/conf/config.yaml"]
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: otel-collector-config