Opsphere
← All articles

Kubernetes Monitoring Best Practices for Production Teams

Updated

Kubernetes Monitoring Best Practices for Production Teams

Abstract layered Kubernetes monitoring illustration

Kubernetes produces thousands of time series, and most of them will never explain an outage. Good monitoring starts from user-facing service health, works down through the cluster layers, and keeps only the signals that lead to a decision. This guide covers what to collect, how to label it, how to alert, where anomaly detection fits and how to scale the setup.

Opsphere
Bring Operational Context Together
Opsphere unifies context across Kubernetes, observability, cloud and engineering tools to help teams investigate incidents from a single interface.
Visit

TL;DR

  • Problem: Default Kubernetes dashboards and alerts flood teams with cause-level noise while real user impact goes unnoticed.
  • Takeaway: Start from the two or three services that cost you the most when they fail. Alert on symptoms such as errors and latency. Keep causes (restarts, throttling, node pressure) on dashboards for diagnosis.
  • Data discipline: Bound your labels, set retention on purpose and sample traces. Cardinality is what breaks budgets and Prometheus servers first.
  • Anomaly detection: Use it as an early-warning layer next to threshold and SLO alerts, not instead of them.
  • Next step: Write down your service-level indicators for one critical service this week, then audit every alert against the question “what action does this trigger?”

Table of Contents

Start with services, then work down the layers

Most teams begin with node CPU and memory because those graphs ship by default. A better order starts from the user journey. Identify the two or three services whose downtime costs you something, define measurable indicators for each (availability, latency, throughput, correctness), and work backwards across the layers beneath them: control plane, nodes, workloads, services and ingress, network and DNS, storage and external dependencies.

For the service-level view, the four golden signals from the Google SRE book are the standard starting point: latency, traffic, errors and saturation. Measure latency for successful and failed requests separately, because fast errors hide inside a healthy-looking average.

Three frameworks cover most of the rest:

Framework Use it for What it measures
Four golden signals User-facing services Latency, traffic, errors, saturation
RED Request-serving components (ingress, APIs) Rate, errors, duration
USE Consumed resources (nodes, disks, etcd) Utilization, saturation, errors

The habit that matters is classifying every panel and alert as either a symptom (“is the service healthy?”) or a cause (“why is it unhealthy?”). Symptoms page people. Causes help them diagnose.

The layers that actually fail

Control plane

The API server is the front door of the cluster. Track request latency and 4xx/5xx rates, scheduler pending pods and etcd health. Microsoft’s AKS guidance groups proactive alerts into cluster infrastructure, application health and the control plane, with examples such as etcd utilization above 2 GB and API server 429, 500 and 503 responses. The same structure works on any distribution, even though the thresholds are Azure’s examples.

When the control plane is unhealthy, downstream signals mislead. That is why Splunk’s troubleshooting guidance suggests checking API server latency, scheduler queue length and crash loops before digging into individual workloads.

Nodes

Watch Ready and NotReady counts, CPU, memory and disk available versus requested. The kubelet exposes metrics at /metrics/cadvisor, /metrics/resource and /metrics/probes, and add-ons such as kube-state-metrics enrich them with the status of Kubernetes objects.

Workloads

The signals that predict real trouble at the pod level are pending pods, restart counts, CPU and memory against requests and limits, and CPU throttling. Red Hat’s overview of modern Kubernetes monitoring lists these along with per-pod network latency. Throttling deserves special attention: a container can sit well below its node’s capacity and still be slow because it keeps hitting its own CPU limit.

Network, storage and dependencies

DNS latency, ingress error rates, persistent volume attach failures and calls to databases or third-party APIs often explain incidents that look like application bugs. Kubernetes Events capture many of these directly: scheduling failures, probe failures, OOM kills and volume attach errors.

Collect metrics, logs, traces and events together

Kubernetes documentation describes observability as collecting and analyzing metrics, logs and traces to understand the internal state and health of a cluster. Each signal answers a different question:

  • Metrics: cheap, aggregated, ideal for alerting and trends.
  • Logs: the local explanation: the exception, the retry, the failed dependency call.
  • Traces: the path of one request across services, useful when flows are complex.
  • Events: the cluster’s own account of what it did: evictions, reschedules, failed probes.

If you use OpenTelemetry, its Kubernetes observability blueprint maps receivers to sources: the k8s_cluster receiver for object state (as kube-state-metrics does), kubeletstats for node, pod and container resource metrics, and hostmetrics for host-level data. It also recommends collecting Events and object state through the k8s_objects receiver, run as a single-replica Deployment with leader election.

Correlation is the real goal

The fastest investigations follow a path: a metric shows a latency regression or SLO violation, a trace identifies the slow hop or dependency, and a log explains the local failure. The CNCF’s guidance on Kubernetes observability recommends consistent metadata across signals and shared request identifiers between traces and logs, so engineers can pivot without searching three systems by hand.

Metric trace and log investigation path

In practice, that means agreeing on a small set of labels early: service, namespace, team or owner, environment and version. Teams that skip this step end up with dashboards nobody can filter and alerts nobody can route.

Control cardinality before it controls your bill

Cardinality is the number of unique label combinations behind a metric. In Prometheus, every distinct label set is a new series, so putting a user ID or a raw URL in a label can multiply series counts and take down the server. Bound labels to values such as route templates, status codes and pod names.

Practical rules:

  1. Never label by unbounded identifiers. User IDs, request IDs and container IDs belong in logs or traces, not metric labels.
  2. Aggregate at the right level. Red Hat’s guidance suggests aggregating cluster-level stats where possible and matching scrape intervals to how fast the data changes.
  3. Set retention and downsampling deliberately. Keep raw data short, and keep rollups for capacity planning and trend analysis.
  4. Sample traces. Head sampling decides at the start of a request. Tail sampling in the collector can keep every errored or slow trace while dropping most fast successes.
  5. Build in cost order. Metrics give the most insight per dollar, so build a broad metrics base first, add structured logs, then add traces where request flows justify them.

As the Apptio observability chapter points out, high-cardinality data can overwhelm tools and raise costs, so label strategy deserves planning rather than cleanup after the invoice arrives.

If you run Prometheus yourself, our overview of how Prometheus monitoring works and where teams hit limits covers the scaling points in more depth. For deeper kernel-level visibility with lower instrumentation effort, see eBPF monitoring and where it fits.

Alert on symptoms and ownership

Alert fatigue is a design problem, not a discipline problem. Grafana’s alerting best practices put it plainly: users do not care that a pod restarted, they care when the application is slow or failing. Alerts should tie to user impact, explain why they fired, link the relevant dashboards and runbooks, and name an owner and system scope.

The Prometheus alerting guidance adds two useful constraints: keep alerting simple, and page on latency at only one point in a stack so one slow dependency does not wake three teams.

Five questions before an alert goes live

The Google SRE book suggests testing each rule against five questions:

  1. Does it detect an urgent, actionable, user-visible condition?
  2. Will you ever ignore it as benign?
  3. Does it definitely indicate users are negatively affected?
  4. Can you take action on it?
  5. Are other people being paged for the same issue?

If an alert fails these, move it to a dashboard or a ticket queue.

SLO burn rate over raw thresholds

SLOs give teams a shared reliability goal, and error budgets give them time to act. Alerting on budget burn rate pages you when a problem is consuming reliability fast enough to matter, rather than when a single metric crosses an arbitrary line. For a worked example of tying SLOs to scaling decisions, see our piece on SLO-first predictive autoscaling.

Review on a schedule

Treat alerts like code. Review them every sprint, delete or tune the ones that never lead to action, group related alerts into one notification and use inhibition rules so a critical alert suppresses its related warnings. Every alert needs a runbook link and an owner. Teams that want a deeper look at noise reduction can read about reducing false positive alerts.

Add anomaly detection as an early-warning layer

Static thresholds catch what you predicted. Anomaly detection catches drift you did not predict: a pod’s memory growing steadily toward an OOM kill, an unusual rise in restarts, or network I/O that deviates from its normal daily shape.

A common approach learns a baseline for each workload and flags deviations. Grafana’s write-up on anomaly detection with Prometheus shows a z-score method (average plus or minus a multiple of the standard deviation) and makes an important point: anomalies give context but are not strong enough signals on their own. Tie them back to your SLO-based alerts as part of root-cause analysis.

A workable setup:

  • Keep SLO burn-rate alerts as the paging layer.
  • Run anomaly detection on a short list of leading indicators: memory trajectory, restart frequency, error-rate shape, saturation of queues.
  • Route anomaly findings to a ticket or a chat channel first. Promote one to a page only after it has repeatedly preceded real incidents.
  • Record what each anomaly turned out to be. That history is what makes the next investigation faster.

Practices for GKE and Google Cloud

Teams running on Google Cloud get a lot of this wired in. By default, GKE sends system, audit and application logs to Cloud Logging and system metrics to Cloud Monitoring. Configured third-party and user-defined metrics flow through Managed Service for Prometheus. You can add control plane, kube-state and cAdvisor/kubelet metric packages, and use dashboards, recommended alert policies, SLOs and Logs Explorer for troubleshooting.

Some specifics worth applying:

  • Smooth alert conditions. GKE’s own proactive monitoring example alerts when container CPU limit utilization stays above 0.8 for five continuous minutes, using a one-minute rolling mean so brief spikes do not fire it.
  • Match evaluation to data resolution. When alerting through Grafana on Cloud Monitoring data, set the evaluation interval at or above the minimum data resolution (most GCP metrics have one-minute granularity), reduce multiple series with expressions and test queries in Explore first.
  • Check Autopilot coverage. Control plane metrics were announced as generally available in GKE for clusters on version 1.23.6 and up, but not for Autopilot clusters, so confirm what your cluster type exposes in current documentation.
  • Pivot from logs. Cloud Logging can show related pod, node and cluster metrics, events and alerts from the log line itself, which shortens the metric-to-log hop.

Scale across clusters without losing clarity

The first cluster is easy. By the fifth, the problems change: duplicated dashboards, inconsistent labels, runaway storage and credentials scattered across scrapers. Plan for these:

  • Standardize metadata first. Every cluster should emit the same label keys (cluster, environment, team, service). Without them, cross-cluster queries are guesswork.
  • Decide what is local and what is global. Keep high-resolution data near each cluster and ship aggregated rollups to a central view. This limits cost and keeps one noisy cluster from degrading everything.
  • Secure the telemetry path. Metrics endpoints on the API server and kubelet are protected by RBAC, so scrapers need a service account with only the access they require. Treat dashboards and logs as sensitive, since they can expose internal hostnames and error payloads.
  • Keep ownership explicit. Every dashboard and alert should map to a team. Alerts without owners are the first ones ignored.
  • Prefer read-only access for investigation tooling. Anything that helps engineers query production during an incident should not be able to change it.

Where Opsphere fits

Monitoring tools tell you something is wrong. During an incident, engineers still jump between metrics, traces, deployment history, cloud events and tickets to work out why. Opsphere is an operational intelligence platform that connects to your existing tools instead of replacing them, so you do not duplicate telemetry. It brings AWS, Kubernetes, observability, CI/CD and security context into one investigation interface, using AI agents, Model Context Protocol and more than 300 read-only operational tools.

Opsphere

For teams already using Prometheus and Grafana, the integrations let engineers run dashboards, alerts and PromQL queries inside operational investigations, while the MCP operational gateway gives AI assistants structured, governed access to that context. Security details are on our security page. Plans on the pricing page start with a Community tier at €0 per month, followed by Developer at €19 per month, Enterprise at €450 per month and Team at €500 per month, plus Custom Solutions for larger needs. You can also explore the Opsphere platform.

A 30-day rollout checklist

Week 1: define. Pick two or three critical services. Write availability and latency SLIs and an SLO for each. Agree on the label standard.

Week 2: collect. Deploy kube-state-metrics and kubelet scraping (or the OpenTelemetry equivalents). Capture Events. Set label limits and scrape intervals. Turn on structured logging for the critical services.

Week 3: alert. Create symptom-based alerts and burn-rate alerts. Attach an owner and a runbook to each. Demote every cause-level page to a dashboard.

Week 4: tune. Review every page fired during the month. Delete or rewrite any that required no action. Add one or two anomaly detection jobs on leading indicators and review their findings weekly.

Measure the outcome with numbers you already have: pages per on-call shift, share of alerts with an action taken, and time from page to first useful finding.

A 30-day rollout checklist

FAQ

What are the most important Kubernetes metrics to monitor?

Start with service-level signals: latency, traffic, errors and saturation. Beneath them, track node readiness, CPU and memory against requests and limits, CPU throttling, pod restarts and pending pods, plus API server latency and error rates and etcd health. These cover the layers where production issues most often start.

What is the difference between Kubernetes monitoring and observability?

Monitoring watches known signals and alerts when they cross conditions you defined. Observability means collecting enough metrics, logs, traces and events, with consistent metadata, to investigate questions you did not anticipate. In practice you need both: monitoring to detect, observability to explain.

How do I avoid alert fatigue in Kubernetes?

Page only on symptoms tied to user impact, such as error rates, latency and SLO burn, and give every alert an owner and a runbook. Review alerts regularly and remove the ones that never lead to action. Keep cause-level signals like restarts and throttling on dashboards.

Should anomaly detection replace threshold alerts?

No. Anomaly detection is best as an early-warning layer that flags drift before it becomes an outage, while threshold and SLO alerts remain the paging layer. Promote an anomaly to a page only after it has proven it precedes real incidents.

How do I keep Prometheus costs under control in Kubernetes?

Limit label cardinality by avoiding unbounded values like user IDs and raw URLs, aggregate where you can, match scrape intervals to how quickly data changes and set retention and downsampling policies. Sample traces rather than storing all of them.

Opsphere
Discuss Kubernetes Monitoring
Reach out to discuss how Opsphere brings operational context together across Kubernetes, observability and cloud tools for your team.

Sources

This article is provided for general informational purposes only and does not constitute professional, legal, security, or compliance advice. Please evaluate recommendations against your organization’s specific environment and requirements.