Golden Signals Monitoring: The Four Signals and How to Use Them
Golden Signals Monitoring: The Four Signals and How to Use Them

Golden signals monitoring is a way to watch a user-facing service through four measurements: latency, traffic, errors and saturation. The idea comes from the monitoring chapter of Google’s Site Reliability Engineering book, which says that if you can measure only four things about a service, these are the four. For a small team, the value is a short, shared definition of “healthy” that every service can be held to. Opsphere is built on the same premise: operational context should come from the tools you already run, in one place, without another pile of dashboards.
TL;DR
- Problem: Most teams collect far more metrics than they can read during an incident, and each service ends up with its own idea of what to alert on.
- Takeaway: Latency, traffic, errors and saturation are the minimum health view for any user-facing service. Alert on symptoms users feel, and use saturation to warn before they feel them.
- Kubernetes: Apply the same four signals per service first, then per pod and node to find the cause.
- Next step: Pick your three most important services, define one SLI per signal for each, and review the alert list against those twelve measurements.
Table of Contents
- What the four golden signals are
- Latency: measure success and failure separately
- Traffic: the context for everything else
- Errors: count what users would call a failure
- Saturation: the early warning
- Is there a fifth golden signal?
- How the golden signals relate to RED and USE
- Golden signals in Kubernetes
- Turning signals into alerts people trust
- Where golden signals stop and investigation starts
- How Opsphere fits
- FAQ
- Sources
What the four golden signals are
The four golden signals are a short list of metrics that describe whether a service is working for the people using it. Each answers a different question.
| Signal | Question it answers | Typical measurement |
|---|---|---|
| Latency | How long do requests take? | Request duration, usually as p50, p95 and p99 percentiles |
| Traffic | How much demand is the service handling? | Requests per second, messages consumed, sessions |
| Errors | How many requests fail? | Share of requests that return errors or wrong results |
| Saturation | How full is the most constrained resource? | CPU, memory, disk, connection pool or queue depth against a limit |
The first three describe what users experience. The fourth describes how close the service is to a limit, which is why it is the one that can warn you before users notice anything.
The Google chapter states the promise plainly: if you measure all four and page a human when one is problematic (or, for saturation, nearly problematic), the service is at least decently covered by monitoring. That is a coverage argument, not a claim that four numbers explain every failure.
Latency: measure success and failure separately
Latency is the time it takes to serve a request. Two details matter in practice.
Separate successful from failed requests. The SRE book points out that a fast error can make an average look healthy. A request that fails immediately after a broken dependency rejects it takes almost no time, so mixing it with successes drags the latency number down exactly when the service is in trouble. Track the latency of successful requests on its own, and count slow errors as errors.
Use percentiles, not averages. An average hides the slow tail. If one request in a hundred takes ten seconds, the mean barely moves, but those users notice. Chart p50 for the typical experience and p95 or p99 for the tail, and set the threshold on the tail.
In Prometheus-style monitoring, a histogram of request duration lets you compute those percentiles at query time. Our guide to how Prometheus monitoring works and where teams hit limits covers what to watch for as histogram cardinality grows.

Traffic: the context for everything else
Traffic is the demand placed on the service. For a web API it is requests per second. For a queue consumer it is messages processed per second. For a database it may be queries or transactions.
Traffic rarely pages anyone on its own. Its job is to explain the other signals:
- Latency rising while traffic is flat points at the service or a dependency.
- Latency rising with traffic points at capacity.
- Errors rising while traffic drops can mean clients are giving up or a load balancer is failing before requests reach you.
A sudden drop to near zero deserves an alert too. A service that is silent is often a service that is broken upstream.
Errors: count what users would call a failure
Errors are the rate of requests that fail. That includes the obvious cases, such as HTTP 5xx responses, and the less obvious ones: a 200 response with the wrong content, or a response that arrives after the timeout your SLO allows.
Express errors as a ratio of total requests rather than a raw count, so the threshold holds as traffic grows or shrinks. A useful starting point is an SLO such as 99.9% of requests succeed over 30 days, with alerts based on how fast you are burning through the allowed failures, not on a single spike.
Be deliberate about which errors count. A client sending a malformed request is usually not a service failure, while a timeout calling your own database is. Write that rule down once per service so every on-call engineer reads the dashboard the same way.
Saturation: the early warning
Saturation measures how full the service’s most constrained resource is. It is the signal most often skipped, because it is less obvious than latency or errors, and the one that most often gives you time to act.
The constrained resource is not always CPU. Depending on the service it may be:
- memory, where hitting the limit means the process is killed
- disk space or disk I/O
- database connection pool slots
- thread or worker pool usage
- queue depth and consumer lag
Many systems degrade before they reach 100% utilization, so set the alert on the level where behavior starts to change, found by load testing or by reading past incidents. Pair it with a trend: a disk at 70% that fills at 5% a day is more urgent than a pool at 85% that has sat there for months.
Is there a fifth golden signal?
The canonical set is still four. You will see proposals for a fifth, most often liveness or availability, and sometimes cost or saturation of dependencies. Liveness is worth tracking, but it is largely covered by the existing four: a service that is down produces errors, or traffic that falls to zero, and the SLO built on the error ratio expresses availability directly.
The practical answer is to keep the four as the baseline and add signals only where a specific service has a failure the four cannot show. Examples include data freshness for a pipeline, or replication lag for a database. Treat those as service-specific SLIs, not as a change to the framework.
How the golden signals relate to RED and USE
Two other frameworks come up in the same conversations. The RED method (rate, errors, duration) is aimed at request-driven services. The USE method (utilization, saturation, errors) is aimed at resources such as hosts and disks. InfoQ’s overview of the SRE golden signals notes that the Google signals overlap with both.
A simple way to hold them together:
- Use the golden signals, or RED, to judge the service from the user’s side.
- Use USE to examine the nodes, pods, disks and pools underneath when a service signal goes bad.
This maps to how investigations actually go: a symptom appears at the service level, and the cause is found at the resource level.
Golden signals in Kubernetes
Kubernetes adds layers, so the same four signals need to be read at more than one level.
Start at the service. Measure latency, traffic and errors where requests enter: the ingress controller, the service mesh or the application’s own instrumentation. This tells you whether users are affected. It works whatever the pods underneath are doing.
Then go down a level to find the cause. Saturation is where Kubernetes needs the most care, because a pod has requests and limits, and the question is how close it is to its limit, not to the node’s capacity.
| Signal | Service level | Pod and node level |
|---|---|---|
| Latency | Ingress or application request duration percentiles | Dependency latency, DNS lookups, node network delays |
| Traffic | Requests per second per service and route | Requests per pod, to spot uneven load balancing |
| Errors | Share of 5xx or failed responses | Restarts, OOMKilled terminations, failed probes |
| Saturation | Autoscaler headroom, queue depth | CPU and memory usage against limits, throttling, pending pods, node pressure |
Two Kubernetes-specific pitfalls come up repeatedly:
- CPU throttling hides in plain sight. A container can sit well under the node’s capacity and still be throttled because it reached its own CPU limit. Compare usage to the pod’s limit, not to the node.
- Restarts are errors too. A pod that restarts every few minutes may keep the service-level error ratio low, because other replicas absorb the traffic, while the restart count shows the real problem.
For a fuller treatment of what to collect and alert on across clusters, see Kubernetes monitoring best practices for production teams. When an alert fires and you need a response routine, the Kubernetes incident response playbook picks up where monitoring ends.
Turning signals into alerts people trust
The signals only help if the alerts built on them are few and meaningful. A practical routine:
- Define an SLI per signal for each critical service. For example, p95 latency of successful requests, requests per second, error ratio and the saturation of the one resource that limits it.
- Page on symptoms. Page when users are affected: error ratio or tail latency beyond the SLO. Do not page on a single pod’s CPU.
- Ticket on saturation trends. A resource projected to fill within days is a task for working hours, not a 3 a.m. page, unless it will fill before anyone is awake.
- Write what the alert means. Each alert should link to the dashboard, the likely causes and the first diagnostic step.
- Review monthly. Count alerts that fired without needing action. Remove or retune them.
The measurable outcomes to track are simple: pages per on-call shift, share of pages that needed action, and time from alert to identified cause.
Where golden signals stop and investigation starts
The four signals tell you that something is wrong and roughly where. They do not tell you why. After the page, the engineer still has to look at recent deployments, infrastructure changes, logs, traces, cloud events and the state of dependencies. In most teams those live in separate tools, and the time goes into switching between them and rebuilding the picture by hand.
This is the tool fragmentation problem for small teams. Adding a dedicated product for each signal, or copying telemetry into yet another store, widens the gap instead of closing it. The more useful pattern is to keep each tool as the source of truth for its own data and add one layer that can query across them during an investigation.
How Opsphere fits

Opsphere is an AI-powered operational intelligence platform that connects to your existing AWS, Kubernetes, observability, CI/CD and security tools without requiring you to replace them or duplicate telemetry. It exposes more than 300 read-only operational tools through AI Agents and the Model Context Protocol (MCP), so an engineer can ask about a service whose latency or error signal has moved and have the evidence gathered from the systems that already hold it. Read-only access is a deliberate security choice: the platform investigates, and your tools remain the source of truth.
Teams that begin with the four signals as their alerting baseline get a clear starting question for each investigation: which signal moved, and what changed around it. You can read how this works for small SRE teams or see the Opsphere platform overview.

Plans start with Community at €0 per month and Developer at €19 per month, and Team is €500 per month. Details are on the pricing page. To see it on your own stack, start there and connect one service you already monitor against the four signals.
FAQ
What are the four golden signals used for monitoring IT systems?
They are latency, traffic, errors and saturation. Together they describe whether a user-facing service is fast enough, how much load it carries, how often it fails and how close it is to a resource limit. Teams use them as the minimum set of metrics to dashboard and alert on for every service.
Are the golden signals the same as SLIs and SLOs?
No, but they connect. The signals say what to measure. An SLI is a specific measurement of one of them, such as the share of requests served in under 300 ms. An SLO is the target for that SLI over a period, such as 99.9% over 30 days.
How do you monitor golden signals in Kubernetes?
Measure latency, traffic and errors at the ingress or application level for each service. Then measure saturation at the pod and node level, comparing CPU and memory usage to each pod’s limits and watching throttling, restarts and pending pods. Use the pod-level data to explain a service-level symptom.
Is liveness a fifth golden signal?
The canonical list remains four. Liveness is usually reflected in the error ratio and in traffic dropping to zero, so a separate signal is rarely needed. Add service-specific measurements, such as data freshness, only where the four cannot show a known failure.
Sources
Recommended
- Single Pane of Glass Monitoring: What It Is and How to Build One
- Prometheus Monitoring: How It Works and Where Teams Hit Limits
- eBPF Monitoring: How It Works and Where It Fits in 2026
This article is provided for general informational purposes only and does not constitute professional, legal, security, or compliance advice. Please evaluate recommendations against your organization’s specific environment and requirements.
