Opsphere
← All articles

Cloudflare Incident Investigation: A Step-by-Step Workflow

Cloudflare Incident Investigation: A Step-by-Step Workflow

Abstract illustration of a structured incident investigation

Cloudflare sits in front of production traffic as a reverse proxy, DNS provider, cache, firewall and code runtime. When something breaks, the symptom you see (a 5xx, a blocked checkout, a stale page) rarely says which layer caused it. This guide gives SRE, DevOps, platform and security teams a repeatable way to investigate Cloudflare-related incidents, and shows where a read-only operational layer such as Opsphere helps correlate Cloudflare evidence with cloud, Kubernetes, deployment and code data.

Opsphere
Bring Incident Evidence Together
Opsphere unifies Cloudflare-related operational context with cloud, Kubernetes, observability, CI/CD and engineering tools in one interface.
Explore Opsphere

TL;DR

  • Problem: A Cloudflare incident often has several overlapping causes: DNS, a WAF rule, a Worker release, an unhealthy origin, or cache hiding a change.
  • Takeaway: Work from the outside in. Check Cloudflare status, identify the failing layer from the response and Ray ID, then line up Cloudflare changes against deploys and origin signals on one timeline.
  • Guardrail: Investigate with read-only access first. Record evidence before you change any rule, record or Worker.
  • Next step: Build a short checklist from the workflow below, and connect Cloudflare read-only tools to the systems you already use so the timeline is assembled in one query instead of six tabs.

Table of Contents

What Cloudflare does in your request path

Cloudflare provides DNS and CDN services and acts as a reverse proxy for proxied HTTP and HTTPS traffic. Requests route through its network, where it can apply load balancing, attack protection, caching and SSL/TLS handling before forwarding traffic to the origin. Its Anycast network spans 335+ cities, so a visitor reaches a nearby data center rather than your origin.

For incident work, this means one request can pass through several decision points before your application sees it:

  1. DNS: does the hostname resolve, and is the record proxied?
  2. Edge security: DDoS protection, managed rules, custom rules, rate limits.
  3. Rules and Workers: Snippets, Workers and rule lists that rewrite, redirect or respond.
  4. Cache: a hit is served without touching the origin.
  5. Load balancing and origin: pool and origin health, then your own infrastructure.

Cloudflare lists WAF, DDoS Protection, Bot Management, Rate Limiting, CDN, DNS, Load Balancing and Log Explorer among its application services. Each can be the cause, and each leaves different evidence.

How one incident produces several symptoms

The hard part is that failures at different layers look alike from the outside. A few realistic patterns:

Symptom Possible Cloudflare-side cause Possible origin or app-side cause
Users cannot reach the site DNS record changed or unproxied; platform incident Origin IP changed, load balancer down
Some users see a block page WAF custom or managed rule, rate limit, bot score Application deploy that changed request shape
5xx on a subset of paths Worker release throwing errors; rule redirecting wrongly Backend dependency failing on those paths
Old content after a release Cached response masking the new version Release did not roll out everywhere
Intermittent 522 or 524 Origin pool marked unhealthy or flapping Overloaded origin, firewall blocking edge IPs

The same incident can hold two of these at once. A Worker release can raise error rates while cache hides the problem for cached paths, so the first dashboards you open look healthy.

Step-by-step Cloudflare incident investigation workflow

Step 1: Scope the blast radius before touching anything

Cloudflare’s own triage guidance recommends recording the affected source IP, destination, ports, protocols, start time, recent configuration changes, and whether all traffic or only specific flows are affected before changing rules. Apply the same discipline to web incidents. Write down:

  • Which hostnames, paths and regions fail, and which still work.
  • The first failing timestamp, in UTC.
  • The HTTP status, any Cloudflare error code and the Ray ID from the response.
  • Whether failures hit all clients or a segment (a country, ASN, user agent, API clients only).

This scoping often splits the problem immediately. All paths and all clients failing points to DNS, certificates or a platform issue. One path failing for one segment points to a rule or Worker.

Step 2: Rule out a Cloudflare platform issue

Cloudflare’s support documentation says unexpected behavior should be checked against the Cloudflare Status page. You can also use the incident and maintenance RSS or Atom feeds, notifications and the location-status view. Check the status page for active incidents and scheduled maintenance affecting the locations your users hit. Two minutes here prevents an hour of hunting for a change that never happened on your side.

If status is clean, assume the cause is in configuration, code or origin and keep going.

Step 3: Identify the failing layer from the response

The response tells you a lot. Cloudflare’s documentation notes that errors can occur on proxied sites, and that the HTTP status and any 1xxx code help identify the cause in logs, analytics or monitoring. A rough decoder:

  • 1xxx codes: generated by Cloudflare itself, usually DNS, configuration or access rules.
  • 403 with a Cloudflare block page: a security feature acted on the request.
  • 52x codes: Cloudflare reached for the origin and could not get a healthy response.
  • Your app’s own 5xx with no Cloudflare page: the origin or a Worker returned it.

For a 522, Cloudflare’s documentation lists Cloudflare IP ranges being blocked or rate limited at the origin, an overloaded or offline origin, disabled keepalives, and a DNS origin IP that no longer matches the server as common causes. That is a ready-made checklist for the origin side of the investigation.

Step 4: Check DNS and proxy status

DNS changes are among the fastest ways to break production, and they are easy to miss because they look like an application problem. Verify:

  • The record exists, points to the intended target and has the expected proxied or DNS-only setting.
  • The origin IP in the record matches what your cloud provider currently assigns. Elastic IPs, load balancer addresses and Kubernetes ingress addresses all change under some conditions.
  • No recent edits to the zone. Check the audit log for who changed what and when.

Record type matters too. If an incident involves a CNAME pointing to a hostname that stopped resolving, the distinction in how A and CNAME records behave explains why a change at the target can break a name you never touched.

Step 5: Trace WAF and security decisions with the Ray ID

When legitimate users are blocked, the Ray ID is the key. A common walkthrough: take the Ray ID from the block page, open the zone’s Security Analytics, and filter events by Ray ID to see which rule produced the block.

Two cautions from Cloudflare’s network firewall documentation: traffic may be blocked independently by network-layer DDoS protection, Advanced TCP Protection, or custom and managed firewall rules, and an allow rule in one system does not bypass protection in another. So when you find one rule, confirm it is the only one acting. Also note whether a rule list (IP list, ASN list) was edited shortly before the incident; a list change alters behavior without any visible edit to the rule itself.

For a visitor-side block, remember that only the website owner can contact Cloudflare for technical support, so the block-page details must reach your team. Make sure your support channel asks for the Ray ID.

Step 5: Trace WAF and security decisions with the Ray ID

Step 6: Inspect Workers, Snippets and KV

Code running at the edge changes behavior without a deploy in your own pipeline, which is why it gets overlooked. Check:

  • Worker deployments: what version went live, when, and on which routes. Compare the deploy time to the first failing timestamp.
  • Routes and custom domains: a Worker on a broad route can intercept traffic you did not intend. Cloudflare notes that a Worker on a Custom Domain performing a fetch to its own hostname causes a 522 error.
  • Snippets: small rule-attached code that can rewrite headers or redirect.
  • KV: a Worker reading configuration or feature flags from KV behaves differently after a key change, with no code deploy at all.

A useful question: did anything change at the edge in the 30 minutes before the first error? Worker versions, Snippet edits and KV writes all belong on the same timeline as your application releases.

Step 7: Look at cache behavior

Caching at edge nodes means a cache hit is served close to the user without reaching your origin. That is good for performance and risky for diagnosis in two ways:

  • Masking: a broken origin release looks fine while cached responses live, then fails abruptly as entries expire or are purged.
  • Staleness: users see old content after a fix, and you chase a deploy that actually succeeded.

Compare cache status headers on a failing and a working path. If a purge happened near the incident, treat it as a change event: it can turn a hidden problem into a visible one within seconds.

Step 8: Check load balancing and origin health

If Cloudflare Load Balancing fronts your origins, review pools, origin weights and health checks. Typical findings: a health check path that started returning a redirect or 401 after a deploy, marking healthy origins as down; a pool with a single origin that failed; a failover pool that was never sized for full traffic.

Then move to the origin itself: load balancer targets, pods and nodes, recent rollouts, resource saturation, and whether security groups still allow Cloudflare’s IP ranges.

Checks for load balancer and origin health

Step 9: Pull logs and run network diagnostics

Logpush exports request and event logs to a destination you control, which lets you slice by status, path, rule, colo and Ray ID at scale. Use it to confirm what dashboards suggested: which rule fired, which Worker version served, what the cache status was, what the origin status code was. When the path between Cloudflare and origin is in doubt, a traceroute from the relevant network location adds evidence on routing or packet loss that application logs cannot show.

Step 10: Build the timeline, form hypotheses, then change one thing

Put every event in one UTC timeline: Cloudflare DNS edits, rule and list changes, Worker versions, KV writes, cache purges, origin deploys, Kubernetes rollouts, infrastructure changes and alerts. Then test hypotheses in parallel rather than serially:

  1. Config change at the edge (rule, list, Worker, DNS).
  2. Application or infrastructure release at the origin.
  3. Dependency failure behind the origin.
  4. Cloudflare platform issue.

Each hypothesis should name the evidence that would confirm or reject it. Change one thing at a time, and write down the before state so rollback is a lookup, not a guess.

Where the investigation slows down

The workflow above is simple on paper. In practice the evidence lives in the Cloudflare dashboard, a log destination, your cloud console, a Kubernetes cluster, an observability tool, a CI/CD system and a Git repository. Engineers spend the first part of an incident switching tabs and copying Ray IDs and timestamps between them.

This is a correlation problem, not a data problem. The data already exists in the right places. What is missing is one place to ask a question that spans them.

How Opsphere fits: a read-only layer over your existing tools

Opsphere is an operational intelligence layer that connects to the tools you already run and lets you query and correlate evidence across them. It does not replace Cloudflare, it does not store your Cloudflare telemetry, and it does not make changes automatically. The Cloudflare tools are read-only.

On the Cloudflare side, Opsphere has published read-only tools for Workers, Logpush, Snippets and KV and read-only tools for Load Balancing pools, origins and health checks. A separate release added Cloudflare cache purge, traceroute and incident rollup tools to the live set. Those map to Steps 6, 8 and 9 above.

Because the same interface also reaches AWS, Kubernetes, observability, deployment and code systems, the question “what changed near 14:05 UTC?” can cover an edge Worker version, an origin rollout and a health check change together. Teams reach this through the MCP Operational Gateway, from AI assistants and developer tools they already use, including Claude Code and GitHub Copilot CLI, or in Slack through the Slackbot integration. Teams that want to choose their own model can read about bringing their own LLM.

Example questions for a Cloudflare incident

Phrase these as plain-language queries against your connected sources:

  • Which Workers were deployed on this zone in the last two hours, and which routes do they serve?
  • Which origins in the production pool are marked unhealthy, and what do their health checks request?
  • Did any cache purge or Snippet change happen before the first 5xx?
  • Which Kubernetes deployments in the origin cluster rolled out in the same window?
  • Show Logpush-derived error counts by status for the affected hostname alongside the deploy timeline.

Every answer should come back with its evidence so the team can verify it in the source system. Cloudflare remains the source of truth for Cloudflare state, and the layer’s value is speed to a verified hypothesis.

Securing access when AI agents query operations data

Giving an AI assistant access to production tooling raises a fair security question. Read-only scope is the first control. Identity is the second. Cloudflare’s own guidance describes MCP portals that centralize AI tools as an Access application, with identity provider and device posture checks gating which agents and tools people can reach. If your organization has moved to identity-based access, the principles in this Zero Trust network design guide apply equally to the tools your incident responders use. Opsphere’s approach to this is described on its security page.

Make the workflow stick: a pre-incident checklist

Most of the time saved in an incident is prepared beforehand.

  • Capture Ray IDs everywhere. Support forms, error pages and client logs should surface them.
  • Turn on log export for the zones that matter, and know where the logs land and how long they are retained.
  • Tag edge changes. Worker, rule and DNS changes should appear in the same change feed as application deploys.
  • Keep health check endpoints stable. Treat them as contract, and test them in CI.
  • Allow Cloudflare IP ranges at the origin and document who owns that rule.
  • Subscribe to status notifications for the locations and products you use.
  • Write a runbook per symptom, starting from the table above, and rehearse it.

Promo: investigate Cloudflare incidents with one query

If your team is tired of assembling Cloudflare, cloud and deployment evidence by hand, Opsphere gives you a read-only way to do it in one interface. Plans start with a free Community tier, a Developer plan at 19 € per month, a Team plan at 500 € per month and an Enterprise plan at 450 € per month, with Custom Solutions available on request. See the pricing page for what each includes.

Opsphere

Ready to try it on your own stack? Start with Opsphere and connect Cloudflare alongside your cloud and observability tools, or explore the Opsphere Web Client for remote operations queries.

FAQ

What exactly does Cloudflare do?

Cloudflare provides DNS and CDN services and acts as a reverse proxy for proxied web traffic. Requests pass through its network, where it can apply caching, load balancing, attack protection and TLS handling before forwarding to your origin. It also offers security products such as a WAF and rate limiting, and a developer platform that includes Workers.

Why is Cloudflare blocking my access to a site?

A security feature on the site’s Cloudflare configuration, such as a firewall rule, rate limit or DDoS protection, acted on your request. Cloudflare says only the site owner can contact it for support, so send the owner the Ray ID and error code from the block page. Site owners can then search that Ray ID in Security Analytics to see which rule fired.

How do I know whether an outage is Cloudflare’s or mine?

Start with the Cloudflare Status page and its location view for active incidents or maintenance. If status is clean, scope which hostnames, paths and clients fail, then check the response code and Ray ID. Errors in the 52x family mean Cloudflare could not get a healthy response from your origin, which points to your side.

Does Google use Cloudflare?

The evidence we can cite is limited. In 2015, Cloudflare announced that it had partnered with Google as a Cloud Platform Technology Partner, letting Google Cloud Platform customers extend their infrastructure to Cloudflare’s edge. That describes a partnership for customers, not whether Google runs its own services on Cloudflare.

Can Opsphere change my Cloudflare configuration?

No. Opsphere’s Cloudflare tools are read-only. They help your team query and correlate evidence, and any change to rules, records or Workers stays in Cloudflare under your own access controls.

Sources

Opsphere
Discuss Your Incident Workflow
Contact Opsphere to discuss how its read-only operational tools can help your team investigate incidents across cloud and engineering systems.

This article is provided for general informational purposes only and does not constitute professional, legal, security, or compliance advice. Please evaluate recommendations against your organization’s specific environment and requirements.