Opsphere
← All articles

AWS Incidents for SREs: Check CloudTrail Before Chasing Logs

AWS Incidents for SREs: Check CloudTrail Before Chasing Logs

Geometric CloudTrail incident investigation title card

AWS is a major cloud provider, pairing a large service portfolio with global infrastructure that engineering teams build on. When something breaks, start with the affected Region: check CloudTrail event history or the AWS Management Console there, then move to aggregated logs. Region and Availability Zone awareness separates a fast diagnosis from a long one.

TL;DR:

  • Choose a Region by balancing user latency, data residency rules, and service prices; compare options with the AWS Pricing Calculator, including transfer charges.
  • CloudTrail event history retains 90 days, while IAM events appear under the US East 1 prefix; create a trail to preserve older evidence.
  • External probes can stay green during a single zone fault; compare Availability Zone health and shift traffic away from the affected zone.
  • With EC2, teams patch guest operating systems, while AWS handles more of the platform for RDS or Lambda; data, access, and encryption remain customer responsibilities.
Opsphere
opsphere.io
Investigate AWS Incidents With Context
Bring AWS operational context together with Kubernetes, observability and engineering tools to investigate incidents from one interface.
Explore Opsphere

Table of Contents

How AWS global infrastructure shapes operations

AWS describes itself as the world’s most comprehensive and broadly adopted cloud. It publishes that it offers over 240 services on infrastructure spanning many Regions and Availability Zones. For operations teams, the useful part is how that footprint is organized.

  • Region: a separate geographic area. Resources, pricing and service availability are scoped to it.
  • Availability Zone (AZ): an isolated location inside a Region. Spreading workloads across AZs limits the blast radius of a single-zone fault.
  • Edge locations: points that serve content closer to users, which matters for latency-sensitive traffic.

Mapping your workloads to this structure drives practical decisions: where data may legally reside, how much latency users see, and how much redundancy you actually have. A service running in one AZ has no zone-level redundancy, whatever the architecture diagram implies.

Core AWS service categories and what they mean for on-call

Knowing the service families tells you where to look when an alert fires and who owns the fix.

  • Compute: EC2 for virtual machines, Lambda for event-driven functions. EC2 means patching and scaling are yours; Lambda shifts that to AWS but adds concurrency and timeout behavior to watch.
  • Storage: S3 for object storage. Bucket policies and access patterns are frequent incident causes.
  • Databases: RDS and Aurora for managed relational workloads, DynamoDB for NoSQL. Watch connection limits, throttling and failover events.
  • Networking: VPC with subnets, route tables, security groups and gateways. Many “the service is down” reports end in a routing or security group change.
  • Serverless and integration: Lambda, API Gateway, SQS and EventBridge, where failures often surface as retries and dead-letter queues.
  • Observability: CloudWatch for metrics, logs and alarms, CloudTrail for API activity.
  • Security: IAM for identities and permissions, plus encryption and key management.

The managed versus unmanaged trade-off is simple. Managed services cut operational overhead such as patching and capacity planning, but you give up some visibility into internals. Unmanaged resources give you control and the responsibility that comes with it. Choose with your monitoring and on-call ownership in mind, not only cost.

Regions, availability zones, and pricing: latency, compliance, and cost

Region choice is a three-way trade-off. AWS Well-Architected guidance says resource pricing varies by Region and recommends selecting Regions based on latency, data residency and cost, using the AWS Pricing Calculator to compare options.

  • Latency: place workloads near users and near the services they call most.
  • Compliance: data residency rules can fix the Region regardless of price.
  • Cost: compare Regions in the AWS Pricing Calculator before committing, and include data transfer in the estimate.

Data movement deserves its own line item. AWS guidance on selecting cost-effective Regions and AZs notes that these choices affect data transfer costs, performance and availability, and suggests minimizing inter-region transfers. Teams also tie cost control to automation. One example is a partner write-up on policy as code rules for cloud waste.

Pro Tip: Run non-production and test workloads in a lower-cost Region where compliance allows, and keep an eye on cross-region traffic so the savings are not erased by transfer charges.

IAM and CloudTrail: security notes for incident investigation

Identity problems look like application problems. A role loses a permission, an access key is rotated, a policy changes, and a service starts returning access denied. Start by identifying who or what acted: IAM users, roles, the policies attached to them, and any access keys in play. An access key is a credential pair that lets code or a person call AWS APIs, so an unexpected key is worth treating as a finding.

IAM role and policy gate block an API request

CloudTrail is the record of API activity, but it has caveats. AWS notes that CloudTrail does not capture IAM activity the way the IAM console shows it: IAM events are logged under the us-east-1 prefix, and event history covers 90 days unless you create a trail to persist logs.

Event history covers only 90 days, so create a trail if you need older evidence.

A reliable investigation combines sources, as AWS guidance on viewing IAM activity recommends:

  • IAM Access Advisor for last-accessed information per identity.
  • CloudTrail event history for recent API calls.
  • CloudWatch log groups for application and service logs.
  • Athena queries over CloudTrail logs for longer-term or complex analysis. Our write-up on AWS data query tools covers Athena and related query paths.

Where AWS incidents show up in practice

These scenarios come up repeatedly for operations teams.

  1. Serverless and API backends: a Lambda timeout or throttle shows up as elevated API latency. Check concurrency, downstream dependencies and recent deployments first, then confirm in CloudWatch.
  2. Partial AZ impairment: a fault confined to one zone can leave external uptime probes green, because probes in other AZs or Regions keep passing. AWS container guidance on recovering from AZ impairments in EKS reflects this. Compare health per zone and shift traffic away from the impaired one.
  3. Deployment regression: align the alert start time with the last rollout. If error rates jumped right after, roll back first and investigate second.
  4. IAM misconfiguration: find the denied call in CloudTrail, identify the role and policy, then compare against the last permission change.
  5. Managed database signals: failover events, connection exhaustion and storage pressure appear in RDS and Aurora metrics before applications complain.

Each scenario ends in a runbook action: verify per-AZ health, confirm the change window, roll back or fail over, and record the evidence.

Official learning paths, certifications, and community channels

AWS publishes a steady stream of guidance, and engineers do well to follow it.

  • AWS Blogs carry product announcements, technical deep-dives and operational guidance.
  • AWS Whitepapers and Guides collect architecture best practices, Well-Architected lenses and security guidance.
  • The AWS Cloud Solutions Architect Professional Certificate on Coursera covers EC2, S3, DynamoDB and CloudWatch and prepares learners for the Solutions Architect Associate exam.
  • The AWS Cloud Technology Consultant Professional Certificate adds hands-on labs across IAM, EC2, S3 and CloudFormation.
  • Community channels include r/aws on Reddit and AWS re:Post for practitioner answers. Check the official AWS service health page for status updates during an event.

The AWS shared responsibility model

AWS secures the cloud itself: the physical facilities, hardware, and the foundational software that runs the services. You secure what you put in the cloud: your data, identity and access configuration, network rules, operating system patching on EC2, and application code. The line shifts with the service. With EC2, you manage the guest operating system. With a managed service such as RDS or Lambda, AWS takes on more of the platform, but access policies, encryption choices and data classification remain yours.

For incident work, this model tells you who can fix what. A hardware or zone-level fault is AWS’s to resolve, though you still own the architecture that tolerates it. A public bucket, an overly broad IAM policy or an exposed access key is yours.

Practical steps that follow from the model:

  • Map each workload to the services it uses and write down which controls are yours.
  • Go beyond IAM basics: segment networks with VPC subnets and security groups, encrypt data at rest and in transit, and turn on monitoring for API activity.
  • Review permissions regularly with last-accessed data, not guesswork.

Typical AWS operational challenges and how to address them

Most recurring problems fall into a few buckets.

Fragmented evidence. Logs sit in CloudWatch, API history in CloudTrail, deployments in a CI/CD system, and alerts in a third tool. Answer: agree on a standard investigation path and keep trails and log retention beyond the 90-day event history.

Misleading health signals. Single external probes can pass during a confined AZ fault. Answer: monitor per zone and compare internal and external views.

Permission drift. Policies change faster than anyone remembers. Answer: use IAM Access Advisor to prune unused access, and tie permission changes to tickets.

Cost surprises. Cross-region transfer and idle resources inflate bills. Answer: estimate with the AWS Pricing Calculator, minimize inter-region traffic, and automate waste checks.

Unclear ownership. Teams disagree on whether an issue is the platform’s or the application’s. Answer: use the shared responsibility model to assign it, and document the result in the runbook.

Our guide to diagnosing Bedrock Agents and their Lambdas shows how this evidence-first approach applies to a specific runtime.

AWS with Kubernetes and CI/CD pipelines

Many teams run Kubernetes on AWS through Amazon EKS, which places cluster behavior inside the same Region and AZ structure as everything else. That matters during incidents: node groups, load balancers and persistent volumes are tied to zones, so a single-AZ impairment can degrade a cluster unevenly. AWS has published guidance on using EKS zonal shift and Istio to recover from AZ impairments.

CI/CD pipelines add the change dimension. Deployments, infrastructure as code runs and IAM role assumptions by pipeline identities all leave traces in CloudTrail, and they are usually the first suspect when behavior changes. Keep these habits:

  • Tag deployments with timestamps so you can line them up against alert start times.
  • Give pipeline roles narrow permissions and review them like any other identity.
  • Treat CloudFormation or Terraform runs as change events worth correlating with incidents.

The integration challenge is that cluster state, AWS resources and pipeline history live in separate tools. That gap is where investigation time disappears.

Opsphere: read-only context for AWS incident investigation

Opsphere does not replace AWS. We are the operational intelligence layer that connects AWS infrastructure and service context with Kubernetes, observability, deployments and engineering systems. Opsphere is read-only by default, and we help teams investigate AWS incidents with evidence from multiple systems at once.

  • Surface recent infrastructure changes next to a degrading service.
  • Correlate alerts with deployments, logs and metrics.
  • Investigate IAM or networking anomalies and unhealthy resources with aggregated evidence.

See how the Opsphere AWS integration works, review the platform, and compare plans on our pricing page when you are ready to evaluate a connected operational layer.

Opsphere

FAQ

What is AWS and what is its purpose?

AWS is a cloud platform that provides compute, storage, databases, networking and many other services on demand. Its purpose is to let teams run applications without owning physical infrastructure. AWS describes itself as the world’s most comprehensive and broadly adopted cloud.

What is an AWS access key?

An AWS access key is a credential pair used to sign API requests from code, scripts or the command line. Because it grants programmatic access, it should be rotated and scoped narrowly. Use IAM Access Advisor and CloudTrail to see what an identity has done, as AWS describes.

Is there an AWS down detector?

AWS publishes service health information, and community channels such as r/aws and AWS re:Post surface early reports. Third-party outage trackers exist, but external probes can miss faults confined to one AZ. Confirm with your own per-zone metrics.

How often does AWS have outages?

We do not cite an outage frequency here, because a reliable figure requires a primary source. What matters operationally is that faults are often scoped to one Availability Zone or Region, so design and monitor with that in mind.

Sources

Opsphere
Discuss Your AWS Operations
Contact Opsphere to discuss bringing AWS and other operational context together for incident investigation and cloud operations.

This article is provided for general informational purposes only and does not constitute professional, legal, security, or compliance advice. Please evaluate recommendations against your organization’s specific environment and requirements.