AWS Security Blog
Incident response guide for AWS CloudTrail investigations – Part 2
In Part 1 of this guide, we examined two common incident scenarios: cross-account Amazon Simple Storage Service (Amazon S3) data deletion with ransomware implications, and cryptocurrency mining deployed through AWS CloudFormation using exposed AWS Management Console credentials. We also introduced key incident response terminology and investigative frameworks for analyzing AWS CloudTrail events.
In this second part, we explore a more complex, multi-stage attack: how a web application vulnerability can cascade into credential harvesting and unauthorized access to Amazon Bedrock services across multiple AWS Regions. We also cover additional investigation techniques and hardening steps to strengthen your security posture.
Scenario 3: SSRF to IMDSv1 credential harvesting with multi-Region Amazon Bedrock service misuse
This scenario examines how a web application vulnerability can cascade into a multi-Region event targeting Amazon Bedrock services. The investigation demonstrates how threat actors chain together multiple techniques, using Amazon Elastic Compute Cloud (Amazon EC2) Instance Metadata Service version 1 (IMDSv1) through server-side request forgery (SSRF) and cross-Region pivoting to access Amazon Bedrock.
Your security team receives multiple alerts: failed AWS Identity and Access Management (IAM) operations in the us-east-1 Region, successful console sign-ins without multi-factor authentication (MFA), and unusual Amazon Bedrock API calls from us-east-2. Initially, these might seem like unrelated events across different services and Regions. However, as our Security Incident Response Team (SIRT) discovered, they represent a carefully orchestrated event chain that began with a web application vulnerability and culminated in unauthorized access to your organization’s AI infrastructure.
Architecture and progression
The architecture in figure 1 maps a multi-stage attack that exploits the trust relationship between Amazon Elastic Compute Cloud (Amazon EC2) instances and AWS services. A threat actor identified a server-side request forgery (SSRF) vulnerability in a web application running on an EC2 instance that had an attached webdev IAM role. Rather than attempting to escalate privileges directly, the threat actor used this foothold to reach the Instance Metadata Service version 1 (IMDSv1) endpoint and retrieve the temporary credentials issued to the webdev role. Because IMDSv1 returns credentials in response to a basic request with no session token, an SSRF flaw is enough to harvest them, which is why these credentials became the pivot point for everything that followed. The attack unfolded in five stages. Each stage is numbered in figure 1 so you can follow the progression from the initial web request through to the cross-Region Amazon Bedrock activity:
- Initial access: The threat actor exploited the SSRF vulnerability in the web application to make server-side requests on the instance’s behalf.
- Credential harvesting: Those requests reached the IMDSv1 endpoint and returned the temporary credentials for the
webdevrole. - Permission testing: Using the harvested credentials, the threat actor attempted IAM operations to probe the boundaries of what the role could do.
- Service pivoting: When IAM actions were denied, the threat actor shifted focus to Amazon Bedrock, a service the role could reach.
- Region hopping: The threat actor moved operations from us-east-1 to us-east-2, likely to evade Region-specific monitoring and access controls.
Figure 1: Scenario 3 architecture
CloudTrail evidence and structured extractions
In this section, we walk through the CloudTrail evidence that documents the attack from start to finish. Each of the four events that follow maps to one or more stages in the progression described previously, and together they trace how the threat actor moved from harvested credentials to active misuse of Amazon Bedrock. For each event, we present the relevant portion of the CloudTrail log record, highlight the fields that matter most for the investigation, and include a forensic legend that explains what each highlighted field reveals.
We cover the following events:
- Permission boundary testing (15:53:49 UTC): A failed
CreateUsercall in us-east-1 that reveals the compromised role and the IMDSv1 credential source. - Console access establishment (15:59:29 UTC): A successful console sign-in without MFA, showing the pivot from programmatic to interactive access.
- Bedrock service reconnaissance (17:20:00 UTC): A
ListFoundationModelscall in us-east-2 that marks the Region hop and the shift to AI services. - Active model exploitation (17:25:48 UTC): A
Conversecall that invokes the Amazon Nova Pro model, confirming unauthorized usage.
As you read each event, focus on how the fields connect one stage to the next. The same webdev role, the same source IP address, and the recurring ec2RoleDelivery value are the threads that tie these otherwise separate events into a single attack chain.
Event 1: Permission boundary testing (15:53:49 UTC): The first suspicious activity appeared as a failed CreateUser API call in us-east-1. The CloudTrail log records an AssumedRole session attempting to create an IAM user named adm1n but receiving an AccessDenied error. The webdev role is visible in the userIdentity field, readOnly is false (indicating a write operation attempt), and the user-agent shows AWS Command Line Interface (AWS CLI) on Windows, suggesting programmatic access from the harvested credentials.
Event 2: Console access establishment (15:59:29 UTC): Six minutes later, the threat actor successfully signed in to the AWS Management Console using the same credentials. The ConsoleLogin event records that MFA wasn’t used (MFAUsed: No), and the source IP (75.3.231.105) provides attribution data. The user agent indicates Chrome browser on Windows 10.
Event 3: Amazon Bedrock service reconnaissance (17:20:00 UTC): Nearly two hours later, the threat actor pivoted to Amazon Bedrock, making a ListFoundationModels API call in us-east-2. This event exhibits several patterns: a Region change from us-east-1 to us-east-2 (potential defense evasion), a shift from IAM to AI services, readOnly: true (reconnaissance rather than modification), and sessionCredentialFromConsole: “true”, which ties the call to the console session established in Event 2 rather than a fresh IMDSv1 credential retrieval.
Event 4: Active model exploitation (17:25:48 UTC): Five minutes after the reconnaissance call, the threat actor moved from enumeration to active exploitation, invoking the Amazon Nova Pro model through the Converse API in us-east-2. The additionalEventData field quantifies the unauthorized usage at 944 input tokens and 126 output tokens, confirming that the threat actor successfully prompted the model and received a response.
Notable event fields to track
As you review the event logs, a handful of fields do most of the investigative work in this scenario. Understanding what each one reveals, and why it matters, is what turns a collection of individual log records into a coherent attack narrative.
The userIdentity field is the starting point for attribution. In this scenario it carries the EC2 instance ID as the session name, which is what let us trace the harvested credentials back to a specific compromised instance rather than a human user. Whenever you see an assumed-role session, this field answers the first question of any investigation: whose credentials are these, and where did they come from?
The readOnly field reveals the intent behind an action. A value of true marks reconnaissance, such as the ListFoundationModels call the threat actor used to enumerate available models, while false marks an attempt to change or use something, such as the CreateUser call or the Converse invocation. Sorting events by this field quickly separates the threat actor’s information gathering from the actions that caused actual impact.
The awsRegion field is easy to overlook, but in this scenario it exposed the threat actor’s evasion strategy. The shift from us-east-1 to us-east-2 wasn’t incidental; threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them. Watching this field helps you spot activity that has deliberately moved away from where your detection is strongest.
Finally, the userIdentity.invokedBy field identifies when an AWS service, rather than a user or a set of harvested credentials, made the request on your behalf. CloudTrail populates it only when the caller is an AWS service, such as through a service-linked role, a service role, or a forward access session. It doesn’t appear in the events for this scenario because the threat actor called Amazon Bedrock directly with the harvested webdev credentials. That absence is itself informative: it confirms the requests came from a principal acting on its own rather than from a legitimate service-driven workflow. As agent-based and service-integrated Amazon Bedrock workloads become more common, checking this field separates expected service activity from credentials driven directly by a threat actor.
Investigation priorities
With the full attack chain mapped, from SSRF through credential harvesting to Amazon Bedrock service misuse, the investigation turned to a harder question: what did each stage actually cost us, and what would stop it from happening again? A few priorities shaped that work.
The first was figuring out where the credentials came from and how far the exposure reached. It was clear the threat actor had valid credentials for the webdev role, but the more useful question was why a web application role could reach Amazon Bedrock at all. The customer confirmed there was no business reason for it, so we needed to understand whether that permission was a deliberate misconfiguration or an oversight, and then look for other EC2 instances carrying the same role attachment. One compromised instance is an incident; a fleet of instances with the same over-scoped role is a much bigger problem waiting to happen.
Next, we wanted to know what the threat actor did after they got into Amazon Bedrock. Reconnaissance and active use carry very different consequences, so we traced which foundation models were touched and whether any were actually invoked or only enumerated. That distinction matters for scoping the damage, and it signals whether data exfiltration is a concern. Unusual model usage, unexpected prompt volume, or output patterns that don’t match any legitimate workload are the signals that reconnaissance has turned into something worse.
The Region hop was its own line of inquiry. The move from us-east-1 to us-east-2 was almost certainly deliberate, and the investigation focused on understanding what the threat actor gained by it. In practice, that meant comparing the two Regions: were the monitoring and access controls in us-east-2 weaker than in us-east-1, and what else did the threat actor reach in the secondary Region once they were there? Inconsistent controls across Regions are one of the most common ways activity slips past detection.
Tying it all together was the timeline, which shows how quickly the threat actor moved through the chain:
- 15:53:49: Failed IAM operation (us-east-1)
- 15:59:29: Successful console login (us-east-1)
- 17:20:00: Amazon Bedrock reconnaissance (us-east-2)
- 17:25:48: Active model invocation (
Conversecall) (us-east-2)
Following the credentials across those events fills in the rest of the story. IMDSv1 handed the threat actor temporary credentials for the webdev role, and the same role appears in every event that followed, which confirms the credentials were reused rather than replaced. Nowhere in that sequence was MFA required, and that single gap is what let one harvested credential stay useful across two hours, two Regions, and two very different services.
Incident response checklist
The following checklist captures the actions needed to contain the incident, remediate the vulnerability, and assess the scope of unauthorized AI service usage. Each item names where to look and what a finding looks like, so the checklist stays usable under the time pressure of a live incident.
- Contain and remediate the entry point:
- Identify the specific web application feature that made the outbound request (URL fetchers, webhook callbacks, PDF or image renderers, and link-preview generators are the usual culprits), then confirm it can reach
http://169.254.169.254. - Audit the rest of the application for the same pattern, because one unvalidated URL parameter usually means others exist.
- Enforce IMDSv2 on the affected instance and across the fleet with
aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1. Setting--http-tokensrequired means credentials are only returned when the caller presents a session token it obtained through a PUT request, which a basic SSRF cannot do. Setting the hop limit to1keeps the metadata response on the instance itself, so a request coming from a container or proxy an extra hop away never receives it.
- Identify the specific web application feature that made the outbound request (URL fetchers, webhook callbacks, PDF or image renderers, and link-preview generators are the usual culprits), then confirm it can reach
- Scope the Bedrock usage:
- List the foundation models the
webdevrole could reach by reviewing its IAM policy and any resource-based policies, so you know the full set of models that were exposed, not only the one that was invoked. - Determine what was sent to and returned by the model. CloudTrail records the Converse call and the token counts, but only Amazon Bedrock model invocation logging captures the input prompts and model responses. If it was enabled, pull the log entries for the session; if it wasn’t, note that the prompt and response content can’t be recovered and enable it now.
- Flag any compliance exposure based on what those prompts and responses contained. Unauthorized processing of regulated data (such as personally identifiable information (PII), protected health information (PHI), or cardholder data) through the model might trigger notification obligations.
- List the foundation models the
- Check for wider compromise and persistence:
- Query CloudTrail across all Regions and services—not only Amazon Bedrock—for every event tied to the
webdevrole’s session, to confirm what else the same credentials touched. - Correlate the CloudTrail timestamps with VPC Flow Logs and application logs for source IP
75.3.231.105to build the network-level picture around each API call. - Search for IAM write events from the session (
CreateUser,CreateRole,CreateAccessKey, andAttachRolePolicy) that indicate an attempt to establish persistence beyond the temporary credentials. The failedadm1nCreateUsercall is the known starting point; confirm nothing similar succeeded.
- Query CloudTrail across all Regions and services—not only Amazon Bedrock—for every event tied to the
- Watch for ongoing or hidden impact:
- Review Amazon Bedrock usage in CloudWatch and your billing data for invocation spikes or unexpected token consumption that fall outside normal workload patterns.
- Inspect the invocation logs for signs of sensitive data being processed or extracted through the model.
- Check the same logs for prompt injection attempts, where the input tries to override the model’s instructions or extract system prompts.
Key takeaways
This scenario reveals how a single application vulnerability can cascade into broad unauthorized access when multiple security controls are missing. The following takeaways highlight the key defensive gaps and hardening priorities.
- Least-privilege IAM for workload roles: The
webdevrole’s access to Amazon Bedrock across multiple Regions had no business justification for a web application workload, which the customer confirmed during the investigation. Apply least-privilege principles to EC2 instance roles by scoping permissions to only the services and actions the application requires. Use AWS IAM Access Analyzer to identify unused permissions and tighten policies proactively. Overly permissive roles transform a single application vulnerability into broad lateral movement across unrelated services. - IMDSv1 compared to IMDSv2: Organizations must immediately switch to IMDSv2 and disable IMDSv1 across their entire cloud infrastructure. The
ec2RoleDelivery: “1.0"field in the logs explicitly confirms the use of IMDSv1, which permits credential retrieval without an authentication token. This architectural weakness makes SSRF-based credential theft trivial, because a web application flaw that can make an outbound request is enough to read the role’s temporary credentials with no further authentication. Transitioning to IMDSv2 mitigates this attack surface by enforcing local, session-based tokens, effectively breaking the threat actor’s exploitation chain. In this scenario, IMDSv2 alone would have stopped the attack at its first step. - Region-based defense evasion signals a deliberate operator: The shift from us-east-1 to us-east-2 for Amazon Bedrock access wasn’t incidental. Threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them, and activity in a secondary Region is more likely to go unnoticed. This kind of cross-Region movement is a marker of operational security awareness rather than opportunistic access, and it should raise the priority of an investigation. Treat consistent detection coverage across all Regions, including the ones you do not actively use, as a baseline requirement.
- Interface switching and permission probing reveal the threat actor’s method: This event chain reveals a threat actor comfortable moving between AWS interfaces and testing boundaries before committing. The failed
CreateUserattempt was systematic probing to understand the scope of the harvested credentials, and when IAM actions were denied, the threat actor pivoted to a service the role could actually reach. The combination of programmatic access through the AWS CLI and interactive console access demonstrates the same adaptability. Recognizing this pattern of probe, adapt, and pivot helps responders anticipate the next move instead of reacting to each event in isolation. - AI services need visibility beyond CloudTrail: Amazon Bedrock and other AI services are high-value targets, and CloudTrail alone doesn’t capture the whole story. CloudTrail records who called Amazon Bedrock and whether the call succeeded, but not what was asked or answered. Enable Amazon Bedrock model invocation logging to capture full prompts and responses for compliance auditing. For agent-based workloads, Amazon Bedrock AgentCore Observability, built on AWS Distro for OpenTelemetry (ADOT), provides session-level traces showing tool execution order and latency. Consider also enabling Amazon GuardDuty AI Protection, which analyzes Amazon Bedrock-related CloudTrail activity to detect anomalous invocations, cost harvesting, and prompt injection attempts. Correlating these signals—CloudTrail, Model Invocation Logging, and agent telemetry—gives investigators the complete picture. For implementation guidance, see Monitoring and Auditing AI Workloads on AWS.
Advanced forensic indicators and evasion techniques
Beyond the specific attack patterns in this scenario, investigators should be aware of several evasion techniques that threat actors use to confuse defenders and blend into legitimate activity. The top three that we observe across incident response with customers are:
- Root user compared to IAM user named root: When you first create an AWS account, you begin with a single sign-in identity that has complete access to all AWS services and resources in the account. This identity is called the AWS account root user. In some previous investigations, threat actors have also created IAM users in an AWS account named
root. The difference is visible in the type field of theuserIdentityelement of the CloudTrail log record, which indicates the type of user that logged the record. - Role and user name imitation: Threat actors attempt defense evasion by creating names for IAM users and roles that imitate those reserved for use by AWS. For example, the service-linked role
AWSServiceRoleForSupportis a unique IAM role linked directly to AWS Support. Threat actors have created roles with the nameAWSServiceRoIeforSupport(note the use of an upper-case letterIinstead of a lower-case letterlinRole) in an attempt to trick users into thinking actions taken by this role have been performed by AWS Support. - Users named HIDDEN_DUE_TO_SECURITY_REASONS: The
userNamefield contains the stringHIDDEN_DUE_TO_SECURITY_REASONSwhen the recorded event is a console sign-in failure caused by incorrect user name input. CloudTrail doesn’t record the contents in this case because the text could contain sensitive information. However, threat actors have used this string as an actual username to trick investigators into thinking the name has been obfuscated. This technique is usually associated with a correspondingCreateUserorCreateRoleCloudTrail event.
Conclusion and next steps
CloudTrail event fields help security teams identify identities with unintended access, track threat actor actions, and remediate affected resources. Understanding fields like userIdentity, eventName, and sourceIPAddress improves incident investigation and threat detection. Implementing best practices such as enabling comprehensive logging, using Amazon Athena for analysis, securing logs, and automating responses helps ensure that CloudTrail serves as a robust forensic and incident response tool.
If you suspect unauthorized activity in your AWS environment, AWS Security Incident Response is available to help. The service continuously monitors and triages findings from Amazon GuardDuty and third-party security tools integrated through AWS Security Hub, automatically filtering alerts to surface the most relevant events. In addition to proactive triage, customers can initiate security cases through the service. You can choose to handle these cases internally or receive support from the Security Incident Response Team (SIRT), a dedicated group of security experts available at all times to assist with investigation, containment, and recovery throughout the incident lifecycle.
Additional resources
The following resources provide further guidance on securing your AWS environment and strengthening your investigative capabilities.
- AWS Security Best Practices
- The Agentic AI Security Scoping Matrix: A framework for securing autonomous AI systems
- CloudTrail Log Event Reference
- AWS Security Incident Response Guide
If you have feedback about this post, submit comments in the Comments section below.