Extend Amazon CloudWatch Beyond Native Connectors with Cribl Stream

Teams running hybrid or multi-cloud estates collect telemetry from proprietary appliances, on-premises Application Performance Monitoring (APM) tools, home-grown agents, and Apache Kafka streams that all must be parsed, normalized, and routed before it’s useful. Amazon CloudWatch lets you collect logs from 60+ AWS services, 20+ third-party tools, and your own applications using CloudWatch pipelines, HTTP endpoints, and agents. CloudWatch pipelines also transform and standardize logs into formats such as Open Cybersecurity Schema Framework (OCSF), with no infrastructure to manage.

Cribl is an AWS Partner offering an AI Platform for Telemetry. Its observability pipeline, Cribl Stream, lets customers ingest logs from hundreds of upstream sources, normalizes them to OCSF or OpenTelemetry (OTel), and writes directly into a CloudWatch log group through the PutLogEvents API. Because the data lands alongside your AWS-native and application telemetry, you can query everything together in Amazon CloudWatch Logs Insights from a single place.

In this post, you will learn how to:

• Identify which sources CloudWatch covers natively and where Cribl Stream extends that coverage.
• Design an architecture that unifies custom telemetry with AWS-native logs.
• Configure the integration without writing custom transformation code.
• Apply the pattern across security, platform engineering, and operations.

Before you begin, make sure you have the following in place:

• An AWS account with permissions to create Amazon CloudWatch log groups and AWS Identity and Access Management (IAM) roles.
• IAM permissions for Cribl Stream to write logs: logs:CreateLogStream, logs:PutLogEvents, and logs:DescribeLogStreams
• A Cribl Stream deployment, or the ability to launch a Cribl.Cloud managed (SaaS) subscription, or a self-managed deployment from AWS Marketplace.
• For self-managed deployments, an Amazon VPC with private subnets and outbound internet access, or a CloudWatch Logs VPC endpoint (com.amazonaws.logs).
• The AWS Command Line Interface (AWS CLI) installed and configured for the log group and IAM steps in this post

How Cribl Stream extends CloudWatch source coverage

Cribl Stream sits between upstream systems and CloudWatch Logs. It offers hundreds of source integrations, including syslog, Apache Kafka, file-based collectors, APM tools, security agents, and many other HTTP, TCP and UDP-based sources, along with OCSF and OTel-aware pipeline steps that normalize events without custom code. It also lets you replay raw data for use cases such as audits, security events, and investigations.

Cribl Stream’s built-in CloudWatch Logs destination writes normalized events directly into a log group through the PutLogEvents API. There is no intermediate storage, no SQS queue, and no Lambda layer. A persistent queue inside Cribl Stream protects against transient CloudWatch Logs failures. Once the data lands in the CloudWatch log group, engineers query it through CloudWatch Logs Insights exactly as they would query any other CloudWatch log group using the same AWS console, same query languages, same retention and access controls.

The net effect is that Cribl Stream replaces the custom Lambda functions with a configured path that handles parsing, normalization, and delivery, while CloudWatch remains the single destination and query surface.

Architecture

The flow has three layers: Upstream systems, Cribl Stream, and Amazon CloudWatch Logs.

Figure 1: Cribl Stream normalizes upstream telemetry and writes directly into a CloudWatch log group through PutLogEvents; CloudWatch Logs Insights provides the query layer.

Cribl Stream runs on an Amazon Elastic Compute Cloud (Amazon EC2) instance launched from AWS Marketplace or as a container in your Amazon Virtual Private Cloud (Amazon VPC). It receives events from upstream systems, normalizes each one to OCSF or OTel inside a Cribl pipeline, and writes to a CloudWatch log group.

Amazon CloudWatch Logs is the destination. Each logical upstream source writes to its own log group (for example, /apps/any-company-firewall), which keeps retention, IAM access, and metric filters manageable per source. Amazon CloudWatch Logs Insights provides the query surface, so you can write queries in Logs Insights QL, OpenSearch PPL, or SQL, and build dashboards and alarms on the results.

Design tip: Use a consistent log group naming convention (for example, /<common-key>/<vendor>-<source>) so that Logs Insights queries and IAM policies scale as more sources onboard.

Cost & migration considerations: Cribl-delivered data is priced as custom logs in CloudWatch. For infrequently accessed data (archival, compliance, or low-query security logs), the Infrequent Access log class offers reduced ingestion costs. If a source routed through Cribl later gains a native CloudWatch connector, you can seamlessly migrate it to CloudWatch pipelines.

Configuration highlights

The integration has four components: a Cribl Stream deployment, a source configuration, a CloudWatch log group with an IAM role, and the Cribl Stream CloudWatch Logs destination. OCSF or OTel normalization inside Cribl itself is out of scope for this post, so follow the Cribl public documentation for mapping guidance.

Deploy Cribl Stream from AWS Marketplace

Cribl Stream offers both a managed SaaS option (Cribl.Cloud Suite) and a self-managed deployment option. For environments where telemetry cannot leave the VPC, launch self-managed Cribl in private subnets with outbound access to the CloudWatch Logs VPC endpoint (com.amazonaws.<region>.logs).

Configure the source in Cribl Stream

Cribl Stream’s source configuration defines how data arrives. Common shapes are syslog/TCP/UDP for appliances, HTTP event collector for agents that push JSON, and Apache Kafka for high-volume streams that need back-pressure and replay. Keep one source per upstream system to simplify routing and troubleshooting.

To add and configure a source, follow the Cribl Sources documentation. For a specific source type, see its page under Syslog, or the equivalent page for an HTTP event collector and so on.

Create the CloudWatch log group and IAM role

Create one landing log group per source, for example, /apps/any-company-firewall  and set up a retention policy that aligns with your security and compliance requirements. For feeds you access infrequently, set the log group’s log class to Infrequent Access at creation time to reduce ingestion costs:

aws logs create-log-group \ 
--log-group-name /apps/any-company-firewall \ 
--log-group-class STANDARD \ 
--tags cw:datasource:name=any-company,cw:datasource:type=firewall 

You can replace the –log-group-class value to INFREQUENT_ACCESS for infrequently accessed data to reduce the ingestion cost.

Adding the cw:datasource:name and cw:datasource:type tags at creation registers the log group as a recognized data source in CloudWatch Logs Management. This enables service-centric discovery, schema-aware querying, and consistent categorization of Cribl-managed log groups alongside AWS and third-party sources in the console.

Next, attach an IAM role to the Cribl Stream instance with write permissions on the log group:

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:<region>:<account-id>:log-group:/apps/any-company-firewall:log-stream:*"
}]
}

Scope the resource ARN to the /apps/any-company-firewall:log-stream:* prefix so Cribl Stream cannot write outside its assigned log groups.

Configure the Cribl Stream CloudWatch Logs destination

In the Cribl UI, create a CloudWatch Logs destination with the target Region, log group, a templated log stream name (for example, ${host}-${worker}), IAM-role authentication, and JSON format. Turn on Cribl’s persistent queue so that transient CloudWatch Logs failures do not drop events. If event volume is high, enable compression and tune batch size to stay under the PutLogEvents quota.

Validate through CloudWatch Logs Insights

Once data is flowing, open the CloudWatch console and run a Logs Insights query against the landing log group to confirm OCSF or OTel fields are parsed correctly:

fields @timestamp, class_uid, src_endpoint.ip, dst_endpoint.ip, dst_endpoint.port
| sort @timestamp desc
| limit 50

The same log group is now available to dashboards, metric filters, and alarms.

Query across data sources

Because data delivered by Cribl lands in CloudWatch Logs alongside native AWS logs, a single Amazon CloudWatch Logs Insights query can span both. The pattern is to select every relevant log group in the console (or pass them together on StartQuery) and use coalesce to read across schema differences. The built-in @log field identifies which source each row came from.

Trace a suspicious IP across firewall and Amazon VPC Flow Logs. Run across the firewall log group delivered by Cribl (for example, /apps/any-company-firewall) and the Amazon VPC Flow Logs log group:

fields @timestamp, @log,
coalesce(src_endpoint.ip, srcaddr) as src_ip,
coalesce(dst_endpoint.ip, dstaddr) as dst_ip,
coalesce(dst_endpoint.port, dstport) as dst_port,
coalesce(activity_name, action) as action
filter src_ip = "203.0.113.42" or dst_ip = "203.0.113.42"
| sort @timestamp desc
| limit 100

OCSF fields from the firewall feed and native VPC Flow Log fields line up in the same result set. Two practical notes: coalesce is a fallback for environments where some sources are OCSF, and others are native AWS. When every source is normalized to OCSF, the native field paths never trigger, but the query still works. Also remember that VPC NAT translates a public Elastic IP to the elastic network interface’s private IP before the flow record is written, so an external 203.0.113.42 typically only appears in the firewall feed; pivot through the matching private IP to see the same activity in VPC Flow Logs.

Surface failed sign-ins across an identity provider and AWS CloudTrail. Run across the Cribl-delivered identity log group (for example, /apps/any-company-idp) and the AWS CloudTrail log group:

fields @timestamp, @log,
coalesce(actor.user.name, userIdentity.userName) as user,
coalesce(activity_name, eventName) as action,
coalesce(status, errorCode) as status
| filter action like /(?i)logon|login|signin|consoleLogin/and status like /(?i)fail/
| stats count() as failures by user, bin(5m)
| sort failures desc

Distributed failed-auth patterns that only appear when you correlate the identity provider with CloudTrail surface in a single result. The regex covers both the OCSF Logon activity name and native CloudTrail ConsoleLogin events. The status field’s behavior is OCSF mapping, since some pipelines populate it from the source’s errorCode, while others leave it null on success-only events. If failed sign-ins are missing in your environment, validate first that status (or unmapped.errorCode) is populated for the events you expect to see.

Attribute application traffic to an IAM identity

The cross-source pattern is most powerful when an actor’s behavior touches all three layers: a web application, an AWS API, and the underlying network. The example below simulates that scenario end to end. It uses a Cribl Stream Datagen source to emit Apache access logs, the AWS Command Line Interface (AWS CLI) to generate AWS CloudTrail events, and Amazon VPC Flow Logs from a single Amazon EC2 instance. All three feeds land in Amazon CloudWatch Logs in OCSF, so a single Amazon CloudWatch Logs Insights query attributes the entire chain back to one Amazon EC2 instance role.

The simulation needs three things: a fixed actor public IP (here, 198.51.100.27, the Elastic IP attached to a t3.micro EC2 instance in a dedicated VPC), Cribl Stream emitting weblogs with the same client IP, and AWS CLI commands run from inside the EC2 so CloudTrail records src_endpoint.ip = 198.51.100.27 and the instance role as the identity.

Generate AWS CloudTrail events from the actor’s IP

From inside the EC2 instance, we used AWS Systems Manager Session Manager to run a small set of read-only AWS CLI commands. Each call appears in CloudTrail with src_endpoint.ip = 198.51.100.27 and actor.user.uid set to the instance role’s session ARN:

aws sts get-caller-identity --region us-east-1
aws iam list-users --region us-east-1
aws s3api list-buckets --region us-east-1
aws ec2 describe-instances --region us-east-1 --max-items 3
aws kms list-keys --region us-east-1 --max-items 3
aws secretsmanager list-secrets --region us-east-1 --max-items 3
aws cloudtrail describe-trails --region us-east-1

The same activity also produces VPC Flow Log records from the EC2’s elastic network interface (private IP 10.20.1.91 in the example below) outbound to AWS API endpoints on TCP port 443.

Query across all three log groups

In the CloudWatch console, select the Cribl-delivered weblog log group (/apps/web-logs), the AWS CloudTrail log group, and the Amazon VPC Flow Logs log group, then run:

fields @timestamp, @log,
src_endpoint.ip, http_request.url.path, http_response.code,
api.operation, api.service.name,
dst_endpoint.port, traffic.bytes, actor.user.uid 
| filter src_endpoint.ip = "198.51.100.27" or src_endpoint.ip = "10.20.1.91"
| sort @timestamp desc 
| limit 200

Because all three feeds are normalized to OCSF, the join collapses to a single field filter and no coalesce needed across schemas. The @log column attributes each row to its source. Sample output (truncated):

The actor.user.uid field on the CloudTrail rows carries the IAM principal that issued each call, which in this simulation is the EC2 instance role attached through an instance profile. The result is a single timeline that ties simulated user requests at the application layer to the AWS APIs they trigger and to the network egress that follows, all attributed to one source IP and one IAM identity. The same query, with native CloudTrail and Amazon VPC Flow Log fields, would require coalesce across srcaddr, sourceIPAddress, and src_endpoint.ip and a separate parser for the weblog stream, overhead that OCSF normalization removes.

Agentic access through the Amazon CloudWatch MCP server

Amazon CloudWatch Logs Insights serves analyst-driven investigations. For agentic and conversational workflows, the same log groups are also accessible through the Amazon CloudWatch MCP server, which serves as a unified interface for interacting with CloudWatch’s monitoring and observability tools through the Model Context Protocol (MCP). MCP is the open standard owned by Linux Foundation for connecting Large Language Models (LLMs) to contextual data sources. The CloudWatch MCP server enables alarm-based incident response, alarm recommendation, metric and log analysis, and log pattern detection. MCP-compatible AI assistants such as Kiro, Claude Code, and GitHub Copilot integrate with the server to enable natural language interactions with your observability data.

For the OCSF-normalized streams that Cribl Stream writes into CloudWatch Logs, this means the same data backs both access patterns. A security analyst can run the Amazon CloudWatch Logs Insights query from the previous section, and an on-call engineer working in an MCP-compatible AI assistant or an AI Agent itself can ask, “Summarize failed sign-ins from the identity provider feed and AWS CloudTrail in the last hour, grouped by user,” and receive the same correlation in conversational form. Agentic workflows, including automated runbooks, AI-driven incident triage, and chat-based investigations, consume the Cribl-delivered telemetry without any additional integration work.

To enable the agentic path, configure the CloudWatch MCP server in an MCP-compatible client and grant the minimum read-only AWS Identity and Access Management (IAM) permissions on the relevant log groups. The CloudWatch MCP server repository documents the required permissions and client configuration.

Clean up

If you deployed resources while following the configuration steps in this post, remove them to avoid ongoing charges.

• Delete the Cribl Stream deployment: If you subscribed to Cribl.Cloud Suite, cancel the subscription through the AWS Marketplace console. If you launched a self-managed EC2 instance from AWS Marketplace:
  • Terminate the EC2 instance.
  • Delete the associated security group.
  • Delete the key pair.
• Delete the CloudWatch log groups: (Warning: Deleting log groups permanently removes all log data. If you need to retain logs for compliance or audit purposes, export them to Amazon S3 before deleting them).
  • In the Amazon CloudWatch console, navigate to Log groups.
  • Select each /apps/* log group created during the walkthrough.
  • Choose Delete log group.
• Delete the IAM role to prevent unused permissions from persisting in your account:
  • In the IAM console, navigate to Roles.
  • Select the role you created for the Cribl Stream instance.
  • Choose Delete.
• Delete the policy (if created):
  • If you created a standalone policy, navigate to Policies in the IAM console.
  • Select the policy.
  • Choose Delete.
• Delete the VPC endpoint (if created): If you added a CloudWatch Logs VPC endpoint for private subnet access, delete it through the Amazon VPC console under Endpoints.

Conclusion and next steps

CloudWatch’s pre-built connectors cover most AWS and mainstream third-party telemetry. Cribl Stream extends coverage to the long tail, proprietary, on-premises, multi-cloud and custom sources by handling parsing and OCSF normalization upstream and writing directly into a CloudWatch log group. CloudWatch Logs Insights then provides a single query surface across native and data delivered by Cribl.

To get started:

• Review the Amazon CloudWatch pipelines documentation and confirm which of your sources are already covered natively before adding a Cribl Stream path.
• Read the Cribl Stream CloudWatch Logs destination documentation for destination settings, authentication, and tuning.
• Subscribe to Cribl.Cloud Suite on AWS Marketplace for a managed SaaS deployment option of Cribl Stream.
• Explore the AWS Observability Best Practices Guide for broader telemetry pipeline patterns.
• Set up the Amazon CloudWatch MCP server to extend the log groups to MCP-compatible AI assistants for natural language and agentic workflows.