Centralized CloudTrail monitoring across 100+ AWS accounts

Organizations running workloads across dozens or hundreds of AWS accounts face a common challenge: centralized security monitoring at scale. Security teams need to search through hundreds of gigabytes of AWS CloudTrail logs daily to detect threats and satisfy compliance requirements for SOC 2, PCI DSS, and HIPAA audits. They also need to provide role-based access to multiple teams with different responsibilities.

Without purpose-built infrastructure, this often involves manual log searching that takes hours and compliance report generation that takes days. It also leads to fragmented code bases of custom AWS Lambda functions managing index lifecycles across environments. A single shared search domain without consistent access control compounds the problem further.

In this post, we show you how to build a centralized CloudTrail monitoring solution on Amazon OpenSearch Service. Terraform manages the full stack, from domain provisioning to access control and lifecycle policies. The solution handles 200 GB/day of CloudTrail logs, provides automated threat detection alerts, and gives 4 different teams isolated, role-appropriate access to the data.

Solution overview

The following diagram shows the architecture. CloudTrail logs flow from 100+ AWS accounts through an organization trail into a centralized S3 bucket. Amazon Simple Queue Service (Amazon SQS) notifications trigger the OpenSearch Ingestion pipeline. The pipeline auto-scales between 2 and 10 OpenSearch Compute Units (OCUs) to parse and index the logs into the OpenSearch domain. Four team-specific roles access the data through OpenSearch Dashboards with tenant isolation.

The key components are:

• CloudTrail aggregation. An organization trail sends logs from 100+ accounts into a centralized Amazon Simple Storage Service (Amazon S3) bucket.
• Ingestion. An Amazon OpenSearch Ingestion pipeline picks up new logs through Amazon SQS notifications on the S3 bucket. It automatically scales between 2 and 10 OCUs based on queue depth. Throttling on lower-environment queues prevents development and test spikes from starving production ingestion.
• Amazon OpenSearch Service domain. 6 or1.4xlarge data nodes (OpenSearch Optimized instances) with 3 dedicated r8g.large master nodes, fine-grained access control, encryption at rest, and node-to-node encryption.
• Infrastructure as code. Index templates, Index State Management (ISM) policies, roles, role mappings, tenants, alerting monitors, and dashboards are all declared in Terraform and applied consistently across environments.

Prerequisites

To implement this solution, you need the following:

• An organization in AWS Organizations with CloudTrail enabled across member accounts.
• Terraform v1.5+ with the AWS provider and the OpenSearch provider.
• A virtual private cloud (VPC) with private subnets for the OpenSearch domain.
• IAM roles for each team that will access the OpenSearch domain.
• An Amazon Simple Notification Service (Amazon SNS) topic for security alert notifications.
• Familiarity with Amazon OpenSearch Service, Terraform, and AWS CloudTrail.
• Sample Terraform code is available in the GitHub repository

Implementation

This section walks through the Terraform code for each component of the solution, starting with the workload profile that informed our sizing decisions.

Workload profile

Before sizing the cluster, we defined the workload characteristics and SLAs for the centralized CloudTrail monitoring platform:

This is a write-heavy ingestion workload. The primary use case is automated alerting and periodic compliance queries rather than continuous interactive search. This workload profile informed the decision to use OR1 (storage-optimized) instances with zero replicas, prioritizing indexing throughput over search parallelism.

Domain provisioning

Start by provisioning the Amazon OpenSearch Service domain with encryption, fine-grained access control, and VPC placement:

resource "aws_opensearch_domain" "cloudtrail" {
  domain_name    = var.domain_name
  engine_version = "OpenSearch_3.3"

  cluster_config {
    instance_type          = "or1.4xlarge.search"
    instance_count         = 6
    zone_awareness_enabled = true
    zone_awareness_config {
      availability_zone_count = 3
    }
  }

  dedicated_master_config {
    dedicated_master_enabled = true
    dedicated_master_type    = "r6g.large.search"
    dedicated_master_count   = 3
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 500
    iops        = 3000
    throughput  = 125
  }

  encrypt_at_rest { enabled = true }
  node_to_node_encryption { enabled = true }

  domain_endpoint_options {
    enforce_https       = true
    tls_security_policy = "Policy-Min-TLS-1-2-PFS-2023-10"
  }

  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = false
    master_user_options {
      master_user_arn = var.master_user_arn
    }
  }

  vpc_options {
    subnet_ids         = var.vpc_subnet_ids
    security_group_ids = var.vpc_security_group_ids
  }

  tags = {
    Environment = "production"
    Project     = "centralized-cloudtrail-monitoring"
    ManagedBy   = "terraform"
  }
}

This solution was built on OR1 instances, which are storage-optimized and use Amazon Elastic Block Store (Amazon EBS) (gp3 or io1) for local storage, with data copied synchronously to Amazon S3 as it arrives. This storage structure provides increased indexing throughput because indexing is performed exclusively on primary shards. Replicas are backed by Amazon S3 through segment replication, eliminating the CPU overhead of document replication on replica nodes. For new deployments, we recommend OR2 instances, which offer up to 26% higher indexing throughput compared to OR1 while maintaining the same storage-optimized architecture.

Ingestion pipeline

The Amazon OpenSearch Ingestion pipeline provides serverless, auto scaling ingestion from Amazon S3 into the OpenSearch domain. It picks up new CloudTrail logs through Amazon SQS notifications on the centralized S3 bucket and scales between 2 and 10 OpenSearch Compute Units (OCUs) based on queue depth:

resource "aws_iam_role" "osis_pipeline" {
  name = "cloudtrail-osis-pipeline-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "osis-pipelines.amazonaws.com" }
    }]
  })
}

resource "aws_iam_policy" "osis_pipeline" {
  name = "cloudtrail-osis-pipeline-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["s3:GetObject", "s3:ListBucket"]
        Effect   = "Allow"
        Resource = [var.cloudtrail_bucket_arn, "${var.cloudtrail_bucket_arn}/*"]
      },
      {
        Action   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Effect   = "Allow"
        Resource = var.cloudtrail_sqs_queue_arn
      },
      {
        Action   = ["es:DescribeDomain", "es:ESHttp*"]
        Effect   = "Allow"
        Resource = "${aws_opensearch_domain.cloudtrail.arn}/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "osis_pipeline" {
  role       = aws_iam_role.osis_pipeline.name
  policy_arn = aws_iam_policy.osis_pipeline.arn
}

resource "aws_cloudwatch_log_group" "osis_pipeline" {
  name              = "/aws/vendedlogs/OpenSearchIngestion/cloudtrail-pipeline"
  retention_in_days = 30
}

resource "aws_osis_pipeline" "cloudtrail" {
  pipeline_name = "cloudtrail-ingestion"
  pipeline_configuration_body = <<-EOT
    version: "2"
    cloudtrail-pipeline:
      source:
        s3:
          notification_type: "sqs"
          codec:
            json:
          compression: "gzip"
          sqs:
            queue_url: "${var.cloudtrail_sqs_queue_url}"
          aws:
            sts_role_arn: "${aws_iam_role.osis_pipeline.arn}"
            region: "${data.aws_region.current.name}"
      processor:
        - date:
            from_time_received: true
            destination: "@timestamp"
      sink:
        - opensearch:
            hosts: ["https://${aws_opensearch_domain.cloudtrail.endpoint}"]
            index: "cloudtrail-%{yyyy.MM.dd}"
            aws:
              sts_role_arn: "${aws_iam_role.osis_pipeline.arn}"
              region: "${data.aws_region.current.name}"
  EOT
  min_units = 2
  max_units = 10
  log_publishing_options {
    is_logging_enabled = true
    cloudwatch_log_destination {
      log_group = aws_cloudwatch_log_group.osis_pipeline.name
    }
  }
  tags = {
    Environment = "production"
    Project     = "centralized-cloudtrail-monitoring"
    ManagedBy   = "terraform"
  }
}

The pipeline uses the S3 source plugin with SQS-based notifications. When new CloudTrail log files land in S3, an SQS message triggers the pipeline to fetch and parse them. The min_units and max_units parameters control auto scaling. The pipeline starts at 2 OCUs and scales up to 10 based on queue depth, handling ingestion spikes without manual intervention. For lower environments (development and testing), you can apply throttling on the Amazon SQS queue to prevent non-production spikes from starving production ingestion capacity.

Index template

Define index templates up front to avoid painful reindexing later. The following template sets explicit mappings for CloudTrail fields, optimizes for write throughput with async translog durability, and integrates with ISM for automatic rollover:

resource "opensearch_index_template" "cloudtrail" {
  name = "cloudtrail-template"
  body = jsonencode({
    index_patterns = ["cloudtrail-*"]
    priority       = 100
    template = {
      settings = {
        number_of_shards                                  = 6
        number_of_replicas                                = 0
        "index.refresh_interval"                          = "10s"
        "index.translog.durability"                       = "async"
        "index.translog.sync_interval"                    = "30s"
        "plugins.index_state_management.rollover_alias"   = "cloudtrail"
      }
      mappings = {
        properties = {
          "@timestamp"        = { type = "date" }
          eventSource         = { type = "keyword" }
          eventName           = { type = "keyword" }
          awsRegion           = { type = "keyword" }
          sourceIPAddress     = { type = "ip" }
          errorCode           = { type = "keyword" }
          errorMessage        = { type = "text" }
          recipientAccountId  = { type = "keyword" }
          userIdentity = {
            properties = {
              type      = { type = "keyword" }
              arn       = { type = "keyword" }
              accountId = { type = "keyword" }
              userName  = { type = "keyword" }
              sessionContext = {
                properties = {
                  sessionIssuer = {
                    properties = {
                      type     = { type = "keyword" }
                      arn      = { type = "keyword" }
                      userName = { type = "keyword" }
                    }
                  }
                }
              }
            }
          }
          requestParameters = { type = "object", enabled = true }
          responseElements  = { type = "object", enabled = true }
        }
      }
    }
  })
}

Defining mappings before ingestion prevents mapping conflicts and avoids the need to reindex data after the fact.

Lifecycle management (ISM policy)

The following ISM policy replaces custom Lambda functions with a single declarative policy. The rollover action uses two OR conditions: min_index_age and min_primary_shard_size. Whichever threshold is reached first triggers the rollover. This keeps shard sizes bounded while ensuring timely rotation even during low-volume periods:

resource "opensearch_ism_policy" "cloudtrail_lifecycle" {
  policy_id = "cloudtrail-lifecycle"
  body = jsonencode({
    policy = {
      description   = "CloudTrail lifecycle - rollover, retain 30d, delete"
      default_state = "hot"
      ism_template  = [{ index_patterns = ["cloudtrail-*"], priority = 100 }]
      states = [
        {
          name    = "hot"
          actions = [{ rollover = { min_primary_shard_size = "30gb", min_index_age = "1d" } }]
          transitions = [{ state_name = "delete", conditions = { min_index_age = "30d" } }]
        },
        {
          name        = "delete"
          actions     = [{ delete = {} }]
          transitions = []
        }
      ]
    }
  })
}

This approach reduces lifecycle management code by approximately 60% compared to per-environment Lambda functions, and changes deploy in a single terraform apply.

Note: For indexes ingesting more than 100 GB/day (such as CloudTrail at 200 GB/day in this deployment), you can override min_index_age to 12h to roll over more frequently. The two conditions are OR-based in OpenSearch ISM. If a shard reaches 30 GB before 1 day, it rolls over on size. If 1 day passes before 30 GB, it rolls over on age.

Multi-team access control

When multiple teams need different access levels to the same data, define all roles declaratively and use for_each to create them consistently. The following example defines 4 team roles with varying permissions:

locals {
  team_roles = {
    security_ops = {
      description         = "Security Operations - full read, alert management"
      cluster_permissions = ["cluster_monitor", "cluster:admin/opendistro/alerting/*"]
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search", "get"] },
        { index_patterns = [".opendistro-alerting-*"], allowed_actions = ["read", "write", "search", "get", "delete"] }
      ]
    }
    incident_response = {
      description         = "Incident Response - full read for investigation"
      cluster_permissions = ["cluster_monitor"]
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search", "get"] }
      ]
    }
    compliance_auditors = {
      description         = "Compliance - read-only"
      cluster_permissions = []
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search"] }
      ]
    }
    devops = {
      description         = "DevOps - infra metrics and limited CloudTrail"
      cluster_permissions = ["cluster_monitor"]
      index_permissions = [
        { index_patterns = ["infra-metrics-*"], allowed_actions = ["read", "search", "get"] },
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search"] }
      ]
    }
  }
}

resource "opensearch_role" "teams" {
  for_each            = local.team_roles
  role_name           = each.key
  description         = each.value.description
  cluster_permissions = each.value.cluster_permissions

  dynamic "index_permissions" {
    for_each = each.value.index_permissions
    content {
      index_patterns  = index_permissions.value.index_patterns
      allowed_actions = index_permissions.value.allowed_actions
    }
  }

  dynamic "tenant_permissions" {
    for_each = [each.key]
    content {
      tenant_patterns = [each.key]
      allowed_actions = ["kibana_all_write"]
    }
  }
}

resource "opensearch_roles_mapping" "teams" {
  for_each      = local.team_roles
  role_name     = opensearch_role.teams[each.key].role_name
  backend_roles = var.team_iam_roles[each.key]
}

resource "opensearch_tenant" "teams" {
  for_each    = local.team_roles
  tenant_name = each.key
  description = "Dashboard workspace for ${replace(each.key, "_", " ")}"
}

This approach maps IAM roles (not individual users) to OpenSearch roles. Adding a new team means adding one entry to the locals block and running terraform apply. Each team gets an isolated tenant in OpenSearch Dashboards, preventing cross-team interference with saved searches, visualizations, and dashboard configurations. We chose OpenSearch Dashboards because tenants, roles, visualizations, and saved objects can all be managed programmatically through the Terraform OpenSearch provider, keeping the entire stack under infrastructure-as-code governance. For teams building new visualizations outside of Terraform-managed workflows, we recommend OpenSearch UI. This next-generation analytics interface supports multiple data sources, provides workspaces for team isolation, and remains available during cluster upgrades.

Alerting

Define alerting monitors in Terraform to detect security-critical events automatically. The following monitor catches CloudTrail tampering attempts (StopLogging, DeleteTrail) and sends alerts through Amazon SNS:

resource "opensearch_monitor" "cloudtrail_tampering" {
  body = jsonencode({
    name     = "CloudTrail Tampering Detection"
    type     = "monitor"
    enabled  = true
    schedule = { period = { interval = 1, unit = "MINUTES" } }
    inputs = [{
      search = {
        indices = ["cloudtrail-*"]
        query = {
          size = 5
          query = {
            bool = {
              must = [{ terms = { eventName = ["StopLogging", "DeleteTrail",
                "UpdateTrail", "PutEventSelectors", "DeleteEventDataStore"] } }]
              filter = [{ range = { "@timestamp" = { gte = "now-1m" } } }]
            }
          }
        }
      }
    }]
    triggers = [{
      name     = "trail_tampering_detected"
      severity = "1"
      condition = { script = {
        source = "ctx.results[0].hits.total.value > 0"
        lang   = "painless"
      } }
      actions = [{
        name             = "notify_security"
        destination_id   = var.sns_destination_id
        message_template = { source = "CRITICAL: CloudTrail tampering detected." }
      }]
    }]
  })
}

Results and performance

After deploying the solution, we measured steady-state performance against the SLAs defined in the workload profile:

Key outcomes:

• Threat detection dropped from hours to minutes. Automated alerting replaced manual log searching. The CloudTrail tampering monitor detects suspicious activity within one minute of the event.
• Compliance reports generate on demand. With 600+ saved searches and 100+ dashboards, compliance teams produce SOC 2, PCI DSS, and HIPAA audit evidence in minutes rather than days.
• Four teams operate independently. Each team has its own isolated tenant in OpenSearch Dashboards, preventing cross-team interference with saved searches and dashboard configurations.
• Zero custom Lambda functions. ISM policies, index templates, and access control are all managed declaratively through Terraform, eliminating the previous fragmented code base.

Best practices

• Define index templates before ingesting anything. Changing mappings on existing indices means reindexing. Get this right first.
• Set rollover thresholds based on your actual ingestion rate. At 200 GB/day, rolling over at 30 GB keeps shard counts manageable while balancing query performance.
• Test ISM transitions in a lower environment first. Warm and cold migrations on large indices take time.
• Map IAM roles, not users. People change teams. Roles stay stable. This simplifies access management.
• Put an Amazon SQS queue between S3 and the ingestion pipeline. This gives you per-environment throttling control without modifying pipeline configuration.
• Use for_each aggressively. Roles, tenants, index patterns, and monitors all follow a pattern across teams or environments, so use for_each to eliminate copy-paste drift.
• Consider OpenSearch UI for new visualization workflows. This solution uses OpenSearch Dashboards for Terraform-managed tenants and roles. OpenSearch UI is a next-generation interface that supports multiple data sources, stays available during cluster upgrades, and includes workspaces for team isolation. It is the recommended interface for creating new dashboards and visualizations going forward.

Optional: Extending with cold storage for longer retention

For organizations with compliance requirements mandating longer retention (for example, 7 years for PCI DSS or HIPAA), you can extend the ISM policy with warm and cold tiers. The following example adds tiered storage that moves data through hot, warm, cold, and delete states:

states = [
  {
    name    = "hot"
    actions = [{ rollover = { min_primary_shard_size = "30gb", min_index_age = "1d" } }]
    transitions = [{ state_name = "warm", conditions = { min_index_age = "30d" } }]
  },
  {
    name = "warm"
    actions = [
      { warm_migration = {} },
      { force_merge = { max_num_segments = 1 } }
    ]
    transitions = [{ state_name = "cold", conditions = { min_index_age = "365d" } }]
  },
  {
    name    = "cold"
    actions = [{ cold_migration = { timestamp_field = "@timestamp" } }]
    transitions = [{ state_name = "delete", conditions = { min_index_age = "2555d" } }]
  },
  {
    name        = "delete"
    actions     = [{ cold_delete = {} }]
    transitions = []
  }
]

Warm storage uses force-merge to reduce segment count (lowering query overhead), while cold storage moves data entirely to Amazon S3 for minimal cost. This tiered approach keeps hot-tier performance high while meeting long-term audit requirements.

Cleanup

To avoid incurring ongoing charges, remove the resources created in this post by running:

terraform destroy

This removes the OpenSearch domain, ingestion pipeline, IAM roles, SQS queues, and all associated configurations. Verify that you have exported any data or dashboards you want to retain before running destroy.

Conclusion

In this post, we showed you how to build a centralized CloudTrail monitoring solution on Amazon OpenSearch Service with Terraform managing the entire stack. The approach moves from fragile, manually configured systems with redundant Lambda code to a version-controlled, peer-reviewed, consistently deployed infrastructure.

Threat detection drops from hours to minutes with automated alerting. Compliance reports that took days now generate on demand. And your team spends time on security analysis instead of infrastructure maintenance.

To get started, use the AWS Terraform provider aws_opensearch_domain resource for the domain, then use the Terraform OpenSearch provider for index templates, ISM policies, roles, and monitors. Configure your ingestion pipeline to transform and enrich incoming CloudTrail logs before indexing, building a modern, scalable security foundation that grows with your organization.

The complete source code for this solution is available in the GitHub repository: GitHub repository

For more on the services used in this solution:

• Amazon OpenSearch Service
• AWS CloudTrail
• Amazon OpenSearch Ingestion
• Index State Management
• Fine-grained access control

About the authors