AWS Public Sector Blog
How to transfer data to the CISA Cloud Log Aggregation Warehouse (CLAW) using Amazon S3
In a previous Public Sector Blog post, we introduced Amazon Web Services (AWS) Trusted Internet Connections (TIC) 3.0 overlays to help agencies securely apply TIC 3.0 requirements to AWS deployments. In this post, we show you how you can push or pull your security telemetry data to the National Cybersecurity Protection System (NCPS) Cloud Log Aggregation Warehouse (CLAW) using Amazon Simple Storage Service (Amazon S3) or third-party solutions.
As federal agencies strive to adopt best practices and proactively enhance their cybersecurity posture, they contribute to a safer digital landscape for the nation. One specific action agencies must take as part of the TIC program is sharing their security telemetry data with the Cybersecurity and Infrastructure Security Agency (CISA) CLAW. Managed by NCPS, CLAW collects and analyzes agency cloud security data and allows CISA analysts to provide security situational awareness and support to the agencies. NCPS allows log collection in both the AWS commercial and GovCloud (US) Regions. NCPS TALONs collects logs in the commercial Region and then forwards them to CLAW in the GovCloud Region. For simplicity’s sake, we will only refer to CLAW, since this is the final destination for all the logs.
Connecting with NCPS for access
Before starting, you must contact the NCPS team for onboarding and logging location specifics. NCPS may ask for different AWS log types based on your environment, but you will generally send your VPC flow logs. The examples here will work for any of the various log types you may be asked to share. The NCPS team can be reached at NCPSProgramOffice@hq.dhs.gov.
Log aggregation strategies
You can send logs to CLAW from each individual AWS account or a central log aggregation point. If you are currently using a solution like AWS Control Tower, your log data for services like AWS CloudTrail or Amazon GuardDuty are already being sent to the log archive account, as shown in Figure 1. This same design is used when you manually deploy a landing zone pattern or with accelerators like the Landing Zone Accelerator.
Figure 1. Architecture diagram for landing zone provisioning by AWS Control Tower.
NCPS will allow agencies to send logs from a standalone AWS account or a central log aggregation point. However, it is better to use a central log aggregation point because this reduces the operational burden by limiting the number of roles and policies needed to deliver logging data. NCPS recommends delivering logs from the central logging account when possible.
Choosing a suitable data transfer method
AWS offers native integrations with the CLAW. Agencies have the following options for data transfer:
1. Cloud-native push or pull logs between S3 buckets.
– Push logs to NCPS’ S3 bucket.
– Provide NCPS access to pull logs from an agency’s S3 bucket.
2. Third-party log forwarding.
Cloud native: Amazon S3 push
You should push your logs to NCPS’ S3 bucket if you want control over what data is shared and when the data is shared. Some agencies may only want to share a subset of data based on a moving pattern and not a single static bucket prefix. One benefit of pushing logs to an S3 bucket is that the logs will be available to NCPS as soon as they are pushed, facilitating near real-time monitoring and analysis. With this method, you are responsible for managing the infrastructure and permissions required to push logs to another account, which can add operational overhead and complexity. You should be comfortable setting up cross-account access with AWS Identity and Access Management (IAM) policies and roles.
Amazon S3 push example
In this example, log files are sent from your account to the CLAW account in the same Region. This method is more automatic and should require little to no maintenance once set up. Figure 2 shows the log account after being configured for cross-account bucket replication.
Prerequisites
- The source and destination buckets must have versioning enabled.
- The source and destination bucket owners must have proper IAM permissions to allow replication. For questions, contact NCPS using the email provided.
Figure 2. Log account after being configured for cross-account bucket replication.
For detailed instructions, refer to Configuring replication when source and destination buckets are owned by different accounts.
Steps to complete:
- Enable versioning on the source bucket (“agency-bucket”). From the Amazon S3 console, select the bucket, go to Properties, and enable versioning.
- Provide the Amazon Resource Name (ARN) of your bucket to NCPS. NCPS will add a bucket policy to their destination bucket (“claw-bucket”), which allows the source account and bucket (“agency-bucket”) to replicate objects into the CLAW. NCPS will share with you the ARN of the destination bucket (“claw-bucket”).
- In the source bucket (“agency-bucket”), choose Management, then Replication rules, and choose Create replication rule.
- Enter a rule name like “ReplicateToCLAW”. Under Destination, select the radio button “Specify a bucket in another account” and then enter the 12 digit AWS account ID and bucket name. Choose Save to create the rule.
Objects added to the source bucket (“agency-bucket”) will now be automatically replicated to the destination bucket (“claw-bucket”) in near real-time. The CLAW account now has a copy of the objects from the agency account.
Cloud native: Amazon S3 pull
An agency can allow the CLAW to pull logs from their S3 bucket when they want to limit their operational burden. Once permissions are set up correctly, the CLAW will manage the process of pulling logs. This gives the CLAW flexibility and control over the schedule and rate of data ingestion. You must also carefully manage bucket policies and IAM roles to securely allow access to the data.
Amazon S3 pull example
In this example, the CLAW will pull log files from your account into the CLAW account in the same Region using an AWS Lambda function. New logs deposited into the source bucket generate a notification to the CLAW that triggers the Lambda function.
Figure 3 shows the CLAW account has been configured with an execution role with a permissions policy giving the NCPS CLAW Lambda function permission to access the required S3 bucket. This is accomplished within the agency log account by modifying the S3 bucket policy to explicitly allow the ARN of the NCPS Lambda function to GET objects from the specified S3 bucket. This information will be shared with you when you email the NCPS team.
Prerequisites
- The source and destination buckets must have versioning enabled.
- The source and destination bucket owners must have proper IAM permissions to allow the Lambda function to read and write objects. For questions, contact NCPS using the email provided.
Figure 3. Configured CLAW account.
- Enable versioning on the source bucket (“agency-bucket”). From the Amazon S3 console, select the bucket, go to Properties, and enable versioning.
- Enable source bucket (“agency-bucket”) notification. From the Amazon S3 console, select the bucket, go to Properties, navigate to Event Notifications, and create a notification for “All object create events”.
- Specify NCPS Amazon Simple Queue Service (Amazon SQS) queue ARN as the destination for the notification.
- Add a bucket policy to the source bucket (“agency-bucket”) to allow the NCPS Lambda execution role in the destination account to GET objects. This gives the CLAW account permissions to pull data from the source bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cisa-account-number:role/cisa-role-name"
},
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::your-bucket-name/*"
}
]
}Objects added to the source bucket (“agency-bucket”) will automatically generate an S3 event notification to a queue, which triggers a Lambda function to pull data into the destination bucket (“claw-bucket”). While Amazon S3 event notifications are designed to be delivered at least once and in seconds, they can take a minute or longer.
The CLAW account now has a copy of the objects from the agency account.
Third-party log forwarding
While the cloud-native approaches mentioned in this post are the preferred approach, NCPS also supports agencies forwarding through common platforms and technologies, such as Splunk, Elasticsearch, and Logstash, as well as other commercial tools. Each option has its strengths and may be preferred based on an agency’s requirements regarding data analysis capabilities, scalability, cost, and existing infrastructure.
Splunk – For agencies with an established practice using Splunk for advanced analytics and visualization, NCPS supports Splunk forwarding logs to CLAW-controlled Splunk ingestion points. Splunk provides numerous extensions and integrations for various data sources and use cases, potentially reducing setup time and effort. Further, Splunk has built-in security and compliance features, which can be advantageous for sensitive and regulated data.
Logstash – The Elastic Logstash Kibana (ELK) stack has a strong community and a wide range of plugins, offering flexibility in how logs are processed, stored, and visualized. Logstash can be configured to forward to another Logstash instance in the CLAW account.
Both Splunk and the ELK stack are capable of ingesting large volumes of data at scale. If an agency’s environment already uses either Splunk or the ELK stack for other purposes, then integrating log forwarding into the existing setup saves in terms of operations and learning curve.
An agency should consider its specific requirements for log analysis processes, including cost, scalability, compliance needs, and the existing technical skill set of the agency’s team.
Collaborate with CISA and share best practices
CISA is a valuable partner in your agency’s cybersecurity journey. Engage with CISA representatives at NCPSProgramOffice@hq.dhs.gov to seek guidance, share experiences, and learn from other federal agencies’ best practices. By fostering collaboration, agencies can collectively improve the efficiency and effectiveness of data transfer to the CLAW.
Conclusion
In this post, we’ve shown how you can meet your reporting requirements for providing the CLAW with telemetry from your AWS account. By understanding the different log aggregation strategies, implementing robust security measures, and collaborating with CISA, agencies can simplify the delivery of essential logs. Adopting best practices and staying proactive allows federal agencies to enhance their cybersecurity posture and contribute to a safer digital landscape for the nation. To learn more, review the NCPS Cloud Interface Reference Architecture documentation and contact your AWS account team.