AWS Partner Network (APN) Blog
How Leidos Standardized its Application Logging into Amazon Security Lake with LOIS
By Ryan Poorman, Sr. Cloud Engineer – Leidos
By Lawrence Sampson, Sr. Cloud Architect – Leidos
By Chris Renzo, Sr. Solutions Architect – AWS
 |
| Leidos |
 |
As networks become increasingly complex and systems generate more data than ever before, making sense of the plethora of data is critical.
The origin of this data can derive from a variety of sources, not limited to vendor logs, system logs, and developer logs. Related, these applications log in their own unique way to their own location; there is often no standardization on content or location.
This post will describe how Leidos is addressing the issues of application logging normalization and aggregation by employing the Open Cybersecurity Schema Framework (OCSF) and Amazon Security Lake via integration with LOIS – the Leidos OCSF Integration Suite.
Leidos is an AWS Premier Tier Services Partner and Managed Service Provider (MSP) with several AWS Competencies including Security and Level 1 MSSP Consulting. Leidos delivers technology, engineering, and science solutions to solve the world’s toughest challenges in the defense, intelligence, civil, and health markets.
Evolution of Logging
Application logging is the process of saving events occurring within an application. During the early days of software development, event logging by applications consisted of the occasional message or numeric code sent to a sys log. With the introduction of logging mechanisms such as Log4J, messages became far more useful. Log messages could be grouped by severity: debug, info, warning, error, and fatal.
A major upgrade was the ability to format the message itself and ease by which logging could be pulled into an application. However, the location of the logs, timestamp on the message, and messages themselves remain unique per each application. Though the mechanism of producing event messages had become standardized, the category or type of event the message was addressing, as well as its format and location, were not standardized.
If custom application logs, as well as vendor logs, could be both normalized and centrally aggregated, then administrators as well as security information event management (SIEM) tools could go to one place to do their analysis, without the overhead of normalizing the data themselves. Further, policies could be universally applied across the entire dataset.
OCSF addresses the normalization aspect of this problem, while Amazon Security Lake addresses aggregation, and LOIS provides the bridge to enable applications to produce OCSF-compliant messages and centralize them into Amazon Security Lake.
Message Content Normalization: OCSF
The OCSF standard defines a framework and schema for reporting events which are grouped by categories. Examples of the activities defined by OCSF are HTTP, API, DNS, Authentication, and Authorization; see a complete list.
These constructs define schemas that allow a developer to apply to their messages. For example, if an application contains a REST endpoint, then using the API event class with the cloud profile will enable a developer to produce a normalized message governed by OCSF that can be parsed and analyzed by any tool that understands the OCSF schema.
As stated by Paul Agbabian in his paper titled “Understanding the Open Cybersecurity Schema Framework,” there are five core constructs which comprise OCSF: data types, event class, category, profile, and extension. Agbabian also states that “OCSF is agnostic to storage format, data collection and ETL processes. The core schema is intended to be agnostic to implementations. The schema framework definition files and the resulting normative schema are written as JSON.”
An example of an OCSF message is shown below:
{
"message":"The following SAML Service API endpoint was triggered: https://xxxxxxxxxx.xxxxxxxxxxx.us-east-1.amazonaws.com/sp/authn/response",
"observables":[],
"cloud":{
"account":{
"name":"N/A",
"uid":"yyyyyyyyyyyy"
},
"provider":"Amazon Web Services",
"region":"us-east-1"
},
"type_uid":600300,
"activity_name":"SAML Service: Generic API",
"actor":{
"authorizations":[]
},
"api":{
"operation":"N/A"
},
"category_name":"Application Activity",
"activity_id":0,
"class_uid":6003,
"raw_data":"Pjwvc2FtbDI6QXR0cmlidXRlU3RhdGVtZW50Pjwvc2FtbDI6QXNzZXJ0aW9uPjwvc2FtbDJwOlJlc3BvbnNlPg==",
"category_uid":6,
"src_endpoint":{
"intermediate_ips":[
"xx.xxx.xx.xxx"
],
"ip":"xx.xxx.xx.xxx",
"port":0
},
"severity_id":1,
"resources":[],
"enrichments":[],
"status_code":"200",
"http_request":{
"http_headers":[],
"url":{
"categories":[],
"category_ids":[],
"hostname":"sp.lois-saml-integration.com",
"path":"",
"port":0,
"scheme":"https",
"url_string":"https://xxxxxxxxxxxxxxxxxxxx.us-east-1.amazonaws.com/sp/authn/response"
},
"user_agent":"Jersey/3.1.0 (HttpUrlConnection 17.0.8.1)",
"x_forwarded_for":[
"xx.xxx.xx.xxx"
]
},
"metadata":{
"labels":[
"Leidos OCSF SDK"
],
"log_provider":"Leidos OCSF SDK",
"product":{
"name":"OCSF Application Logger",
"vendor_name":"Leidos",
"version":"v1.2.4"
},
"profiles":[
"cloud"
],
"version":"1.0.0"
},
"time":1697749648604
}Message Location Standardization: Amazon Security Lake
Amazon Security Lake service is a data lake that centralizes OCSF messages across AWS environments, and external cloud service providers through the use of the Leidos SDKs. The service automatically partitions incoming data into a query-efficient Parquet format normalized into the OCSF schema.
AWS has made the following services available in the OCSF format via Amazon Security Lake in Parquet:
- AWS CloudTrail Lambda Data Events
- AWS CloudTrail Management Events
- AWS CloudTrail S3 Data Events
- Amazon Route 53 Resolver Query Logs
- AWS Security Hub Findings
- Amazon Virtual Private Cloud (VPC) Flow Logs
This is significant because it demonstrates AWS’s commitment to OCSF and the need for a centralized location for events. In addition to AWS making its services OCSF compliant, other companies have adopted OCSF as their logging format and push their logs into Amazon Security Lake. See the full list of companies that have integrated with Amazon Security Lake.
With AWS and a myriad of vendors being OCSF compliant, messages from these sources can now go to a central location (Amazon Security Lake) and in a standard format (OCSF) upon which additional processing can be conducted.
Solution Architecture: LOIS
The Leidos OCSF Integration Suite is a capability developed by Leidos focused on enabling applications to push their logs into Amazon Security Lake.
LOIS is comprised of three parts:
- OCSF message generation through the Leidos OCSF Loggers. These SDKs currently support Java and Python.
- OCSF message ingestion, validation, and transformation into Amazon Security Lake.
- Visualization of the custom sources created and managed by LOIS.
Leidos SDKs
The Leidos OCSF Loggers are software development kits available in Java and Python. These SDKs enable developers to generate OCSF compliant messages.
LOIS streamlines the process of logging data into OCSF format by providing pre-defined APIs that are built on top of the OCSF activity schemas. The SDKs ensure a developer’s log messages comport to the schema definitions. Application logs cannot be solely focused on developer troubleshooting, or for ensuring auditing compliance five years in the future.
Logging is a necessary part of any given application’s security posture, and as such it ought to be robustly included in the evaluation of code quality and health. LOIS provides the capability for an application’s events to be included in Amazon Security Lake so they can be correlated against other events. This capability eliminates logging silos.
Developed by developers for developers, the Leidos OCSF Loggers provide APIs that can be integrated into to an existing application’s logging implementation. With flexibility being a fundamental pillar of LOIS, the Leidos OCSF SDKs do not need to be used with any other capability LOIS offers.
The following is an example of the LOIS Java SDK generating a basic log message:
package com.leidos.ocsf;
import com.leidos.ocsf.builder.OCSFEventDirector;
public class App {
public static void main(String[] args){
OCSFEventDirector.ocsfAuthenticationActivityBuilder().withMessage("This is a basic message").log();
}
}
ETL with Amazon CloudWatch
LOIS’s ETL processes centralizes event messages generated by the LOIS SDKs, which can log locally or to Amazon CloudWatch. If an application incorporates LOIS, they have the option to configure the SDK to push the OCSF messages into CloudWatch.
An ETL process scans CloudWatch looking for OCSF entries. When they are detected, the messages are validated, grouped by activity, written out to a Parquet file which is the preferred format by Amazon Security Lake, and then pushed into the Security Lake.
Figure 1 – LOIS ETL process for Amazon CloudWatch.
ETL with Amazon S3
The ETL process with Amazon Simple Storage Service (Amazon S3) was developed to provide an option for those applications without direct access to the AWS cloud. For example, if an application was resident outside of Amazon, LOIS can be configured to write out to a local storage medium.
A transfer tool will then pick up the messages and transfer to an S3 bucket. Once in the bucket, the ETL processes the messages for ingestion into Amazon Security Lake.
Figure 2 – LOIS ETL process for Amazon S3.
Leidos OCSF Visualization
Ultimately, data is meaningless unless it can tell a story. LOIS helps you visualize this story through Amazon QuickSight via pre-defined dashboards. These are meant to sync with the custom sources that were established using LOIS’s integration with Amazon Security Lake.
Alternatively, since the data at this stage is already normalized and aggregated, security professionals are empowered to create their own displays based on the information they deem most important.
Security is not just about storing data; it’s about understanding through visualization. These visualizations built on the foundation of data normalization and aggregation, assist with making informed decisions, faster, with data you can understand.
Figure 3 – LOIS OCSF dashboard.
Use Case: Security Markup Assertion Language, SAML
If a REST endpoint within an application is receiving a high volume of invalid requests, it should be logged and correlated against other results that may alert an administrator to a potential issue.
For example, in the SAML protocol there’s an identity provider (IdP) and service provider (SP). The IdP authenticates users directed to it from a SP, and when an IdP is used as the authentication means a session ID is used to track requests and responses.
Leidos modified an SP to use LOIS, while the IdP was OCSF-compliant and integrated with Amazon Security Lake. LOIS was applied to the authentication and response handling methods.
Correlating the information in Amazon Security Lake, Leidos was able to find requests that were attackers attempting to gain access to the system. Leidos was then able to identify the type of attacks and their frequency.
Conclusion
The Open Cybersecurity Schema Framework (OCSF) standard, along with the release of Amazon Security Lake, provide the foundation for the normalization and aggregation of event logging.
In combination with the Leidos OCSF Integration Suite (LOIS), developers can now produce OCSF-compliant applications. Having the capability to correlate custom application data alongside AWS services and third-party vendors allows administrators to usher in a new class of services to look for anomalies, cyber or otherwise.
For an enterprise application to function properly, it requires a scalable logging environment. It’s crucial that every application reports its activity to a tool like Amazon Security Lake.
.
.
Leidos – AWS Partner Spotlight
Leidos is an AWS Premier Tier Services Partner and MSP that delivers information technology, engineering, and science solutions to solve the world’s toughest challenges in the defense, intelligence, civil, and health markets.