FSI Services Spotlight: Featuring AWS Batch

In this edition of the Financial Services Industry (FSI) Services Spotlight monthly blog series, we highlight five key considerations for customers running workloads on AWS Batch: achieving compliance, data protection, isolation of compute environments, audits with APIs, and access control/security. Across each area, we will examine specific guidance, suggested reference architectures, and technical code to help streamline service approval of Amazon Batch.

AWS Batch helps you to run large-scale batch processing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of computing resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure, similar to traditional batch computing software. This service can efficiently provision resources in response to jobs submitted to eliminate capacity constraints, reduce costs, and deliver results quickly.

As a fully managed service, AWS Batch helps you to run batch computing workloads of any scale. AWS Batch automatically provisions compute resources and optimizes the workload distribution based on the quantity and scale of the workloads. With AWS Batch, there’s no need to install or manage batch computing software, so you can focus on analyzing results and solving problems.

Financial Services organizations, from fintech startups to longstanding enterprises, have been utilizing batch processing in areas such as High-Performance Computing (HPC) for risk management, end-of-day trade processing, and fraud surveillance. You can use AWS Batch to minimize human error, increase speed and accuracy, and reduce costs with automation to refocus on evolving the business. For example, Zerodha, India’s largest retail stockbroker with active client base of over 10 millions who place millions of orders every day. This post shows how they have reduced their backend processing time from a few hours to minutes using AWS Batch.

We also observe the following use-cases that AWS Batch helps solve, within FSI.

High-Performance Computing

The Financial Services industry has advanced high-performance computing in pricing, market positions, and risk management. Organizations have increased speed, scalability, and cost savings by taking these compute-intensive workloads onto AWS. With AWS Batch, organizations can automate the resourcing and scheduling of these jobs to save costs and accelerate decision-making and go-to-market speeds.

Figure-1 High-Performance Computing

Post-Trade Analytics

Trading desks are constantly looking for opportunities to improve their positions by analyzing the day’s transaction costs, execution reporting, and market performance, among other areas. This requires batch processing of large data sets from multiple sources after the trading day closes. AWS Batch enables the automation of these workloads so that you can understand the pertinent risk going into the next day’s trading cycle and make better decisions based on data.

Figure-2 Post-Trade Analytics

Fraud Surveillance

Fraud is an ongoing concern impacting all industries. Amazon Machine Learning enables more intelligent ways to analyze data using algorithms and models to combat this challenge. When used with AWS Batch, organizations can automate the data processing or analysis required to detect irregular patterns in your data that could indicate fraudulent activity, such as money laundering and payments fraud.

Figure-3 Fraud Surveillance

Achieving Compliance

Security and compliance are a shared responsibility between AWS and the customer. AWS will operate, manage, and protect the infrastructure that runs the AWS services. The customer’s responsibility is determined by the service selected; the more managed services are used, the less customer configuration is required. As Amazon Batch is a managed service, customers are responsible for fewer controls to deploy secure transactional workloads with the seven database engines. On the customer’s side of the shared responsibility model, customers should first determine their requirements for network connectivity, encryption, and access to other AWS resources. We will dive deeper into those topics in the upcoming sections.

AWS Batch falls under the scope of the following compliance programs regarding AWS’ side of the shared responsibility model. The compliance programs covered by AWS Batch include:

SOC 1,2,3
PCI
CSA STAR CCM v3.0.1 and CSA STAR CCM v4.0
ISO/IEC 27001:2013, 27017:2015, 27018:2019, 27701:2019, 22301:2019, 9001:2015
ISMAP
FedRAMP (Moderate and High)
DoD CC SRG (IL2-IL6)
HIPAA
IRAP
MTCS (Regions: US-East, US-West, Singapore, Seoul)
C5
K-ISMS
ENS High (only Amazon S3 Glacier is in scope)
OSPAR (only Amazon S3 Glacier is in scope)
HITRUST CSF
FINMA
GSMA (Regions: US-East (Ohio) and Europe (Paris))
PiTuKri
CCCS Medium
GNS National Restricted Certification

In the following sections, we will cover topics on the customer side of the shared responsibility model. You can obtain corresponding compliance reports under an AWS non-disclosure agreement (NDA) through AWS Artifact. It is important to understand that AWS Batch compliance status does not automatically apply to applications that you run in the AWS Cloud. You need to ensure that your use of AWS services complies with the standards.

Your scope of the shared responsibility model when using AWS Batch is determined by the sensitivity of your data, your organization’s compliance objectives, and applicable laws and regulations. If your use of AWS Batch is subject to compliance with standards like HIPAA, PCI, or FedRAMP, AWS provides resources to help.

Data Protection

AWS Batch is a managed service that’s protected by the AWS global network security. For information about AWS security services and how AWS protects infrastructure, see AWS Cloud Security. To design your AWS environment using the best practices for infrastructure security, see Infrastructure Protection in Security Pillar of the AWS Well‐Architected Framework. You can use AWS published API calls to access AWS Batch through the network. Clients must support the following:

Transport Layer Security (TLS). We require TLS 1.2 and recommend TLS 1.3.
Cipher suites with perfect forward secrecy (PFS) such as DHE (Ephemeral Diffie-Hellman) or ECDHE (Elliptic Curve Ephemeral Diffie-Hellman). Most modern systems such as Java 7 and later support these modes.

Additionally, requests must be signed by using an access key ID and a secret access key that is associated with an AWS Identify and Access Management (IAM) principal. Or you can use the AWS Security Token Service (AWS STS) to generate temporary security credentials to sign requests.

You can call these API operations from any network location, but AWS Batch does support resource-based access policies, which can include restrictions based on the source IP address. You can also use AWS Batch policies to control access from specific Amazon Virtual Private Cloud (Amazon VPC) endpoints or specific VPCs. Effectively, this isolates network access to a given AWS Batch resource to only the specific VPC within the AWS network.

Isolation of compute environments with AWS Batch

Customers can apply network-level controls, such as security groups and network ACLs, to their containerized workloads such as AWS Fargate and Amazon Elastic Kubernetes Service (Amazon EKS) for AWS Batch. On AWS Fargate, this can be done by creating an elastic network interface (ENI) in their specified VPC and attaching it to the Fargate managed instance. This gives customers control over the network-level access of the services they run on Amazon ECS. For more information, see Security Groups for your VPC and Network ACLs.

As a managed service, AWS Batch is protected by the AWS global network security procedures that are described in the Amazon Web Services: Overview of Security Processes whitepaper.

Automating audits with APIs with AWS Batch

AWS Batch is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in AWS Batch. AWS CloudTrail captures all API calls for AWS Batch as events. The calls captured include calls from the AWS Batch console and code calls to the AWS Batch API operations. If you create a trail, you can enable continuous delivery of AWS CloudTrail events to an Amazon Simple Storage Service (Amazon S3) bucket, including events for AWS Batch. If you don’t configure a trail, you can still view the most recent events in the AWS CloudTrail console in Event history. Using the information collected by AWS CloudTrail, you can determine the request that was made to AWS Batch, the IP address from which the request was made, who made the request, when it was made, and additional details.

AWS CloudTrail is enabled on your AWS account when you create the account. When any activity occurs in AWS Batch, that activity is recorded in an AWS CloudTrail event along with other AWS service events in the Event history. You can view, search, and download recent events in your AWS account. For more information, see Viewing Events with CloudTrail Event History.

For an ongoing record of events in your AWS account, including events for AWS Batch, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all Regions in the AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in AWS CloudTrail logs.

All AWS Batch actions are logged by CloudTrail, such as the SubmitJob, ListJobs and DescribeJobs operations. Every event or log entry contains information about who generated the request. The identity information helps you determine the following:

Whether the request was made with root or AWS IAM user credentials.
Whether the request was made with temporary security credentials for a role or federated user.
Whether the request was made by another AWS service.

The following is an example of what a CloudTrail log looks like for the event name, CreateComputeEnvironment action:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AIDACKCEVSQ6C2EXAMPLE:admin",
    "arn": "arn:aws:sts::012345678910:assumed-role/Admin/admin",
    "accountId": "012345678910",
    "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
    "sessionContext": {
      "attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2017-12-20T00:48:46Z"
      },
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AIDACKCEVSQ6C2EXAMPLE",
        "arn": "arn:aws:iam::012345678910:role/Admin",
        "accountId": "012345678910",
        "userName": "Admin"
      }
    }
  },
  "eventTime": "2017-12-20T00:48:46Z",
  "eventSource": "batch.amazonaws.com",
  "eventName": "CreateComputeEnvironment",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.1",
  "userAgent": "aws-cli/1.11.167 Python/2.7.10 Darwin/16.7.0 botocore/1.7.25",
  "requestParameters": {
    "computeResources": {
      "subnets": [
        "subnet-5eda8e04"
      ],
      "tags": {
        "testBatchTags": "CLI testing CE"
      },
      "desiredvCpus": 0,
      "minvCpus": 0,
      "instanceTypes": [
        "optimal"
      ],
      "securityGroupIds": [
        "sg-aba9e8db"
      ],
      "instanceRole": "ecsInstanceRole",
      "maxvCpus": 128,
      "type": "EC2"
    },
    "state": "ENABLED",
    "type": "MANAGED",
    "computeEnvironmentName": "Test"
  },
  "responseElements": {
    "computeEnvironmentName": "Test",
    "computeEnvironmentArn": "arn:aws:batch:us-east-1:012345678910:compute-environment/Test"
  },
  "requestID": "890b8639-e51f-11e7-b038-EXAMPLE",
  "eventID": "874f89fa-70fc-4798-bc00-EXAMPLE",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "recipientAccountId": "012345678910"
}

Access Control with AWS Batch

AWS Identity and Access Management (IAM) is an AWS service that helps an administrator securely control access to AWS resources. IAM administrators control who can be authenticated (signed in) and authorized (have permissions) to use AWS Batch resources. IAM is an AWS service that you can use with no additional charge. To get a high-level view of how AWS Batch and other AWS services work with most IAM features, see AWS Services that work with IAM for details of managing access to AWS Batch.

Conclusion

In this post, we have reviewed AWS Batch, highlighting essential information that can help FSI customers accelerate the service’s approval within these five categories: achieving compliance, data protection, isolation of computing environments, automating audits with APIs, and access control. While not a one-size-fits-all approach, the guidance can be adapted to meet the organization’s security and compliance requirements. Be sure to visit our AWS Industries blog channel and stay tuned for more financial services news and best practices.

AWS for Industries