Running cross-account workflows with AWS Step Functions and Amazon API Gateway

This post is written by Hardik Vasa, Senior Solutions Architect, and Pratik Jain, Cloud Infrastructure Architect.

AWS Step Functions allow you to build scalable and distributed applications using state machines. With the launch of Step Functions nested workflows, you can start a Step Functions workflow from another workflow. However, this requires both workflows to be in the same account. There are many use cases that require you to orchestrate workflows across different AWS accounts from one central AWS account.

This blog post covers a solution to invoke Step Functions workflows cross account using Amazon API Gateway. With this, you can perform cross-account orchestration for scheduling, ETL automation, resource deployments, security audits, and log aggregations all from a central account.

Overview

The following architecture shows a Step Functions workflow in account A invoking an API Gateway endpoint in account B, and passing the payload in the API request. The API then invokes another Step Functions workflow in account B asynchronously. The resource policy on the API allows you to restrict access to a specific Step Functions workflow to prevent anonymous access.

You can extend this architecture to run workflows across multiple Regions or accounts. This blog post shows running cross-account workflows with two AWS accounts.

To invoke an API Gateway endpoint, you can use Step Functions AWS SDK service integrations. This approach allows users to build solutions and integrate services within a workflow without writing code.

The example demonstrates how to use the cross-account capability using two AWS example accounts:

Step Functions state machine A: Account ID #111111111111
API Gateway API and Step Functions state machine B: Account ID #222222222222

Setting up

Start by creating state machine A in the account #111111111111. Next, create the state machine in target account #222222222222, followed by the API Gateway REST API integrated to the state machine in the target account.

Account A: #111111111111

In this account, create a state machine, which includes a state that invokes an API hosted in a different account.

Create an IAM role for Step Functions

Sign in to the IAM console in account #111111111111, and then choose Roles from left navigation pane
Choose Create role.
For the Select trusted entity, under AWS service, select Step Functions from the list, and then choose Next.
On the Add permissions page, choose Next.
On the Review page, enter StepFunctionsAPIGatewayRole for Role name, and then choose Create role.
Create inline policies to allow Step Functions to access the API actions of the services you need to control. Navigate to the role that you created and select Add Permissions and then Create inline policy.
Use the Visual editor or the JSON tab to create policies for your role. Enter the following:
```
Service: Execute-API
Action: Invoke
Resource: All Resources
```
Choose Review policy.
Enter APIExecutePolicy for name and choose Create Policy.

Creating a state machine in source account

Navigate to the Step Functions console in account #111111111111 and choose Create state machine
Select Design your workflow visually, and the click Standard and then click Next
On the design page, search for APIGateway:Invoke state, then drag and drop the block on the page:
In the API Gateway Invoke section on the right panel, update the API Parameters with the following JSON policy:
```
 {
  "ApiEndpoint.$": "$.ApiUrl",
  "Method": "POST",
  "Stage": "dev",
  "Path": "/execution",
  "Headers": {},
  "RequestBody": {
   "input.$": "$.body",
   "stateMachineArn.$": "$.stateMachineArn"
  },
  "AuthType": "RESOURCE_POLICY"
}
```
These parameters indicate that the ApiEndpoint, payload (body) and stateMachineArn are dynamically assigned values based on input provided during workflow execution. You can also choose to assign these values statically, based on your use case.
[Optional] You can also configure the API Gateway Invoke state to retry upon task failure by configuring the retries setting.
Choose Next and then choose Next again. On the Specify state machine settings page:
1. Enter a name for your state machine.
2. Select Choose an existing role under Permissions and choose StepFunctionsAPIGatewayRole.
3. Select Log Level ERROR.
Choose Create State Machine.

After creating this state machine, copy the state machine ARN for later use.

Account B: #222222222222

In this account, create an API Gateway REST API that integrates with the target state machine and enables access to this state machine by means of a resource policy.

Creating a state machine in the target account

Navigate to the Step Functions Console in account #222222222222 and choose Create State Machine.
Under Choose authoring method select Design your workflow visually and the type as Standard.
Choose Next.
On the design page, search for Pass state. Drag and drop the state.
Choose Next.
In the Review generated code page, choose Next and:
1. Enter a name for the state machine.
2. Select Create new role under the Permissions section.
3. Select Log Level ERROR.
Choose Create State Machine.

Once the state machine is created, copy the state machine ARN for later use.

Next, set up the API Gateway REST API, which acts as a gateway to accept requests from the state machine in account A. This integrates with the state machine you just created.

Create an IAM Role for API Gateway

Before creating the API Gateway API endpoint, you must give API Gateway permission to call Step Functions API actions:

Sign in to the IAM console in account #222222222222 and choose Roles. Choose Create role.
On the Select trusted entity page, under AWS service, select API Gateway from the list, and then choose Next.
On the Select trusted entity page, choose Next
On the Name, review, and create page, enter APIGatewayToStepFunctions for Role name, and then choose Create role
Choose the name of your role and note the Role ARN:
arn:aws:iam::222222222222:role/APIGatewayToStepFunctions
Select the IAM role (APIGatewayToStepFunctions) you created.
On the Permissions tab, choose Add permission and choose Attach Policies.
Search for AWSStepFunctionsFullAccess, choose the policy, and then click Attach policy.

Creating the API Gateway API endpoint

After creating the IAM role, create a custom API Gateway API:

Open the Amazon API Gateway console in account #222222222222.
Click Create API. Under REST API choose Build.
Enter StartExecutionAPI for the API name, and then choose Create API.
On the Resources page of StartExecutionAPI, choose Actions, Create Resource.
Enter execution for Resource Name, and then choose Create Resource.
On the /execution Methods page, choose Actions, Create Method.
From the list, choose POST, and then select the check mark.

Configure the integration for your API method

On the /execution – POST – Setup page, for Integration Type, choose AWS Service. For AWS Region, choose a Region from the list. For Regions that currently support Step Functions, see Supported Regions.
For AWS Service, choose Step Functions from the list.
For HTTP Method, choose POST from the list. All Step Functions API actions use the HTTP POST method.
For Action Type, choose Use action name.
For Action, enter StartExecution.
For Execution Role, enter the role ARN of the IAM role that you created earlier, as shown in the following example. The Integration Request configuration can be seen in the image below.
arn:aws:iam::222222222222:role/APIGatewayToStepFunctions
Choose Save. The visual mapping between API Gateway and Step Functions is displayed on the /execution – POST – Method Execution page.

After you configure your API, you can configure the resource policy to allow the invoke action from the cross-account Step Functions State Machine. For the resource policy to function in cross-account scenarios, you must also enable AWS IAM authorization on the API method.

Configure IAM authorization for your method

On the /execution – POST method, navigate to the Method Request, and under the Authorization option, select AWS_IAM and save.
In the left navigation pane, choose Resource Policy.

Use this policy template to define and enter the resource policy for your API.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "states.amazonaws.com"
            },
            "Action": "execute-api:Invoke",
            "Resource": "execute-api:/*/*/*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceArn": [
                        "<SourceAccountStateMachineARN>"
                     ]
                }
            }
        }
    ]
}

Note: You must replace <SourceAccountStateMachineARN> with the state machine ARN from account #111111111111 (account A).

Choose Save.

Once the resource policy is configured, deploy the API to a stage.

Deploy the API

In the left navigation pane, click Resources and choose Actions.
From the Actions drop-down menu, choose Deploy API.
In the Deploy API dialog box, choose [New Stage], enter dev in Stage name.
Choose Deploy to deploy the API.

After deployment, capture the API ID, API Region, and the stage name. These are used as inputs during the execution phase.

Starting the workflow

To run the Step Functions workflow in account A, provide the following input:

{
   "ApiUrl": "<api_id>.execute-api.<region>.amazonaws.com",
   "stateMachineArn": "<stateMachineArn>",
   "body": "{\"someKey\":\"someValue\"}"
}

Paste in the values of APIUrl and stateMachineArn from account B in the preceding input. Make sure the ApiUrl is in the format as shown.

AWS Serverless Application Model deployment

You can deploy the preceding solution architecture with the AWS Serverless Application Model (AWS SAM), which is an open-source framework for building serverless applications. During deployment, AWS SAM transforms and expands the syntax into AWS CloudFormation syntax, enabling you to build serverless applications faster.

Logging and monitoring

Logging and monitoring are vital for observability, measuring performance and audit purposes. Step Functions allows logging using CloudWatch Logs. Step Functions also automatically sends execution metrics to CloudWatch. You can learn more on monitoring Step Functions using CloudWatch.

Cleaning up

To avoid incurring any charges, delete all the resources that you have created in both the accounts. This would include deleting the Step Functions state machines and API Gateway API.

Conclusion

This blog post provides a step-by-step guide on securely invoking a cross-account Step Functions workflow from a central account using API Gateway as front end. This pattern can be extended to scale workflow executions across different Regions and accounts.

By using a centralized account to orchestrate workflows across AWS accounts, this can help prevent duplicating work in each account.

To learn more about serverless and AWS Step Functions, visit the Step Functions Developer Guide.

For more serverless learning resources, visit Serverless Land.

AWS Compute Blog