Automate the configuration of Amazon S3 Object Lambda to process data as its retrieved

Customers often have multiple applications consuming information from a single dataset stored in Amazon S3, but each application requires different variations to meet their specific needs. For example, one application may require that data be in a redacted CSV format, while another application requires unredacted information. As another example, a stock image website may require that non-paying users receive lower resolution, watermarked images, while customers using a premium service get access to high resolution, original images. This can be addressed by operating a proxy layer in front of Amazon S3, maintaining multiple derivative copies of your data so each application as its own unique dataset, or using Amazon S3 Object Lambda to transform data as it is returned to an application.

Customers of all sizes and industries are shifting their workloads to serverless technologies because they don’t have to provision or manage underlying compute resources. With S3 Object Lambda, powered by AWS Lambda functions, your code runs on infrastructure that is fully managed by AWS, eliminating the need to create and store derivative copies of your data or to run expensive proxies, all with no changes required to applications. You can get started with S3 Object Lambda with just a few clicks in the S3 console, or you can use an AWS CloudFormation template to automate the configuration of S3 Object Lambda, which is available on GitHub. Using an AWS CloudFormation template enables you to implement best practices, improve your security posture, reduce errors caused by manual processes, and focus on innovation and implementing business logic, instead of managing the setup process.

In this blog post, we show you how to use an AWS CloudFormation template to automate your S3 Object Lambda configuration. The AWS CloudFormation template will automatically create relevant resources, configure IAM roles, and set up a Lambda function that handles requests through the S3 Object Lambda Access Point. Then, we walk through a real-world scenario where you can use the AWS CloudFormation template to configure an S3 Object Lambda Access Point and start transforming images as they are retrieved.

Solution overview

Let’s begin by setting up an AWS CloudFormation template that will create an S3 Object Lambda Access Point.

Prerequisites

You will have to meet three prerequisites before we get started:

1. Clone the GitHub repository
Clone the S3 Object Lambda GitHub repository to your local environment. You will use the resources in this repository, such as the CloudFormation template and Lambda function, to set up an S3 Object Lambda Access Point.

2. Choose an S3 bucket
Choose an S3 bucket containing objects that you would like to process with an S3 Object Lambda Access Point. You will need the bucket name when we deploy the template.

3. Create a Lambda function deployment package
Create a deployment package in a separate S3 bucket that has versioning enabled.

1. Download the file s3objectlambda_deployment_package.zip under function/node.js14.x/release in the GitHub repository.
2. Upload the Lambda function deployment package to an S3 bucket that supports versioning. You will need the bucket name, object key, and version ID when we deploy the template.

Usage

1. Deploy the CloudFormation template

The template can be deployed using the AWS Command Line Interface (CLI) or the AWS Management Console. In the following example, we will show how to deploy the template using the AWS CLI.

Using the AWS CLI

You can download the template under template/s3objectlambda_defaultconfig.yaml in the GitHub repository. The template can be deployed to your AWS account using the following command.

aws cloudformation deploy --template-file s3objectlambda_defaultconfig.yaml
--stack-name AWS CloudFormation stack name --parameter-overrides ObjectLambdaAccessPointName=Object Lambda access point name
SupportingAccessPointName=Amazon S3 access point S3BucketName=Amazon S3 bucket
LambdaFunctionS3BucketName=Amazon S3 bucket containing your Lambda package LambdaFunctionS3Key=Lambda object key LambdaFunctionS3ObjectVersion=Lambda object version --capabilities capability_IAM

Parameters

• stack-name takes an identifier for your stack. See the specifying stack name and parameters User Guide for choosing a stack name.
• ObjectLambdaAccessPointName takes the name of your S3 Object Lambda Access Point. See the naming S3 Access Points User Guide for choosing a name.
• SupportingAccessPointName takes the name of an S3 Access Point associated with the S3 bucket passed in the S3BucketName parameter. If you do not have an existing S3 Access Point, you can pass the CreateNewSupportingAccessPoint=true parameter override. This makes the CloudFormation template create an S3 Access Point for you.
• S3BucketName takes the bucket name to use with S3 Object Lambda. The bucket must exist in the same AWS account and AWS Region that will deploy this template. The bucket should also delegate access control to Access Points.
• LambdaFunctionS3BucketName takes the name of the S3 bucket where you have uploaded the Lambda function deployment package. The bucket should be in the same AWS Region as your function, but can be in a different AWS account.
• LambdaFunctionS3Key takes the S3 object key of the Lambda function deployment package.
• LambdaFunctionS3ObjectVersion takes the object version ID of the Lambda Function deployment package.
• --capabilities CAPABILITY_IAM is required as the AWS CloudFormation template creates an IAM role for the Lambda function execution.

You can optionally pass other parameters; see the GitHub repository for a full list of parameters.

2. Use the Amazon S3 Object Lambda Access Point

Once the template is successfully deployed, you can begin reading objects from your S3 bucket. By default, the Lambda function does not perform any transformation; when you perform a GetObject request, you will see the objects exactly as they are stored in your S3 buckets.

You can make a GetObject request by passing the S3 Object Lambda Access Point ARN as the S3 bucket parameter.

aws s3api get-object --bucket <object-lambda-access-point-arn>
--key <object-key> <outfile>

For more information, refer to the GetObject API Guide.

3. Implement your transformation

Now that you have set up the S3 Object Lambda Access Point, you can add your transformation logic to the Lambda function. Find detailed instructions on the GitHub repository. Next, we’ll look at a hypothetical scenario where an image-sharing company uses this AWS CloudFormation template to configure an S3 Object Lambda Access Point, then creates a Lambda function to automatically transform their data.

A real-world scenario

To recap, we showed you how to use a CloudFormation template to eliminate multiple steps in configuring an S3 Object Lambda Access Point. Now, let’s walk through a real-world scenario where you can use S3 Object Lambda to transform images as they are retrieved. In this example, we show how a stock photography and video website can use S3 Object Lambda to crop and add customized watermarks to images before data is returned to end users. This design pattern enables you to reduce cost by eliminating the need to create and store derivative copies of your data or to run expensive proxies.

With a Lambda function attached to an S3 Object Lambda Access Point, S3 automatically calls your Lambda function and S3 GET requests return a transformed result – all other requests are processed as normal.

First, let’s use the template to create an S3 Object Lambda Access Point. In this demo, we use the CloudFormation console to run the template. Alternatively, you can use the AWS CLI as shown in the previous section.

Step 1: Specify the template in the CloudFormation console. The following screenshot shows that we downloaded the template from the GitHub repository and uploaded it to the CloudFormation console.

Step 2: Specify stack details to set the essential parameters to the CloudFormation template. In this example, we will upload the Lambda function deployment package to a versioned S3 bucket named “image-watermarking-lambda.”

Step 3: Acknowledge that the template will create IAM resources. The template creates a new Lambda function execution role.

Step 4: Click on Create stack, and the building process will begin. Your stack will take about 5-10 minutes to complete. When CREATE_COMPLETE is displayed, the building process is complete.

Now, let’s use the “watermarking-ol-ap” to access the S3 bucket contents. For now, the objects will be identical to the S3 bucket objects, as the Lambda function does not transform objects yet.

Now, we edit the Lambda function to add the watermark and crop the image. We use the pdf-lib GitHub project to process the image and apply the watermark. We first install this dependency in our Node.js function, in the function/nodejs_14_x/ folder of the GitHub repository.

npm install pdf-lib

We create the image processing code by implementing the transform/s3objectlambda_transformer.ts function in the src folder.

import { PDFDocument, StandardFonts, degrees, rgb } from 'pdf-lib';

export default async function transformObject (originalObject: Buffer): Promise<Buffer> {
// Create an empty PDF document
const pdfDoc = await PDFDocument.create();

// Add a new page
const page = pdfDoc.addPage();

// Embed the original image in the centre of the PDF page
const image = await pdfDoc.embedPng(originalObject);
page.drawImage(image, {
        x: page.getWidth() / 2 - image.width / 2,
        y: page.getHeight() / 2 - image.height / 2,
        width: image.width,
        height: image.height
});

// Add a watermark that says that the page was created by Amazon S3 Object Lambda
const courierFont = await pdfDoc.embedFont(StandardFonts.Courier);
page.drawText('CREATED BY AMAZON S3 OBJECT LAMBDA', {
        font: courierFont,
        x: 125,
        y: 350,
        opacity: 0.9,
        rotate: degrees(30),
        color: rgb(1, 1, 1)
});

// Return the watermarked image
return pdfDoc.save().then(uint8Array => Buffer.from(uint8Array));
}

While creating this function, we have changed the signature of the transformObject method to return a Promise. So, we await the results in the handleGetObjectRequest function by changing the following line:

// Transform the object
const transformedWholeObject = await transformObject(originalObject);

We also update the content-type to application/pdf as we return PDF documents. We make this change to the writeResponse function in get_object_handler.

async function writeResponse (s3Client: S3, requestContext: GetObjectContext, transformedObject: Buffer | Uint8Array,
headers: Headers): Promise<PromiseResult<{}, AWSError>> {
const { algorithm, digest } = getChecksum(transformedObject);
 
return s3Client.writeGetObjectResponse({
        RequestRoute: requestContext.outputRoute,
        RequestToken: requestContext.outputToken,
        Body: transformedObject,
        Metadata: {
               'body-checksum-algorithm': algorithm,
               'body-checksum-digest': digest
        },
        ...headers,
                  ContentType: 'application/pdf'
}).promise();
}

Once our code updates are complete, we should re-deploy the Lambda function to see the transformation.

We first build the Lambda function deployment package using the following steps.

> npm run-script build

> @amzn/s3object-lambda-default-config-nodejs-function@0.1.0 build
> ./node_modules/.bin/esbuild src/s3objectlambda.ts --bundle --outfile=s3objectlambda.js --platform=node && eslint 'src/**/*.ts'

 s3objectlambda.js 1.9mb ⚠️

⚡ Done in 134ms

> npm run-script package

> @amzn/s3object-lambda-default-config-nodejs-function@0.1.0 package
> zip -ur release/s3objectlambda_deployment_package.zip s3objectlambda.js

updating: s3objectlambda.js (deflated 81%)

After updating the Lambda function deployment package, we upload it to the S3 bucket "image-watermarking-lambda."

> aws s3api put-object --bucket image-watermarking-lambda \
--key s3objectlambda_deployment_package.zip \
--body release/s3objectlambda_deployment_package.zip

{
"ETag": "abb36da118881a01ee2bd5f3b332bc14",
"VersionId": "pNBEMDStXlfsJpSKmBfGUhjTUwfcgqUl"
}

Now we will re-deploy the template so that the S3 Object Lambda Access Point can use the new function version. We navigate to the CloudFormation console and select our stack “image-watermarking-demo.”

Select Update, choose Use current template, then select on Next.

In the Specify stack details page, we enter the new Lambda function version id under the LambdaFunctionS3ObjectVersion parameter.

We do not modify the remaining options and choose Update stack. This option updates the S3 Object Lambda Access Point “watermarking-ol-ap” to use the updated Lambda function code.

Once the stack update is complete, we use the S3 Object Lambda Access Point “watermarking-ol-ap” to view the image. We navigate to the Object Lambda Access Points page on the S3 console and click on “watermarking-ol-ap.” This page lists the contents of the bucket, identical to the buckets page.

However, opening any image from the bucket will dynamically invoke your Object Lambda Access Point and Lambda function to transform the image. You can see that a watermark has been added to the image, and that the image was cropped.

Cleaning up

To avoid incurring future charges, you can delete the CloudFormation stack when it is no longer needed.

Conclusion

In this blog, we showed you how to use a CloudFormation template to automate your S3 Object Lambda configuration, which lets you focus on innovation and implementing business logic, instead of managing the operational and administrative activities of getting started. We then looked at a real-world scenario of using that CloudFormation template to deploy a Lambda function to add watermarks to images before data is returned to end users. This blog post describes the basic building blocks for the specific solution, but it can be extended relatively easily. For instance, you could add additional transformation logic that would first convert and resize an image before watermarking it. To do this, you can extend the Lambda function in the GitHub repository to implement your transformation. Then, use the CloudFormation template to create a new S3 Object Lambda Access Point.

To learn more about S3 Object Lambda, visit the S3 User Guide. We also recently published a blog that shows how you can use S3 Object Lambda to authorize requests before granting access to an object.

Thanks for reading this blog post. If you have any comments or questions, feel free to leave them in the comments section.