Step Functions can help ensure that ETL or machine learning jobs execute in order and complete successfully.
Step Functions can provide auditable automation of routine deployments, upgrades, installations, and migrations.
Use Step Functions to connect and coordinate with other AWS services and your own distributed components.
From small production agencies to major broadcast media companies, video teams of all sizes rely on Frame.io to streamline their video post-production process. To deliver reliable video processing at scale, Frame.io used AWS Step Functions to transcode and process high volumes of media files as cost-effectively as possible. Their workflow decides whether to run transcodes on AWS Lambda or AWS ECS, depending on the duration and size of the media file. Frame.io has seen performance gains and a 20x increase in code releases.
“With Step Functions managing integration, we can easily grow and adapt our workflow to the ever changing needs of working with the agriculture industry and thereby help improve farm productivity and profitability.”
—Zac Oler, Principal Software Engineer, Granular
More than 7 billion people depend on farms for healthy and affordable food, and Granular’s mission is to continually innovate new ways to help farmers use technology to run stronger businesses. To deliver on this demand, Granular provides software and analytics tools that help make farming more profitable, efficient, and sustainable. Their platform includes a farm management software suite, a farmland real estate analysis tool, and precision agronomy software for ag retailers, and supports growers farming many millions of acres of crops annually across the United States, Canada and Australia.
Granular has had a long history of expertise in processing high density geospatial sensor data from agriculture machinery. Looking to improve customer experience and retention, Granular wanted to develop a high performance, high speed, scalable technique for running translations of agricultural machine data so that their other products could consume that data as soon as possible. In the past, Granular used home grown Windows solutions for managing the workflow and processing of these massive data sets, but they now wanted to write additional components using Kubernetes and AWS Lambda for easier scaling and management. Realizing that writing their own workflow automation tools would take focus away from business problems, they looked to AWS Step Functions to provide a serverless workflow solution.
Using Step Functions, Granular simplified their core processing framework with an easy to understand, self-documenting workflow and improved the scaling and management of their existing application. “We were able to implement a scalable approach to running code through multiple technology stacks, and can now manage the state of our workflows across a mixed platform environment using Linux, Windows, AWS Lambda, and Amazon EC2,” says Zac Oler, Principal Software Engineer at Granular.
Home24 is a European market leader in online shopping for home and living products. They run their entire data analytics platform on AWS, analyzing multiple data sources including marketing data, social media, search analytics, and internal ERP systems. They used AWS Step Functions to build a flexible, long-running serverless ELT pipeline. Their data engineers can now quickly integrate new data sources and modify pipelines to perform new transformations and analytics. The platform can also withstand API failures since Step Functions automatically retries if service APIs are unavailable.
“Using AWS Step Functions as part of the Xylem Data Lake has transformed our pace of innovation. We can now develop internal and external analytics projects with fewer handoffs to other engineers, fewer interrupts with DevOps, and with more technologies than we ever thought possible.”
—Mitchell Hensley, Vice President, Software Strategy, Xylem
Xylem is a leading global water technology company committed to developing innovative technology solutions to the world’s water challenges. To perform analytics, Xylem was using a Hadoop-based, on-premises data lake to store massive amounts of electric, water, and gas metering data. However, this solution limited their data analysis capabilities and was not flexible enough to allow for the addition of new projects. Data processing was taking four to five hours each day, and if they added new customers, up to several days.
By moving the Xylem Data Lake to an AWS-based platform leveraging AWS Batch, AWS Step Functions, and AWS Lambda, the Xylem data team was able to dramatically expand their data analytics capabilities. Xylem feeds each input file to a Step Functions state machine, which then validates the file, tracks metadata and finally, launches and monitors an AWS Batch job that processes the data. Each state machine runs independently and in parallel. The AWS Batch jobs allocate containers automatically, and Xylem can now parse thousands of files simultaneously, without having to worry about jobs competing for resources. As a result, Xylem has reduced data processing time from 20 hours to only two hours – regardless of the size of the dataset.
Thomson Reuters Corporation is a Canadian multinational mass media and information firm. Like many media companies, they have a high volume of incoming video to process. They needed to transcode over 350 news video clips per day into 14 formats each. Being a trusted provider of news and information, they also needed to do this as quickly as possible. Their files are all in the same format, but they vary wildly in length. To tackle this wide variance, Thomson Reuters developed a serverless split video transcoding application using Step Functions. They can now process video segments in parallel, and are able to process a typical video in 2 minutes instead of 20 minutes.
Coinbase Engineering Blog
"If a team focuses on building quality automation into their workflows, stability follows. By building quality automation with services like AWS Step Functions, Coinbase avoids trading speed for stability, and instead we get both."
—Graham Jenson, Infrastructure Engineer, Coinbase
Coinbase is a secure digital currency exchange platform that makes it easy to buy, sell, and store cryptocurrency like Bitcoin, Ethereum, and Litecoin. As of 2017, Coinbase is the largest bitcoin broker in the world and serves users in 33 countries. Coinbase has a high performing engineering organization. They deploy hundreds of times per day across hundreds of projects, release new features from idea to launch in under an hour, and sustain low failure rates. This is possible because most of their change management and deployment processes are automated and their engineers have adopted a DevOps culture.
A key part of that automation is their secure, automated deployment pipeline, Odin. Odin takes a description of a project release and then safely and securely launches it into AWS. Coinbase automated Odin using AWS Step Functions, allowing them to deploy 12-factor-applications to AWS easily, reliably and securely. Step Functions provided the tools for reliability and end-to-end visibility out of the box, with a serverless operating model that doesn’t require any infrastructure to be provisioned or managed. Designing Odin as a Step Functions state machine lets Coinbase visually monitor the progress of the deployment, and audit each deployment step by step. With new insight into the reasons why services were failing, Coinbase was able to improve their rate of successful mission critical deployments from 90% to 97%. The visibility provided to engineers allowed them to diagnose and resolve their own issues quickly, and reduced the number of support tickets related to unsuccessful deployments.
Read more of Graham's blogs or view the Coinbase Odin deployer to AWS for 12 Factor applications on GitHub.
“Using services like Amazon SageMaker and AWS Step Functions to automate model deployment has enabled us to deliver recurring, quality, mostly-automated models to the business with minimal human intervention and overhead.”
—Jeremy Irwin, Lead Solutions Architect, Cox Automotive
Cox Automotive Inc. makes buying, selling, owning and using cars easier. To support digital advertising, they wanted to increase the volume and quality of their machine learning prediction models. However, they were concerned that their models might drift over time, leading to poor decisions. Injecting Data Scientist oversight into the model retraining process was the best solution, but the challenge was to to do so without slowing down the Decision Science team.
Cox Automotive decided to automate their ML model delivery workflow with AWS, including a model approval pipeline coordinated by AWS Step Functions. Once models are retrained using Amazon SageMaker, the output artifacts and diagnostics are uploaded to Amazon S3, which triggers Step Functions to run a model approval workflow. An email is sent to a Data Scientist, who reviews the model and clicks either “Approve” or “Reject”. If approved, Step Functions sends the artifacts to a destination S3 bucket along with an Amazon SNS notification indicating that the new model is ready for consumption.
Investing in automation has given both teams time to work on higher value activities. Cox Automotive now has a reusable, automated process that allows them to spend more time making more accurate models, and less time building deployment pipelines.
Outsystems offers a low-code visual development platform for rapid application development. They wanted to ensure that their customers wouldn’t run out of disk space—without having to over-provision storage. To do this, Outsystems developed an automatic resizing process with AWS Step Functions, which they exposed as a serverless microservice behind Amazon API Gateway. This made it possible for Outsystems to change their default database from 1024 GB to 400 GB, reducing storage costs by 60%. These databases could then be resized as needed to meet customer demand.
Yelp is focused on local businesses - they’re around to help you find what you're looking for, whatever city you happen to be in. Yelp wanted to modernize a business-critical subscription billing process. The code had grown over the past 10 years, so refactoring seemed hard, expensive and risky. Yelp used Step Functions to approach the transition as a series of small, incremental steps. First, Yelp simply moved the entire process inside Step Functions, which immediately gave them the benefits of a well-defined API and built-in error handling. Over time, Yelp continued to gradually break down the code into more friendly “bite-sized” pieces, and was able to move one task at a time into serverless AWS Lambda functions or containers with minimal effort and no change to code.
When the Food and Drug Administration in the United States published new requirements for nutrition fact panels, there was some significant changes. This had a big impact for Coca-Cola, and meant that a very large number of nutritional fact panels needed to be updated. Using their legacy platform, updates could take 18 hours or more, only to need corrections which required additional time to trace down and resolve. The visual workflows in Step Functions allowed the nutritional analysis teams to sit down with the developers and exchange ideas. Observing the workflow execute in real time allowed the two teams to identify optimizations they hadn’t identified before, and Coca-Cola was able to shorten the processing time from 36 hours down to 10 seconds.