Atlassian migrated from an extract-and-load architecture to a streaming data architecture using Amazon Kinesis, Amazon S3, and AWS Lambda to ingest over a billion events every day into its data lake. With this new architecture, Atlassian has increased its analytics system uptime to 99.95%, reliability to 99.95%, and decreased the time from when an event is sent to the data lake to when it’s available for analytics to 500 milliseconds. As a result, teams across Atlassian have adopted the data lake, and publish and consume data in a self-serve manner.
In this recorded session from re:Invent 2017, we dive deep into assembling a data lake using Amazon S3, Amazon Kinesis, Amazon Athena, Amazon EMR, and AWS Glue. The session features Mohit Rao, Architect and Integration lead at Atlassian, the maker of products such as JIRA, Confluence, and Stride. First, we look at a couple of common architectures for building a data lake. Then we show how Atlassian built a self-service data lake, where any team within the company can publish a dataset to be consumed by a broad set of users.
Get started with Amazon Kinesis