Featured re:Invent 2017 Sessions
Analyzing streaming data in real-time with Amazon Kinesis (ABD301)
Amazon Kinesis makes it easy to collect process and analyze real-time streaming data so you can get timely insights and react quickly to new information. In this session we present an end-to-end streaming data solution using Kinesis Streams for data ingestion Kinesis Analytics for real-time processing and Amazon Data Firehose for persistence. We review in detail how to write SQL queries using streaming data and discuss best practices to optimize and monitor your Kinesis Analytics applications. Lastly we discuss how to estimate the cost of the entire system.
Workshop: Building your first big data application on AWS (ABD317)
Want to ramp up your knowledge of AWS big data web services and launch your first big data application on the cloud? We walk you through simplifying big data processing as a data bus comprising ingest, store, process, and visualize. You build a big data application using AWS managed services, including Amazon Athena, Amazon Kinesis, Amazon DynamoDB, and Amazon S3. Along the way, we review architecture design patterns for big data applications and give you access to a take-home lab so that you can rebuild and customize the application yourself. You should bring your own laptop and have some familiarity with AWS services to get the most from this session.
Workshop: Don’t wait until tomorrow, how to use streaming data to gain real-time insights into your business (ABD321)
In recent years, there has been an explosive growth in the number of connected devices and real-time data sources. Because of this, data is being produced continuously and its production rate is accelerating. Businesses can no longer wait for hours or days to use this data. To gain the most valuable insights, they must use this data immediately so they can react quickly to new information. In this workshop, you learn how to take advantage of streaming data sources to analyze and react in near real-time. You are presented with several requirements for a real-world streaming data scenario and you're tasked with creating a solution that successfully satisfies the requirements using services such as Amazon Kinesis, AWS Lambda and Amazon SNS.
How Amazon Flex uses real-time analytics to deliver packages on time (ABD217)
Reducing the time to get actionable insights from data is important to all businesses and customers who employ batch data analytics tools are exploring the benefits of streaming analytics. Learn best practices to extend your architecture from data warehouses and databases to real-time solutions. Learn how to use Amazon Kinesis to get real-time data insights and integrate them with Amazon Aurora Amazon RDS Amazon Redshift and Amazon S3. The Amazon Flex team describes how they used streaming analytics in their Amazon Flex mobile app used by Amazon delivery drivers to deliver millions of packages each month on time. They discuss the architecture that enabled the move from a batch processing system to a real-time system overcoming the challenges of migrating existing batch data to streaming data and how to benefit from real-time analytics.
Real-time streaming applications on AWS: Use cases and patterns (ABD203)
To win in the marketplace and provide differentiated customer experiences, businesses need to be able to use live data in real time to facilitate fast decision making. In this session, you learn common streaming data processing use cases and architectures. First, we give an overview of streaming data and AWS streaming data capabilities. Next, we look at a few customer examples and their real-time streaming applications. Finally, we walk through common architectures and design patterns of top streaming data use cases.
Cox Automotive empowered to scale with Splunk Cloud & AWS (ABD208)
In this session learn how Cox Automotive is using Splunk Cloud for real time visibility into its AWS and hybrid environments to achieve near instantaneous MTTI reduce auction incidents by 90% and proactively predict outages. We also introduce a highly anticipated capability that allows you to ingest transform and analyze data in real time using Splunk and Amazon Data Firehose to gain valuable insights from your cloud resources. It's now quicker and easier than ever to gain access to analytics-driven infrastructure monitoring using Splunk Enterprise and Splunk Cloud.
Real-time log analytics using Amazon Data Firehose (Jun 2017)
Log analytics is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, gaming, and IoT. Moving your log analytics to real time can speed up your time to information allowing you to get insights in seconds or minutes instead of hours or days. In this session, you will learn how to ingest and deliver logs with no infrastructure using Amazon Data Firehose. We will show how Amazon Managed Service for Apache Flink can be used to process log data in real time to build responsive analytics. Finally, we will show how to use Amazon Elasticsearch Service to interactively query and visualize your log data.
- Understand how to easily build an end to end, real time log analytics solution.
- Get an overview of collecting and processing data in real-time using Amazon Kinesis.
- Learn how to Interactively query and visualize your log data using Amazon Elasticsearch Service.
Streaming ETL for data lakes using Amazon Data Firehose (May 2017)
Data lakes enable your employees across the organization to access and analyze massive amounts of unstructured and structured data from disparate data sources, many of which generate data continuously and rapidly. Making this data available in a timely fashion for analysis requires a streaming solution that can durably and cost-effectively ingest this data into your data lake. Amazon Data Firehose is a fully managed service that makes it easy to prepare and load streaming data into AWS. In this tech talk, we will provide an overview of Firehose and dive deep into how you can use the service to collect, transform, batch, compress, and load real-time streaming data into your Amazon S3 data lakes.
- Understand key requirements for collecting, preparing, and loading streaming data into data lakes.
- Get an overview of transmitting data using Firehose.
- Learn how to perform data transformations with Firehose.
How TrueCar gains actionable insights with Splunk Cloud
Moving your entire data center to the cloud is no easy feat! TrueCar’s technology platform team was tasked with just that—and in search of a more scalable monitoring and troubleshooting solution that could increase infrastructure and application performance, enhance its security posture, and drive product improvements. The company landed on Splunk Cloud running on AWS and deployed it in one day! In this webinar, you’ll learn how TrueCar leverages both AWS and Splunk capabilities to gain insights from its data in real time.
Watch the webinar to learn how TrueCar's experience running Splunk Cloud on AWS with Amazon Data Firehose can help you:
- Gain historical insights with additional data retention
- Provide better visibility into AWS billing
- Obtain security insights and threat detection
Amazon Data Firehose now supports dynamic partitioning to Amazon S3
by Jeremy Ber and Michael Greenshtein, 09/02/2021
CloudWatch Metric Streams – Send AWS Metrics to Partners and to Your Apps in Real Time
by Jeff Barr, 03/31/2021
Stream, transform, and analyze XML data in real time with Amazon Kinesis, AWS Lambda, and Amazon Redshift
by Sakti Mishra, 08/18/2020
Amazon Data Firehose Data Transformation with AWS Lambda
by Bryan Liston, 02/13/2027
Watch Stream CDC into an Amazon S3 data lake in Parquet format with AWS DMS
by Viral Shah, 09/08/2020
Amazon Data Firehose custom prefixes for Amazon S3 objects
by Rajeev Chakrabarti, 04/22/2019
Stream data to an HTTP endpoint with Amazon Data Firehose
by Imtiaz Sayed and Masudur Rahaman Sayem, 06/29/2020
Capturing Data Changes in Amazon Aurora Using AWS Lambda
by Re Alvarez-Parmar, 09/05/2017
How to Stream Data from Amazon DynamoDB to Amazon Aurora using AWS Lambda and Amazon Data Firehose
by Aravind Kodandaramaiah, 05/04/2017
Analyzing VPC Flow Logs using Amazon Athena, and Amazon QuickSight
by Ian Robinson, Chaitanya Shah, and Ben Snively, 03/09/2017