AWS Open Source Blog
Open Source at re:Invent 2018
re:Invent 2018 is coming soon, with so many open source-related sessions that we barely have room to mention them all here! Across almost every topic area, there are sessions about AWS open source projects and services, as well as builder sessions, chalk talks, and more on how to use a wide variety of open source tools in your AWS stack.
So as not to make this post too overwhelming, the descriptions below are heavily edited for brevity and to emphasize their open source aspects. Please click the session links to see full descriptions, speakers, and date/time/venue information.
IMPORTANT: This list may be expanded and updated before, during, and after re:Invent. Also check the re:Invent catalog for changes in date/time/venue, and newly-added sessions and repeat sessions.
AI/ML
AIM204-S – Smarter Event-Driven Edge with Amazon SageMaker & Project Flogo Model inferencing using deep learning frameworks directly onto edge devices or onto AWS Lambda using Project Flogo, an open-source event-driven framework.
AIM301 – Deep Learning for Developers: An Introduction, Featuring Samsung SDS Hear from Samsung SDS about how it developed a deep-learning model for cardiac arrhythmia detection using Apache MXNet, an open-source deep-learning framework.
AIM302 – Machine Learning at the Edge Train machine learning models using Amazon SageMaker and deploy them to an edge device using AWS Greengrass, enabling you process data quickly at the edge, even when there is no connectivity.
AIM314 – Create a “Question and Answer” Bot with Amazon Lex and Amazon Alexa using two open-source projects: QnABot and Lex-Web-UI.
AIM329 – Amazon SageMaker and Chainer: Tips & Tricks Use Chainer, an open-source deep learning framework written in Python, in Amazon SageMaker.
AIM350 – Bring Your Own Apache MXNet and TensorFlow Scripts to Amazon SageMaker
AIM401-R – Deep Learning Applications Using TensorFlow This code-level session also includes tutorials and examples using TensorFlow.
AIM403-R Integrate Amazon SageMaker with Apache Spark, ft. Moody’s Amazon SageMaker includes an Apache Spark library that you can use to easily train models from your Spark clusters. In this code-level session, integrate your Apache Spark application with Amazon SageMaker, then dive deep into starting training jobs from Spark, integrating training jobs in Spark pipelines, and more.
AIM407-R – Build Deep Learning Applications Using Apache MXNet, Featuring Workday How Workday built computer vision and natural language processing (NLP) models using MXNet to automatically extract information from paper documents. Workday also shares its experience using Sockeye, an MXNet toolkit for quickly prototyping sequence-to-sequence NLP models.
AIM412 – Automatic Model Tuning Using Amazon SageMaker In many cases, what separates good models from great ones is the choice of hyperparameters. For example, what is the number of layers you should use; what should be the learning rate; what should be regularization parameters, and so on. Learn how Amazon SageMaker makes discovering the best set of hyperparameters an informed process during training.
AIM414 – Sequence-to-Sequence Modeling with Apache MXNet, Sockeye, and Amazon SageMaker The “encoder-decoder architecture with attention,” a state-of-the-art architecture for natural language processing, implemented in Sockeye and used by the sequence-to-sequence algorithm of Amazon SageMaker.
AIM418 – Build Deep Learning Applications Using MXNet and Amazon SageMaker Deploy machine learning models at scale quickly and easily. Build a model for a computer vision use case, quickly train it to get the best possible results, then easily deploy it to production using Amazon SageMaker.
AIM431 – Deep Learning-Based Text-to-Speech Synthesis with MXNet Learn how to build TTS systems with deep learning techniques for multiple voices using the Gluon interface, an open source library in Apache MXNet.
See also…
The re:Invent Guide: Machine Learning from Lynn Langit and Your Guide to AI and Machine Learning at re:Invent 2018 on the Machine Learning blog. And search the catalog for MXNet, TensorFlow, SageMaker…
Analytics
ANT329 – TIBCO: AI and Data Science Innovation with Amazon SageMaker …utilizing Amazon SageMaker, TIBCO Connected Intelligence Cloud, and open source—with at-scale, in-database compute; visual composition and notebooks; Slack-style collaboration among users; and model lifecycle deployment via low-code tooling such as TIBCO Live Apps…
ANT338 – ML Best Practices: Prepare Data, Build Models, and Manage Lifecycle …Learn to use Apache Spark for data preparation as it unifies data at massive scale across various sources. Train models using TensorFlow, and use MLflow to track experiment runs between multiple users within a reproducible environment…
ANT349 – Increase Your Data Engineering Productivity: Use Jupyter Notebooks with Amazon EMR (builders session) Jupyter Notebooks give big data customers an easy way to leverage the open source Apache Spark in Amazon EMR for data science, interactive analytics, and to prototype jobs before bringing them to production.
ANT387 – How to Use Jupyter Notebooks with Amazon EMR for Better Productivity (chalk talk)
Automotive
AMT302 – Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service …Avis Budget Group (ABG) and Slalom walk through their serverless mobility platform using the AWS connected vehicle reference architecture, Amazon SageMaker, Amazon Kinesis Data Analytics, and AWS Lambda… as well as the operations dashboard ABG created with Uber’s open source framework, deck.gl.
Compute
CMP203-R – Amazon Linux 2: A Stable, Secure, High-Performance Linux Environment The next-generation Amazon Linux operating system that now comes with five years of support.
CMP207 – AWS Compute Leadership Session: What’s New in Amazon EC2, Containers, and Serverless In this keynote, we announce new compute capabilities, and share insights into what makes the AWS compute business unique. We also announce new capabilities for Amazon EC2 instances, EC2 networking, EC2 Spot Instances, Amazon Lightsail, Containers, and Serverless.
CMP310 – Application Portability with Kubernetes Implement a hybrid cloud strategy and deploy the same application onto a VKE-managed Kubernetes cluster using Helm. Best practices on end-to-end network configuration automation and application monitoring with open-source tooling like Prometheus and Grafana.
CMP348 – Distributed, Incremental Dataflow Processing on AWS with GRAIL’s Reflow GRAIL’s open-source, cloud-based batch processing system, Reflow, leverages Amazon EC2, Amazon S3, and Amazon DynamoDB to support the large-scale, high-throughput, and cost-efficient data analysis that enables GRAIL’s research and development efforts.
CMP376 – Another Week, Another Million Containers on Amazon EC2 Netflix’s container management platform, Titus, offers a convenient model for managing compute resources, enables developers to maintain just their application artifacts, and provides a consistent developer experience from a developer’s laptop to production by leveraging Netflix container-focused engineering tools.
CMP416 – Build High-Performance, Cloud-Native, Open-Source Apps on AWS & Save How Bayer transitioned from on-premises packaged software architecture to open-source software and cloud services from AWS to build a modern, scalable, high-performance, open-source app on AWS. Learn about the open-source application architecture and AWS services used.
Containers
There are many sessions about containers this year, across multiple topics. View the Containers topic in the re:Invent catalog, but also search on “containers,” and see this handy re:Invent Guide: Containers by Tung Nguyen.
CON362 – Container Power Hour with Jess, Clare, and Abby Join Jess Frazelle, from GitHub and Microsoft, and Clare Liguori and Abby Fuller, from AWS, for a container power hour to kick off your re:Invent. In this session, learn how to use Git and GitHub to run your containers, and build, test, and deploy processes. GitOps and Actions and AWS Fargate—oh my! This session features a demo from Jess on using the new GitHub Actions to deploy to Fargate.
CON404-R – Care and Feeding of Amazon Linux Amazon Linux 2 provides a Linux distribution that integrates well with AWS, but the process from patch to package to push is not necessarily intuitive to our customers. Iliana Weller discusses the lifecycle of changes making their way into Amazon Linux 2 and the technical details of how we safely deploy repositories and images globally.
Content Delivery
CTD415 – Customizing Content Delivery with Lambda@Edge Hands-on session on using AWS Lambda@Edge and Amazon CloudFront to deliver high-performance and personalized experience to your internet users across the globe. Walk away with a working setup of combining Amazon S3, Amazon DynamoDB, and CloudFront with Lambda@Edge to build websites simultaneously hosted across the world… Reduce origin costs by rendering web page content at the edge with open source libraries such as jQuery and Mustache…
Demo
DEM47 – Run Everything-as-a-Service Everywhere Mesosphere answers: “How can we operate the open source frameworks of our choice, such as Spark, TensorFlow, HDFS, and more, as a service across hybrid clouds?”
DevOps
DEV308-R – Earn Your DevOps Black Belt: Deployment Scenarios with AWS CloudFormation Use cases combining AWS CloudFormation with build and pipeline automation tools to achieve repeatable, consistent and compliant deployments without sacrificing agility. This session will also cover cfn-lint, mu, and SAM.
DEV321-R – What’s New with AWS CloudFormation Latest improvements and best practices for AWS CloudFormation, spaning many use cases, including programmability options, cross-region and cross-account automation, operational safety, and additional integration with many other AWS services. Will also cover cfn-lint.
DEV322-R – What’s New with the AWS CLI Come hear from the core developers of the AWS CLI as we highlight some of the new features and major improvements in AWS CLI v2.
DEV325-R – Hands-On in the AWS Java Ecosystem Using real code examples, we’ll build a serverless application that makes use of the SDKs new HTTP2-based event-streaming APIs and deploy it using AWS Java tooling introduced in 2018. You’ll learn what’s new in 2.0 and the benefits of upgrading, as well as how to take advantage of new tooling in AWS’s already rich Java ecosystem.
DEV327 – Beyond the Basics: Advanced Infrastructure as Code Programming on AWS Advanced use cases of declarative, imperative, and mixed coding scenarios that cloud infrastructure developers can leverage. Examples include demonstrating how to create custom resources and leveraging transforms, like the AWS Serverless Application Model (AWS SAM), to create both simple and complex macros with AWS CloudFormation. Includes discussion of troposphere.
DEV333-R – .NET on AWS: Problem Solving with AWS Software Engineers Experts from the .NET SDK engineering team would love to help answer your questions related to the .NET experience on AWS, including the SDK for .NET, AWS Lambda with .NET, AWS Lambda support for PowerShell, and AWS Tools for Visual Studio Team Services (VSTS).
DEV337-R – Set Up a Serverless App Using React and AWS Amplify Build a fully functional and secure cloud-native application in hours. This demo includes AWS Cloud9, a cloud-based IDE, to build a fully serverless cloud native application in React using the AWS Mobile Hub via the AWS Mobile CLI and AWS Amplify.
DEV341-R – Improve Your Security Posture with AWS CloudFormation How to leverage guidelines from the Center of Internet Security (CIS), how to augment security checks, and how to build and secure AWS resources with additional tools. Includes a section on prowler.
DEV411-R – Infrastructure as Code: AWS Best Practices …how to define and deploy cloud infrastructure using aws-cdk (currently in developer preview on GitHub), which enables you to use object-oriented concepts to define and deploy AWS infrastructure…
DEV427-R – Deploy Serverless Apps with Python: AWS Chalice Deep Dive AWS Chalice is a microframework for writing serverless applications in Python. We take a sample Chalice application and walk through how Chalice builds and deploys it, including the deployment planner, automatic IAM policy generation, and how Chalice builds AWS Lambda deployment packages for Python. Hear from the core Chalice developers about best practices and how to get the out most of the microframework. Chalk talk designed for Python developers who have experience with serverless and want tools to improve their development processes.
Global Partner Summit
GPSTEC405 – Optimize Your SaaS Offering with Serverless Microservices Crack open the IDE and transform a SaaS web app comprised of several monolithic single-tenant environments into an efficient, scalable, and secure multi-tenant SaaS platform using ReactJS and NodeJS serverless microservices. Use Amazon API Gateway and Amazon Cognito… Enforce tenant isolation and data partitioning with OIDC’s JWT tokens. Leverage AWS SAM and AWS Amplify to simplify authoring, testing, debugging, and deploying serverless microservices.
GPSWS404 – AWS Greengrass, Containers, and Your Dev Process for Edge Apps Build and deploy a Greengrass Core manually, then get hands-on experience with a new open source tool that lets you iterate on your applications faster than ever. Finally, we ask for feedback on how the tool meets your needs.
GPSWS405 – Serverless AI with Scikit-learn Take advantage of serverless technologies for artificial intelligence (AI) by making a prediction on-the-fly, with no model hosting and no servers to maintain. How to train a model in Scikit-learn, an open source machine learning library for Python, then load and call the trained model from an AWS Lambda function, and how to load the library and send the data for prediction.
IoT
IOT208-R – Amazon FreeRTOS: IoT Operating System for Microcontrollers As OEMs work to squeeze more functionality onto cheaper and smaller IoT devices, they face challenges in development and operations that result in security vulnerabilities, inefficient code, compatibility issues, and unclear licensing. With Amazon FreeRTOS, it is now easier to build, deploy, and update connected microcontroller-based devices quickly and economically, while retaining confidence that the devices are secure.
IOT315 – Challenges of Embedded IoT Development and How Amazon FreeRTOS is Changing the Scene
…and more sessions on Amazon FreeRTOS.
Media & Entertainment
MAE301 – Dissecting Media Asset Management Architecture and Media Archive TCO A baseline process for moving content into a Media Asset Management system in AWS, leveraging common open source tools for asset registry and using AWS ML services to augment the metadata associated with assets…
MAE309 – Build an AWS Analytics Solution to Monitor the Video Streaming Experience Build and deploy an end-to-end analytics solution for monitoring the video streaming experience. Integrate an open source video player with Amazon Kinesis Data Streams to capture events in real time…
Mobile
MOB201 – From Idea to Customers: Developing Modern Cloud-Enabled Apps with AWS The process of taking your app idea to reality, with all the infrastructure you need to understand along the way. How AWS platform services and SDKs can help you get to a quality release faster, and then scale for success with serverless technologies. How you can build a scalable production-ready app quickly with GraphQL and machine language capabilities.
MOB316-R – Introduction to GraphQL GraphQL is the mechanism by which mobile and web apps can communicate with AWS AppSync to easily query and mutate data using as small a request as possible, enabling battery- and bandwidth-efficient communication. We go through the basics of GraphQL and answer your questions about it.
MOB318-R – Give REST a Rest: Easily Migrate Your APIs to GraphQL We answer your questions on how to migrate your RESTful APIs to GraphQL easily while still maintaining backwards compatibility.
MOB323 – All About the Customer: GraphQL & Real-Time Subscriptions in Customer Service Apps
Follow ALDO’s journey in adopting GraphQL and serverless technologies for their in-store modern apps, offering elevated real-time, personalized customer experiences while optimizing in-store retail operations.
MOB401 – Ten Tips And Tricks for Improving Your GraphQL API with AWS AppSync
Tips, tricks, and best practices for running serverless GraphQL APIs reliably on AWS, including versioning, multiple environments, CI/CD, advanced schema design, monitoring, alerting, and advanced search scenarios.
MOB402 – Authentication & Authorization in GraphQL with AWS AppSync Some common use cases and design patterns to secure user data in a globally-available GraphQL API, and discuss best practices for authentication and authorization in AWS AppSync.
MOB403-R – Migrating Your App from REST to GraphQL Migrate an app that uses REST and RPC APIs to GraphQL on AWS AppSync, and discuss the best practices to follow for the migration.
MOB404-R – Advanced GraphQL Design Patterns with AWS AppSync A well-designed GraphQL schema goes a long way towards ensuring a performant API. Dive deep into the GraphQL execution process, then design optimal schemas for use with AWS AppSync, native resolvers, and subscriptions that work for common integration scenarios. Best practices related to authentication and identity with GraphQL.
MOB420-R – Best Practices for Designing GraphQL APIs That Scale
Security
SEC325-R – Data Protection: Encryption, Availability, Resiliency, and Durability Full range of data protection capabilities provided by AWS, including best practices for using the AWS Encryption SDK with KMS for custom software development.
SEC329 – AWS Encryption SDK: The Busy Engineer’s Guide to Client-Side Encryption Client-side encryption options and implementing client-side encryption using the AWS Encryption SDK in Java, Python, and C. Bring a laptop, and be sure that you have an active AWS account with Administrator privileges before the workshop.
SEC330 – Automating Compliance Certification with Automated Mathematical Proof How auditors can independently validate design and operating effectiveness using open source and community-validated tools.
SEC378 – Securely Deploying at Scale Gain familiarity with DevSecOps best practices, how to ensure that only compliant code gets deployed into production based on your AWS Organization, and how AWS services combined with open source technology can help you achieve this.
SEC403 – Five New Security Automations Using AWS Security Services & Open Source The actual code behind various security automation and remediation functions. We demonstrate each script, describe the use cases, and perform a code review explaining the various challenges and solutions…
Serverless
SRV224-R – AWS SAM: From Local Debugging to CI/CD Start a simple serverless project locally and debug our AWS Lambda functions using AWS Serverless Application Model (AWS SAM) Local. Once it is prepared to go into AWS, we create a CI/CD pipeline for our new serverless application using AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and AWS CloudFormation.
SRV225-R – Serverless REST API In this builders session, learn how to develop your first serverless REST API using AWS Lambda and AWS Serverless Application Model (AWS SAM) Local.
SRV227 – Build a Heavily Trafficked Web App Learn how to build a serverless website that automatically scales using services such as AWS Lambda, Amazon API Gateway, and Amazon S3. We review frameworks that can help you build serverless applications, such as the AWS Serverless Application Model (AWS SAM).
SRV307-R – Building Serverless Applications with GraphQL and Graph Databases Build a data driven, serverless calorie tracker application with real-time, offline, and data syncing capabilities… The application integrates with AWS AppSync to provide real-time data from multiple data sources through GraphQL technology as well as offline capability.
SRV343-R – Best Practices for Safe Deployments on AWS Lambda and Amazon API Gateway There are multiple features available in AWS Lambda, Amazon API Gateway, and AWS Serverless Application Model (AWS SAM) that you can use to implement a continuous deployment pipeline with safe deployment strategies, such as canary releases. We review the options applied to different scenarios, such as microservices architectures, chaos engineering, and A/B testing to discover the best practices for your use cases, and discuss how teams at Capital One created a deployment pipeline using AWS SAM and AWS CodeDeploy for canary deployments for the Capital One Credit Offers API, built on Golang and AWS Lambda.
SRV355 – Best Practices for CI/CD with AWS Lambda and Amazon API Gateway A method for automating the deployment of serverless applications running on AWS Lambda. First, we cover how you can model and express serverless applications using the open source AWS Serverless Application Model (AWS SAM). Then, we discuss how you can use CI/CD tooling from AWS CodePipeline and AWS CodeBuild, and how to bootstrap the entire toolset using AWS CodeStar.
SRV372-R – Build Web Apps with GraphQL, AWS AppSync, and AWS Lambda …Learn to power your applications using AWS AppSync, how GraphQL APIs differ from REST-based APIs… We show you how GraphQL technology can be used to provide real-time data from multiple sources, such as Amazon DynamoDB, Lambda, and HTTP endpoints.
Storage
STG204 – Data Lake Implementation: Processing and Querying Data in Place Best practices for building a data lake in Amazon S3 and Amazon Glacier, leveraging an array of AWS, open source, and third-party analytics tools.
STG313 – Get Your Data into Amazon S3 & Amazon EFS Fast – Starting Today There are many different ways to get your file-based data into Amazon S3 and Amazon EFS, from AWS command line tools, to scripts using open source tools, to proprietary commercial tools built for media workflows. Start using best practices to improve data transfer performance, including scaling out parallel streams and automating workflows. Get hands-on with your environment in this session with guidance from an AWS expert.