AWS Open Source Blog

Open Source at re:Invent 2017

AWS re:Invent 2017

If you’re attending re:Invent next week in Las Vegas, there will be plenty of opportunities to learn about and get hands-on with AWS and open source. The list below is not exhaustive; also see Your Guide to Machine Learning at re:Invent 2017 for more talks and workshops on Apache MXNet, Gluon, TensorFlow, and other machine learning tools and topics.

…and if you can’t attend re:Invent, stay tuned! Videos and slide decks will be out soon.

All About Open Source

ARC213 – Open Source at AWS—Contributions, Support, and Engagement

Zaheda Bhorat – Head of Open Source; Adrian Cockcroft – VP Cloud Architecture Strategy

At re:Invent 2016, Adrian Cockcroft talked about Open Source at AWS—Contributions, Support, and Engagement. He announced that he was creating an Open Source Community Engagement team, and Zaheda Bhorat had just joined AWS as Head of Open Source Strategy. What is AWS doing in open source? Come to this talk to learn more about our projects and how we can collaborate.

CON01 – Birds of a Feather: Containers and Open Source at AWS

Deepak Singh – GM, Container and HPC Services, Amazon Web Services; Arun Gupta – Principal Open Source Technologist, AWS; Patrick Chanezon – Chief Developer Advocate, Docker, Inc.; Kris Nova – Senior Developer Advocate, Heptio; Chris Aniszczyk – CTO, Cloud Native Computing Foundation

Open source is a core part of making cloud native possible for everyone. In this session, thought leaders from CNCF, Docker, the Kubernetes community, and AWS discuss the cloud’s direction for growth and enablement of the open source community. We also discuss how AWS is integrating open source code into our container services, and our contributions to open source projects.


Cloud Custodian Code Day

Join the Capital One DevExchange team at a unique gathering for Cloud Custodian users and developers. (Event open to any AWS re:Invent ticket holder with no additional charge.)


ABD202 – Best Practices for Building Serverless Big Data Applications

Ben Snively – Solutions Architect, AWS

Explore the concepts behind and benefits of serverless architectures for big data, when and how you can use serverless technologies to streamline data processing, minimize infrastructure management, improve agility and robustness, and share a reference architecture using a combination of cloud and open source technologies.

ABD403 – Best Practices for Distributed Machine Learning and Predictive Analytics Using Amazon EMR and Open-Source Tools

Keith Steward – Specialist SA (EMR, AI), AWS

Common use cases and design patterns for predictive analytics using Amazon Elastic Map Reduce. We address accessing data from a data lake, extraction and preprocessing with Apache Spark, analytics and machine learning code development with notebooks (Jupyter, Zeppelin), and data visualization using Amazon QuickSight, and other operational topics.

CON308 – Mastering Kubernetes on AWS

Arun Gupta – Principal Open Source Technologist, AWS; Raffaele Di Fazio – Software Engineer, Zalando

Arun Gupta, Principal Open Source Technologist at AWS and Raffaele Di Fazio, software engineer at leading European fashion platform Zalando, share insights and experiences running Kubernetes on AWS.

DAT401 – The Boss: A Petascale Database for Large-Scale Neuroscience

Mike Colson – Solutions Architect, AWS; Dean Kleissas – VP Engineering, Gigantum

The IARPA Machine Intelligence from Cortical Networks (MICrONS) program is a research endeavor created to improve neurally-plausible machine-learning algorithms by understanding data representations and learning rules used by the brain through structurally and functionally interrogating a cubic millimeter of mammalian neocortex. This effort requires efficiently storing, visualizing, and processing petabytes of neuroimaging data. The Johns Hopkins University Applied Physics Laboratory (APL) has developed an open-source, highly available service to manage these data, called the Boss, which uses AWS to provide a cloud-native spatial database with an innovative storage hierarchy and auto-scaling capability. We provide an overview of the Boss, and how the APL used Amazon DynamoDB, AWS Lambda, and AWS Step Functions for several high-throughput components of the system. We discuss both the challenges and successes with serverless technologies.

DEV315 – GitHub to AWS Lambda: Developing, Testing, and Deploying Serverless Apps

Bryan Cross – Sr. Solutions Engineer, GitHub  

A hands-on demo of how to use GitHub as the core of a DevOps toolchain. Learn how to leverage AWS integrations with Jenkins, the AWS CLI, and open source software to build, test, and deploy a service to AWS Lambda. (Session sponsored by GitHub, Inc.)

DEV332 – Using AWS to Achieve Both Autonomy and Governance at 3M

Nathan Scott – Senior Consultant, AWS; James Martin – Manager, Automation Engineering, 3M; Casey Lee – Chief Architect, Stelligent  

Nathan Scott, Senior Consultant at AWS and James Martin, Automation Engineering Manager at 3M, on how they have achieved both autonomy and governance through self-service automation tools on AWS. Includes a demo from Casey Lee, Chief Architect at Stelligent, on the tools used to accomplish this for 3M, including AWS Service Catalog, AWS CloudFormation, AWS CodePipeline and Cloud Custodian, an open source tool for managing AWS accounts.

DVC202 – Community Knowledge Sharing for AWS

Thanos Baskous – Chief Architect, Blackstone; Corey Quinn – Editor, Last Week in AWS; Joshua Levy – Co-founder, Open Guide to AWS; Manish Pandit – Director of Engineering, Marqeta

The Open Guide to AWS is an open source writing project which has become one of the most popular AWS resources on the web. It’s both a written resource on GitHub, with over 100 contributors, and a large Slack group. Each has become a forum for trading practical knowledge not covered in standard documentation. We talk about the Guide and how it started, share lessons on seeding initial content, the editorial process, and how to foster a healthy extended community and encourage social engagement.

DVC201 – Build AWS Skills Through Community-Led User Groups

Kittaya Kanchanobhas – Assistant to Managing Director, DailiTech; John Varghese – Cloud Steward, Intuit; Sebastian Montini – Engineering Manager, Medallia; Jason O’Conaill – International Director, AWS User Group

Did you know that there are over 300 AWS User Groups worldwide? Join this panel discussion featuring AWS community leaders from around the world, and learn the value of attending community-led AWS Meetups in your region. Community leaders share their experiences, talk through how local communities help developers solve problems and achieve their goals, and discuss the benefits of participating in peer-to-peer AWS knowledge sharing and networking activities.

The above two sessions are part of the re:Invent Developer Community Day, six community-led sessions where AWS enthusiasts share technical insights on trending topics based on first-hand experiences and knowledge shared within local AWS communities.

ENT318 – Leveraging a Cloud Policy Framework – From Zero to Well Governed

Chris Lyon – VP of Engineering, Mulesoft; Vikram Pillai – Chief Architect, Director of Engineering, CloudHealth Technologies

An open source “cloud policy framework” enables users to leverage a community that can help define and tune best practice policies, and help SaaS vendors and ISVs capture the best way to manage an application and share it with customers. (Session sponsored by CloudHealth Technologies)

LFS304 – Born in the AWS Cloud: How Eagle Genomics Uses AWS to Process Billions of DNA Sequence Reads

Angel Pizarro – TBDM, Research and Technical Computing, AWS; Raminderpal Singh – VP, Head of Microbiome Division, Eagle Genomics; Nick James – Senior Bioinformatician, Eagle Genomics

With the increasing use of genomic sequencing for scientific discovery, the rate-limiting step for researchers is not in obtaining genetic code, but in having the capacity for storage and computing power to analyze it. Learn how Eagle Genomics built a cloud solution that uses an open-source workflow engine (eHive, Docker containers to process jobs, and a REST service to manage pipeline runs, all to help customers process genetic sequences up to 20 times faster without additional costs.

MCL303 – Deep Learning with Apache MXNet and Gluon

Sunil Mallya – Sr AI Solutions Architect, AWS; Miro Enev – Sr. Solution Architect, Deep Learning, NVIDIA

Learn to program deep learning models using Gluon, the new intuitive, dynamic programming interface available for the Apache MXNet open-source framework. We’ll also explore neural network architectures such as multi-layer perceptrons, convolutional neural networks (CNNs), and LSTMs.

MCL403 – Building an Intelligent Multi-Modal User Agent with Voice, Natural Language Understanding, and Facial Animation

Keith Steward – Specialist SA (EMR, AI), AWS; Hira Niranjan – Amazon Lex Solutions Architect

Build an intelligent agent combining AI services and other offerings from AWS, and open source technologies like Blender, Apache MXNet, and CLIPS.

SRV302 – Building CI/CD Pipelines for Serverless Applications

Chris Munns – Senior Developer Advocate – Serverless, AWS; Ben Kehoe – Cloud Robotics Research Scientist, iRobot

A method for automating the deployment of serverless applications running on AWS Lambda, including how you can model and express serverless applications using the open-source AWS Serverless Application Model (AWS SAM).

SRV424 – Massively Parallel Data Processing with PyWren and AWS Lambda

Paras Bhuva – Sr. Solutions Architect, AWS; Olivier Klein – Head of Emerging Technologies, Asia-Pacific, AWS

How to achieve fast processing speeds using an open-source project called PyWren to massively parallelize data analytics jobs across hundreds or thousands of AWS Lambda functions.


CON213 – Hands-on Deployment of Kubernetes on AWS

Arun Gupta – Principal Open Source Technologist, AWS

Get hands-on experience using Kubernetes and Kops (Kubernetes Operations) to create, deploy, manage and scale a Kubernetes cluster on AWS. You’ll learn how to deploy your microservices-based applications and use service discovery for them.

DEV337 – Deploy a Data Lake with AWS CloudFormation

Chuck Meyer – Sr. Developer Advocate, AWS CloudFormation, AWS; Luis Colon – Senior Developer Advocate, AWS CloudFormation, AWS

Learn to build AWS CloudFormation templates using proven methods and best practices. Deploy a fully functional data lake architecture using AWS services like Amazon RDS and open source components like Apache Zeppelin.

GPSWKS301 – GPS: Comprehensive Big Data Analytics Architecture Made Easy

Luis Soto – Sr. Category GTM Lead, Amazon; Kim Schmidt – President & CEO,

A modern Big Data architecture involves extending your on-premises data management to AWS, implementing a data pipeline to stream real-time data into cloud data warehouse Amazon Redshift, perform data transformation, discovery, predictive analytics through machine learning, visualize complex information, and be notified to respond to business events. This session is for APN Consulting Partners and organizations looking for ways to accelerate and modernize their Big Data projects. You will learn how to deploy and integrate AWS Services with Third-party Solutions in AWS Marketplace. Reduce your time to market by combining AWS services, open source software and ready-to-run on AWS solutions.

MCL309 – Deep Learning on a Raspberry Pi

Julien Simon – Principal Technical Evangelist, AWS

We introduce you to the open source deep learning framework Apache MXNet, and show you how to install it on a Raspberry Pi. Then, using a camera and a pre-trained object detection model, we show real-life objects to the Pi and listen to what it thinks the objects are, thanks to the text-to-speech capabilities of Amazon Polly.

Deirdré Straughan

Deirdré Straughan

Deirdré has been communicating about technology, and helping others to do so, for 30 years. She has written one book (so far); edited two more (so far); produced and delivered technical training; produced hundreds of videos and live streams of technical talks; written, edited, and managed blogs; and managed events. She has been applying this skill set to cloud computing since 2010, and to open source for even longer. She joined AWS in 2017. You can find her at @deirdres on Twitter.