AWS Open Source Blog
Open Source at re:Invent 2017
If you’re attending re:Invent next week in Las Vegas, there will be plenty of opportunities to learn about and get hands-on with AWS and open source. The list below is not exhaustive; also see Your Guide to Machine Learning at re:Invent 2017 for more talks and workshops on Apache MXNet, Gluon, TensorFlow, and other machine learning tools and topics.
…and if you can’t attend re:Invent, stay tuned! Videos and slide decks will be out soon.
All About Open Source
ARC213 – Open Source at AWS—Contributions, Support, and Engagement
Zaheda Bhorat – Head of Open Source; Adrian Cockcroft – VP Cloud Architecture Strategy
At re:Invent 2016, Adrian Cockcroft talked about Open Source at AWS—Contributions, Support, and Engagement. He announced that he was creating an Open Source Community Engagement team, and Zaheda Bhorat had just joined AWS as Head of Open Source Strategy. What is AWS doing in open source? Come to this talk to learn more about our projects and how we can collaborate.
CON01 – Birds of a Feather: Containers and Open Source at AWS
Deepak Singh – GM, Container and HPC Services, Amazon Web Services; Arun Gupta – Principal Open Source Technologist, AWS; Patrick Chanezon – Chief Developer Advocate, Docker, Inc.; Kris Nova – Senior Developer Advocate, Heptio; Chris Aniszczyk – CTO, Cloud Native Computing Foundation
Open source is a core part of making cloud native possible for everyone. In this session, thought leaders from CNCF, Docker, the Kubernetes community, and AWS discuss the cloud’s direction for growth and enablement of the open source community. We also discuss how AWS is integrating open source code into our container services, and our contributions to open source projects.
Hackathon
Join the Capital One DevExchange team at a unique gathering for Cloud Custodian users and developers. (Event open to any AWS re:Invent ticket holder with no additional charge.)
Sessions
ABD202 – Best Practices for Building Serverless Big Data Applications
Ben Snively – Solutions Architect, AWS
Explore the concepts behind and benefits of serverless architectures for big data, when and how you can use serverless technologies to streamline data processing, minimize infrastructure management, improve agility and robustness, and share a reference architecture using a combination of cloud and open source technologies.
Keith Steward – Specialist SA (EMR, AI), AWS
Common use cases and design patterns for predictive analytics using Amazon Elastic Map Reduce. We address accessing data from a data lake, extraction and preprocessing with Apache Spark, analytics and machine learning code development with notebooks (Jupyter, Zeppelin), and data visualization using Amazon QuickSight, and other operational topics.
CON308 – Mastering Kubernetes on AWS
Arun Gupta – Principal Open Source Technologist, AWS; Raffaele Di Fazio – Software Engineer, Zalando
Arun Gupta, Principal Open Source Technologist at AWS and Raffaele Di Fazio, software engineer at leading European fashion platform Zalando, share insights and experiences running Kubernetes on AWS.
DAT401 – The Boss: A Petascale Database for Large-Scale Neuroscience
Mike Colson – Solutions Architect, AWS; Dean Kleissas – VP Engineering, Gigantum
The IARPA Machine Intelligence from Cortical Networks (MICrONS) program is a research endeavor created to improve neurally-plausible machine-learning algorithms by understanding data representations and learning rules used by the brain through structurally and functionally interrogating a cubic millimeter of mammalian neocortex. This effort requires efficiently storing, visualizing, and processing petabytes of neuroimaging data. The Johns Hopkins University Applied Physics Laboratory (APL) has developed an open-source, highly available service to manage these data, called the Boss, which uses AWS to provide a cloud-native spatial database with an innovative storage hierarchy and auto-scaling capability. We provide an overview of the Boss, and how the APL used Amazon DynamoDB, AWS Lambda, and AWS Step Functions for several high-throughput components of the system. We discuss both the challenges and successes with serverless technologies.
DEV315 – GitHub to AWS Lambda: Developing, Testing, and Deploying Serverless Apps
Bryan Cross – Sr. Solutions Engineer, GitHub
A hands-on demo of how to use GitHub as the core of a DevOps toolchain. Learn how to leverage AWS integrations with Jenkins, the AWS CLI, and open source software to build, test, and deploy a service to AWS Lambda. (Session sponsored by GitHub, Inc.)
DEV332 – Using AWS to Achieve Both Autonomy and Governance at 3M
Nathan Scott – Senior Consultant, AWS; James Martin – Manager, Automation Engineering, 3M; Casey Lee – Chief Architect, Stelligent
Nathan Scott, Senior Consultant at AWS and James Martin, Automation Engineering Manager at 3M, on how they have achieved both autonomy and governance through self-service automation tools on AWS. Includes a demo from Casey Lee, Chief Architect at Stelligent, on the tools used to accomplish this for 3M, including AWS Service Catalog, AWS CloudFormation, AWS CodePipeline and Cloud Custodian, an open source tool for managing AWS accounts.
DVC202 – Community Knowledge Sharing for AWS
Thanos Baskous – Chief Architect, Blackstone; Corey Quinn – Editor, Last Week in AWS; Joshua Levy – Co-founder, Open Guide to AWS; Manish Pandit – Director of Engineering, Marqeta
The Open Guide to AWS is an open source writing project which has become one of the most popular AWS resources on the web. It’s both a written resource on GitHub, with over 100 contributors, and a large Slack group. Each has become a forum for trading practical knowledge not covered in standard documentation. We talk about the Guide and how it started, share lessons on seeding initial content, the editorial process, and how to foster a healthy extended community and encourage social engagement.
DVC201 – Build AWS Skills Through Community-Led User Groups
Kittaya Kanchanobhas – Assistant to Managing Director, DailiTech; John Varghese – Cloud Steward, Intuit; Sebastian Montini – Engineering Manager, Medallia; Jason O’Conaill – International Director, AWS User Group
Did you know that there are over 300 AWS User Groups worldwide? Join this panel discussion featuring AWS community leaders from around the world, and learn the value of attending community-led AWS Meetups in your region. Community leaders share their experiences, talk through how local communities help developers solve problems and achieve their goals, and discuss the benefits of participating in peer-to-peer AWS knowledge sharing and networking activities.
The above two sessions are part of the re:Invent Developer Community Day, six community-led sessions where AWS enthusiasts share technical insights on trending topics based on first-hand experiences and knowledge shared within local AWS communities.
ENT318 – Leveraging a Cloud Policy Framework – From Zero to Well Governed
Chris Lyon – VP of Engineering, Mulesoft; Vikram Pillai – Chief Architect, Director of Engineering, CloudHealth Technologies
An open source “cloud policy framework” enables users to leverage a community that can help define and tune best practice policies, and help SaaS vendors and ISVs capture the best way to manage an application and share it with customers. (Session sponsored by CloudHealth Technologies)
Angel Pizarro – TBDM, Research and Technical Computing, AWS; Raminderpal Singh – VP, Head of Microbiome Division, Eagle Genomics; Nick James – Senior Bioinformatician, Eagle Genomics
With the increasing use of genomic sequencing for scientific discovery, the rate-limiting step for researchers is not in obtaining genetic code, but in having the capacity for storage and computing power to analyze it. Learn how Eagle Genomics built a cloud solution that uses an open-source workflow engine (eHive, Docker containers to process jobs, and a REST service to manage pipeline runs, all to help customers process genetic sequences up to 20 times faster without additional costs.
MCL303 – Deep Learning with Apache MXNet and Gluon
Sunil Mallya – Sr AI Solutions Architect, AWS; Miro Enev – Sr. Solution Architect, Deep Learning, NVIDIA
Learn to program deep learning models using Gluon, the new intuitive, dynamic programming interface available for the Apache MXNet open-source framework. We’ll also explore neural network architectures such as multi-layer perceptrons, convolutional neural networks (CNNs), and LSTMs.
Keith Steward – Specialist SA (EMR, AI), AWS; Hira Niranjan – Amazon Lex Solutions Architect
Build an intelligent agent combining AI services and other offerings from AWS, and open source technologies like Blender, Apache MXNet, and CLIPS.
SRV302 – Building CI/CD Pipelines for Serverless Applications
Chris Munns – Senior Developer Advocate – Serverless, AWS; Ben Kehoe – Cloud Robotics Research Scientist, iRobot
A method for automating the deployment of serverless applications running on AWS Lambda, including how you can model and express serverless applications using the open-source AWS Serverless Application Model (AWS SAM).
SRV424 – Massively Parallel Data Processing with PyWren and AWS Lambda
Paras Bhuva – Sr. Solutions Architect, AWS; Olivier Klein – Head of Emerging Technologies, Asia-Pacific, AWS
How to achieve fast processing speeds using an open-source project called PyWren to massively parallelize data analytics jobs across hundreds or thousands of AWS Lambda functions.
Workshops
CON213 – Hands-on Deployment of Kubernetes on AWS
Arun Gupta – Principal Open Source Technologist, AWS
Get hands-on experience using Kubernetes and Kops (Kubernetes Operations) to create, deploy, manage and scale a Kubernetes cluster on AWS. You’ll learn how to deploy your microservices-based applications and use service discovery for them.
DEV337 – Deploy a Data Lake with AWS CloudFormation
Chuck Meyer – Sr. Developer Advocate, AWS CloudFormation, AWS; Luis Colon – Senior Developer Advocate, AWS CloudFormation, AWS
Learn to build AWS CloudFormation templates using proven methods and best practices. Deploy a fully functional data lake architecture using AWS services like Amazon RDS and open source components like Apache Zeppelin.
GPSWKS301 – GPS: Comprehensive Big Data Analytics Architecture Made Easy
Luis Soto – Sr. Category GTM Lead, Amazon; Kim Schmidt – President & CEO, DataLeader.io
A modern Big Data architecture involves extending your on-premises data management to AWS, implementing a data pipeline to stream real-time data into cloud data warehouse Amazon Redshift, perform data transformation, discovery, predictive analytics through machine learning, visualize complex information, and be notified to respond to business events. This session is for APN Consulting Partners and organizations looking for ways to accelerate and modernize their Big Data projects. You will learn how to deploy and integrate AWS Services with Third-party Solutions in AWS Marketplace. Reduce your time to market by combining AWS services, open source software and ready-to-run on AWS solutions.
MCL309 – Deep Learning on a Raspberry Pi
Julien Simon – Principal Technical Evangelist, AWS
We introduce you to the open source deep learning framework Apache MXNet, and show you how to install it on a Raspberry Pi. Then, using a camera and a pre-trained object detection model, we show real-life objects to the Pi and listen to what it thinks the objects are, thanks to the text-to-speech capabilities of Amazon Polly.