re:Invent open source highlights: Week 3

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. Visit the website to learn more.

Over the past three weeks, re:Invent 2020 has featured hundreds of sessions across many different topics and tracks. In this third and final post of the series, we’ll share open source highlights from week three. If you missed the previous articles, make sure you check out the first and second week’s highlights as well. Finally, re:Invent will be back on January 11th, 2021, so in this article, we’ll also let you know more about what to expect.

Week three had no new sessions from the open source track, but there was still plenty to check out including a couple of great announcements during the keynote from Dr. Werner Vogels, Amazon VP and CTO.

Image of Werner Vogels.

Werner Vogels’s keynote

Vogels touched on several themes, but the one I want to highlight here is observability. Metrics, counters, logs, customers—all have data on how things are working. It can be difficult to get all this data onto a dashboard when massive amounts of data are involved. In observability, we look at how, without reaching into a system, we can infer its state from its outputs. The key pillars are metrics, logging, and tracing. We are beyond the ability of humans to handle observability and intuit from the mass of data what is wrong or going wrong.

With that, Vogels announced the preview of Amazon Managed Service for Grafana and Amazon Managed Service for Prometheus (both in preview), as well as the general availability of the AWS Open Distro for OpenTelemetry.

Marcia Villalba’s blog post, Announcing Amazon Managed Service for Grafana (in Preview), detailed the preview and explained how we are working in partnership with Grafana Labs on this new service. Why Grafana? Well, Grafana is one of the most popular open source technologies used to create observability dashboards for your applications. It has a pluggable data source model and support for different kinds of time series databases and cloud monitoring vendors. Read the post to find out more about how this will work and on details of how to sign up.

It was good to see Grafana Labs also write about this service launch and talk about the new partnership in their post, Our new partnership with AWS gives Grafana users more options. Matt Asay looked more closely at this collaboration in his post, How AWS and Grafana Labs are collaborating to improve Grafana for all. In the post, he shared how we are working closely with Grafana Labs, both in licensing revenue, and with code contributions to help make Grafana even better for wherever users choose to run Grafana.

On the Prometheus side, Jeff Barr wrote about launching a preview of Amazon Managed Service for Prometheus (AMP). In his post, Join the Preview – Amazon Managed Service for Prometheus (AMP), he provides details on Amazon Managed Service for Prometheus (AMP), a fully managed service that is 100 percent compatible with Prometheus. If you are new to Prometheus, it is a leading open source monitoring solution with an active developer and user community. If you are already using Promethues, it supports the same metrics, the same PromQL queries and can also make use of the 150+ Prometheus exporters. The preview has support for Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS), but it can also be used to monitor your self-managed Kubernetes clusters that are running in the cloud or on-premises.

Sticking with Prometheus, Marc Chene and Jaana Dogan’s post, Introducing Amazon Managed Service for Prometheus (AMP), showed how you can use the same open source Prometheus data model, existing instrumentation, and query language to monitor performance with improved scalability, availability, and security without having to manage the underlying infrastructure.

Alolita Sharma and Nizar Tyrewalla shared in their blog post, AWS Distro for OpenTelemetry adds Prometheus and Lambda support and other cool features, news that Open Distro for OpenTelemetry now brings support for Prometheus and AWS Lambda and adds AWS X-Ray support in Go and Python, as well as adding an OpenTelemetry Protocol (OTLP) HTTP exporter, an AWS EMF exporter, and an X-Ray exporter. Read the post for more details.

Aman Brar and Jason Liu wrote about Building a reliable metrics pipeline with the OpenTelemetry Collector for AWS Managed Service for Prometheus, where they shared their experience working with the OpenTelemetry Collector and Prometheus Remote Write Exporter. Read this post to find out more about their experiences in tackling challenges they faced and how they applied lessons learned to ensure the reliability of the AWS Distro for the OpenTelemetry Collector as the de facto agent for sending metrics to AWS Managed Service for Prometheus.

Other launches to check out

Although not part of the keynote, there were also important announcements during week three. In Announcing AWS IoT Greengrass 2.0 – With an Open Source Edge Runtime and New Developer Capabilities, Channy Yun announced the release of AWS IoT Greengrass 2.0. AWS IoT Greengrass makes it easy for device builders to build, deploy, and manage intelligent device software. One of the big updates was the fact that this tool now provides an open source edge runtime. AWS IoT Greengrass 2.0 edge runtime has been open sourced under an Apache 2.0 license and is available on GitHub. Access to the source code allows you to more easily integrate your applications, troubleshoot problems, and build more reliable and performant applications that use AWS IoT Greengrass. Make sure you check out the session from James Gosling and Konstantin Dotchkoff (below) that walks through more details.

On a related note, Channy Yun also shared news of FreeRTOS Long Term Support (LTS) in his post New – FreeRTOS Long Term Support to Provide Years of Feature Stability. FreeRTOS is an open source, real-time operating system for microcontrollers that makes small, low-power edge devices easy to program, deploy, secure, connect, and manage. Read the post for more details.

AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other devices. AWS IoT Core can support billions of devices and trillions of messages and can process and route those messages to AWS endpoints. Last week, we announced that you can now deliver those messages to Apache Kafka clusters, whether these are your own self-managed Apache Kafka clusters or Amazon Managed Streaming for Apache Kafka (Amazon MSK).

Finally, to return to observability, we announced that AWS IoT SiteWise can integrate with Grafana via a new plugin. This plugin lets you easily visualize your AWS IoT SiteWise data in your Grafana dashboards. With this plugin, you can visualize and monitor your equipment data in near-real time using the wide range of visualization options in Grafana dashboards. If you are not familiar with AWS IoT SiteWise, it is a managed service that makes it easy to collect, store, organize, and monitor data from industrial equipment at scale.

Open source-related sessions

The third week of re:Invent also featured several great AWS Amplify and GraphQL/AWS AppSync related sessions.

In the Model and access application data more efficiently with AWS Amplify session, Steve Johnson shared how developers can leverage the power of GraphQL to model app data for faster production apps, that can scale across platforms and use the offline capabilities using Amplify DataStore. In the session, Best practices to securely operate GraphQL at scale with AWS AppSync, Brice Pelle shared how AWS AppSync enables you to securely build and run GraphQL APIs at scale, covering the options available to you and how you can leverage AWS security services. Nader Dabit looked at some of the capabilities that AWS AppSync GraphQL resolvers help you with. In Unify access to siloed data with AWS AppSync GraphQL resolvers. Dabit showed how to simplify how you query your data across different sources.

If you use Elasticsearch, then there were a few sessions during week three for you to check out. In the session What’s new in Amazon Elasticsearch Service, Jon Handler did a deep dive into the service, customer use cases, best practices, and newly launched features, as well as sharing what’s next for Amazon Elasticsearch Service. In the session Security overview for Amazon Elasticsearch Service, Carl Meadows dived deep and shared the rich set of security options that allow you to control access to your cluster and data as well as how to configure it for even your most sensitive data. This session covered many topics, such as how to configure encryption, authorization using IAM and enterprise identity providers using SAML, set up of fine-grained access control for your indices, documents, and fields, and much more.

Following up her session in week one, Madelyn Olson returned with a talk, called Beyond caching: Advanced design patterns in Redis, that explored the use cases, examples, and demos of how you can take advantage of Redis to solve distributed system problems like managing state, processing events, and transforming data in real time.

If MySQL is more your thing, then check out this session from Yoav Eilat on MySQL options on AWS: Self-managed, managed, serverless. This was a great session, in which Eilat provided an overview of the MySQL and MariaDB options available on AWS. This included a deep dive on Amazon RDS, the fully managed MySQL service; Amazon Aurora, a MySQL-compatible database with up to five times the performance; and other additional innovations.

In Run big data analytics faster at lower cost with Amazon EMR, Matthew Liem explored features of Amazon EMR and how they can help you run your Apache Spark, Apache Hive, Apache Flink, and Presto workloads both faster or more cost-effectively. He also took a look into the architectures and design patterns that organizations have employed when migrating their open source analytics applications to Amazon EMR. Sticking with the same theme, Joseph Marques shared an in-depth look at various performance improvements in Amazon EMR for Apache Spark and Presto in the session Turbocharging query execution on Amazon EMR.

Next up is the session from Angus McAllister, where he introduced a new open source project, FHIR Works on AWS, that is helping accelerate the use of Fast Healthcare Interoperability Resources (FHIR) in customer solutions. FHIR is gaining popularity around the world as the standard to use for exchanging healthcare data, and it is being increasingly adopted in Europe and Australasia. In the United States, FHIR is mandated in the 21st Century Cures Act final rule. This is both a challenge and an opportunity for those who manage healthcare data in electronic health records (EHRs) and other systems. This session introduced the FHIR standard as well as FHIR Works on AWS. Check out the session, An introduction to healthcare interoperability and FHIR Works on AWS.

Finally, in What’s new with AWS IoT Greengrass, James Gosling and Konstantin Dotchkoff covered AWS IoT Greengrass 2.0. For those unfamiliar with what AWS IoT Greengrass does, it seamlessly extends AWS to edge devices so that they can act locally on the data that they generate, while still using the cloud for management, analytics, and durable storage. They covered the new features with this major release, with AWS IoT Greengrass now providing customers an open source edge runtime, a rich set of prebuilt software components, tools for local software development, and new features for managing device software on large fleets of devices.

AWS partner sessions

Week three included some excellent partner sessions as well. Below is a list of ones that we checked out.

What do 1,000 people say about open-source observability? by Jonah Kowall, CTO at Logz.io, shared the exciting tools communities are using to solve for observability needs and which best practices work when implementing and scaling open source observability.

In Confluent Cloud metamorphosis: Making Apache Kafka easier Joseph Morais from Confluent, gave an overview of how the Confluent Cloud serverless Kafka offering makes it even easier, more secure, and cost-effective to build event-driven applications on AWS. Joseph then went on to dive deep into some of the capabilities, including features such as infinite storage, self-provisioned AWS PrivateLink endpoints, bringing your own encryption keys and more.

Build cloud-ready apps faster with Red Hat OpenShift Service on AWS by Andrew Cathrow looked at the recently announced a fully managed joint service, Red Hat OpenShift Service on AWS (ROSA), which can be deployed directly from the AWS Management Console and can integrate with other AWS cloud-native services. Andrew looked at the new service, which delivers production-ready Kubernetes that many enterprises use on premises today, enhancing your ability to shift workloads to the AWS Cloud and making it easier to adopt containers and deploy applications faster.

Coming up in re:Invent

The final part of re:Invent kicks off on January 11th. Here are the sessions you can look forward to. These are already in the re:Invent catalog so you can add them to your calendar (links provided) and we look forward to more content and updates in the new year.

Jan 12th

OPN205: Next-gen networking infrastructure with Rust and Tokio
OPN401: Securing Bottlerocket updates with TUF and Rust

Jan 13th

Jan 14th

OPN308: Designing better ML systems: Learnings from Netflix
OPN402: Rust-vmm: Secure VM-based isolation made simple

AWS Open Source Blog