AWS Open Source Blog

Category: Technical How-to

solution architecture described in blog post

Amazon MWAA with AWS CodeArtifact for Python dependencies

This post was written by Dzenan Softic and Sam Dengler. Many organizations rely on Apache Airflow, an open source project, to orchestrate their data pipelines. In 2020, Amazon Web Services (AWS) released Amazon Managed Workflows for Apache Airflow (Amazon MWAA), which lets engineers focus on business solutions rather than on running and maintaining infrastructure for […]

Read More

Auto-instrumenting a Python application with an AWS Distro for OpenTelemetry Lambda layer

Customers want better insight into understanding the behavior of their systems, but not all customers can afford to make significant code changes in their existing pipelines to add more observability. In this walkthrough, we explain how to get telemetry data from AWS Lambda Python functions, without having to change a line of code. Find the […]

Read More

Scaling Cortex with parallel compaction

In this post, Albert Choi, an intern on the Amazon Managed Service for Prometheus team, shares his experience of designing and implementing parallel compactors inside of the Cortex open source project. The addition to the compactors enables Cortex to handle large volumes of active metrics per tenant. This blog post details the work done as […]

Read More
Tierney – stock.adobe.com

Performing canary deployments and metrics-driven rollback with Amazon Managed Service for Prometheus and Flagger

This post was written by Kevin Bell and Stefan Prodan. Canary deployments are a popular tool to reduce risk when deploying software, by exposing a new version to a small subset of traffic before rolling it out more broadly. Creating the machinery to do this kind of controlled rollout, and monitoring for possible problems and […]

Read More

Delta Sharing on AWS

This post was written by Frank Munz, Staff Developer Advocate at Databricks. An introduction to Delta Sharing During the past decade, much thought went into system and application architectures using domain-driven design and microservices, but we are still on the verge of building distributed data meshes. Such data meshes are based on two fundamental principles: […]

Read More

Implementing a hub and spoke dashboard for multi-account data science projects

Modern data science environments often involve many independent projects, each spanning multiple accounts. In order to maintain a global overview of the activities within the projects, a mechanism to collect data from the different accounts into a central one is crucial. In this post, we show how to leverage existing services—Amazon DynamoDB, AWS Lambda, Amazon […]

Read More

Deploying and scaling Apache Solr on Kubernetes

Apache Solr is an open source enterprise search platform built on Apache Lucene. Solr has been powering large-scale web and enterprise applications across industries such as retail, financial services, healthcare, and more. Its features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, and rich document handling. Apache Solr’s distributed deployment comes with […]

Read More
vladwel – stock.adobe.com

Enhancing Spinnaker deployment for dynamic AWS account registration

This post was written by Manabu McCloskey, Gaurav Dhamija, Nima Kaviani, Siddhi Shah, Kevin Kidd, Brandon Leach, and Shrirang Moghe. Multi-account Amazon Web Services (AWS) environments are a recommended best practice through which AWS customers can have clear separation of concerns across teams and applications where rapid innovation, flexible security controls, and varied adoption of […]

Read More

Setting up Amazon Managed Grafana cross-account data source using customer managed IAM roles

Amazon Managed Grafana is a fully managed and secure data visualization service for open source Grafana that enables customers to instantly query, correlate, and visualize operational metrics, logs, and traces for their applications from multiple data sources. Amazon Managed Grafana integrates with multiple Amazon Web Services (AWS) security services, and supports AWS Single Sign-On (AWS […]

Read More