Apache Airflow® with Astro by Astronomer - Pay As You Go

The main purpose of Astro by Astronomer today is Airflow. We are using Airflow in order to orchestrate our data pipelines. We are orchestrating tasks in Snowflake and AWS. Today our data warehouse is Snowflake, and we transform data within Snowflake using DBT, a transformation tool. We use Astronomer Cosmo, which allows us to generate dependencies between DBT jobs automatically. This enables analytics engineers to focus on SQL optimization while the packages construct the DAGs. Airflow is a game changer for Accor as it centralizes our pipeline, allowing us to monitor, rerun jobs, and see logs. We also use a BI tool such as Tableau or Power BI, all orchestrated with Airflow. This year, we have discovered the observe option for retrieving more detailed logs.

Airflow's features such as log monitoring and the capacity to rerun jobs are crucial. It significantly streamlines our processes with the capability to create the entire infrastructure around Airflow swiftly, without needing individual installations of components. It includes the necessary components such as the scheduler and database, all managed by Astronomer. The user interface is user-friendly and informative, with features for managing connections, variables, and aggregating logs under Observe. Deployments handled by Astronomer automate what was once a manual task, saving us time and costs. We have been able to unify our orchestrators, enhancing efficiency and reducing complexity. Through Terraform, we can create environments swiftly, decreasing setup time from weeks to minutes. Furthermore, Astro by Astronomer helps in setting up data quality elements between DAGs.

If we can improve something, it would be to have hibernation for the production deployment. Currently, setting up hibernation needs to be done more like a dev environment rather than a production one.

I started working with Astro by Astronomer three years ago. I discovered this tool through LinkedIn. I am very happy to use it today.

Astro by Astronomer works very well. We did not have any production issues or user interface unavailability.

We have defined scalability for the workers on our deployments and it works very well as expected in the documentation.

We had the opportunity to exchange with customer support, especially in France, and it works very well. All the people are there to answer our needs and questions. It was a pleasure to talk with them and exchange ideas because they had solutions for all of our queries.

Previously in a different company, I used Airflow deployed by Cloud Composer in GCP and some homemade tools in AWS using CloudWatch events and Control M. We needed a modern orchestrator, so we tested several and found that Airflow using Astro by Astronomer fit the most.

Before using Astro by Astronomer, we deployed components by hand, and it was very difficult to deploy and maintain. With Astro by Astronomer today, we do not need to deploy components by hand, but rather it is automatic using Terraform. This results in cost savings and time efficiency.

Previously, to deploy the environment and set up everything, we depended on another team, a DevOps team. We would wait two or three weeks before getting what we wanted. But today, we have the possibility to create environments through Terraform, and we can create environments in less than five minutes.

In one year, we save a lot of time using Astro by Astronomer because we do not need to deploy different components of Airflow one by one. Astro by Astronomer does it for us. It is also the best way to set up data quality elements between the different DAGs to see the quality of a project, regardless of the deployment or the environment we are working on.

It was as we expected in terms of cost because previously we had three different orchestrators, and now we have only one, which is less expensive than before.

We evaluated MWAA, the solution proposed by Amazon to deploy Airflow. We also tested Dagster, Prefect, and other solutions. Astro by Astronomer fit all of the requirements.

The tool allows us to deploy easily, to monitor easily, and to spread practices easily. Everything is very good on the governance and security side of Astro by Astronomer. We had the opportunity to conduct penetration tests and everything was good. Concerning the AI capabilities, today we have the agent in Astro by Astronomer's user interface, and this agent helps us with migration and developing some use cases. The agent met our expectations because we needed an agent for some teams to answer questions about Airflow and how to set up, for example, some monitors. This agent has been very helpful. I give this product a rating of ten out of ten.

My main use case for Astro by Astronomer is to manage our data pipelines, to run DBT processes, and to run all data from the company.

I use Astro by Astronomer in my day-to-day work with schedules from different pipelines, each from different areas of the company. They process data from the sources with the DBT components, and we transform the data inside Snowflake.

Regarding my main use case with Astro by Astronomer, we changed it; first, we had an installation of Astro by Astronomer that was running on our AWS account, but we moved it to Astro Cloud, which is managed by Astronomer because it is easier to manage and we do not have the toil of managing the Kubernetes clusters and the problem of having to upgrade those clusters and the application. The only consideration is that each cluster on Astro Cloud has a fee from Astronomer.

The best features Astro by Astronomer offers are its ease of use and integration with our pipeline, including integration with DBT, Snowflake, running with GitHub Actions, and we managed to integrate with our SSO and with SailPoint as well for managing user access.

Out of all those features, I find myself relying the most on the integration with DBT because I think it is very straightforward; everything that the tool, the CLI that you use to run, to deploy a new container to the pipeline is easy to handle. Also, if you want to upgrade a new DAG or update a new DAG, using the CLI is very easy to use, and adding it to the GitHub pipeline was a very easy thing to do.

Astro by Astronomer has positively impacted our organization by reducing our toil and reducing the time by around twenty-five percent of what we had; before moving to Astro by Astronomer, we had to manage Airflow and manage the Kubernetes cluster as well when we were running the Kubernetes cluster locally, which saved almost one working person per month in total.

When I say we saved almost one working person per month, it is mostly by reducing manual work because we did not have to stop to upgrade the versions of the application and the Kubernetes cluster every three or six months. This process took a lot of time because we had to do it for all three environments that we manage; divided this for the whole year, we can assume this. Also, the time to do investigations on the system was reduced because we had fewer things to deal with since we did not have to manage the cluster on our side.

Astro by Astronomer can be improved; they are doing a lot of progress on observability. How to see how all the pipelines and all the steps of the DAGs connect to each other is something that they improved recently, so we can see this, but as we have another process running outside of Astro by Astronomer, we could not use this for everything. Also, if we wanted to import data from outside, the price that they shared with us was higher than the actual tool that we are using right now.

On a scale of one to ten, I would rate Astro by Astronomer an eight; there are some things that could be improved, perhaps in performance, perhaps in some areas of the web interface that could be more straightforward, but I think it is a solid eight.

I have been using Astro by Astronomer for about two years.

There are no other improvements that I can think of regarding Astro by Astronomer.

The advice I would give to others looking into using Astro by Astronomer is that they share good documentation, and on LinkedIn, they have a learning path that you can use; if you are a premium member of LinkedIn, you can complete this learning path.

I would rate Astro by Astronomer an eight out of ten.

A specific example of how I've used Astro by Astronomer in one of my recent projects is in the flagship project that we have, a framework that is in charge of orchestrating all the different ingestions that are done for the lakes the company has. I orchestrate the ingestion from different sources and then orchestrate the processing to take it to the different layers: bronze, silver, and gold. I would say it is the core of orchestration between layers within the Lakehouse.

It is the core of my day-to-day work. We have more than 50 pipelines running for different lakes across different sources in different lakes, according to domains within the company. Using Airflow is the core to be able to perform all the batch processes and migrations from sources until reaching tables or the semantic layer. It not only orchestrates ingestion; it also orchestrates processing with DBT, modeling with DBT, and processing with EMR and Glue Job.

Besides the ease of use and quick access from the cloud, I find the CLI especially valuable in Astro by Astronomer. The CLI allows me to do everything in a very automated way. Also being able to deploy in a much faster way and having auditing over the pipelines is valuable. I understand that Astro by Astronomer also has an Astro IDE to be able to build pipelines from the browser with a prompt. That is a tool that I think is quite interesting, though I have not tried it yet.

Astro by Astronomer has had a positive impact on my organization mainly in terms of time savings and error reduction, since I seek to standardize all the processes within my framework. Based on how I handle ingestion, the general DAG that I have is standard for all the lakes that exist in the company.

I can go deeper into how I have measured that time saving and error reduction. Before, each lake worked with the DAGs independently. That implied rework in each of the lakes. Now I have a single framework that is replicated in all the DAGs, in all the lakes. This translates into simply replicating and not having each one be independent. There is substantial time saving there. Also at the error level, when some type of error occurs, due to the experience I have been having within the framework, it is known that it can be reflected and automated for the rest of the lakes or remediated for the rest of the lakes.

I think there is an aspect of the tool that has caused me some difficulty, which is the scheduler, and I think it could be more intuitive or efficient. The scheduler's latency, which I know is not sub-second and is not suitable for sub-second latency, is more for batch and minutes is recommended; in seconds it can become intensive. In the future, having something a bit more in seconds would be interesting. I also think there is an issue that it tends to be a bit slow; sometimes the tool does not reflect changes quickly, especially when I want to see something refreshed automatically.

I decided to switch from native Airflow to Astro by Astronomer because of the official support.

My main use case for Astro by Astronomer depends on the project use cases. For something like big data projects on ETL pipeline and ELT pipelines, it truly depends on projects to projects. I would say first you can go with something modern data warehouse and lake house orchestrations. Why do we choose Astro by Astronomer? Because it has built-in data lineage which tracks the data flow down to the table and column level, making it very easy for a data owner to know how the data is flowing. Secondly, it provides multi-tenant and multi-cloud pipeline integrations, allowing integration with any cloud vendors whether it is AWS, Azure, or GCP. Third, I would say that it has wonderful production machine learning and AI pipelines, where MLOps teams can use Astro by Astronomer to orchestrate end-to-end machine learning workflows with distributed frameworks such as Ray, Spark, or SageMaker to evaluate metrics. It creates complex DAGs for you. Lastly, on CI/CD driven analytics engineering, engineering teams moving towards data operations use Astro by Astronomer to automate testing and deployments in my current organization. Developers spin up the local Airflow environment using the Astro CLI, write some test code, and push it to GitHub or GitLab which automatically deploys to production via CI/CD. Astro by Astronomer, a commercialized version of Airflow, works on local dev and production, making it easy for people to test in the dev environment.

A quick specific example of one of these use cases is that in our recent team, we run Snowflake, Databricks, and BigQuery as well as DBT. Astro by Astronomer serves as the central orchestration pipeline or orchestration plane, triggering the DBT transformations and syncing the ingest parts, such as from Kafka or Fivetran, while running quality checks on the data and notifying downstream tools, such as BI tools including Looker or Tableau. We chose Astro by Astronomer for its built-in lineage, which helps track the data flow from the table and column level. This is a use case of Astro by Astronomer that we use in multiple teams within Zalando, where we rely on different cloud software or platform as a service, with Astro by Astronomer forming the central orchestration layer.

Regarding my main use cases, a small example of our e-commerce daily revenue and customer reporting occurs at 2:00 AM. The daily team pulls raw data, aggregates financial metrics, computes churn risk, and refreshes executive Looker dashboards in BI dashboards. The workflow begins when S3 ingestions come in, which go to Snowflake. When the data arrives in Snowflake, we have triggers written on Astro by Astronomer. From there, the DBT pipeline takes over, writing it into staging for clean and deduplicated data, with quality tests validating success or failure, including any malformed data issues. This way, we utilize Astro by Astronomer precisely, rather than writing messy bash operators to execute DBT runs as a single opaque block. Astro by Astronomer, developed by Astronomer Cosmos, alleviates this by allowing us to manage individual DBT models and test them in discrete observable Airflow tasks. It makes life easier by adhering to Astro CLI standards, which are module-based and consistent. It greatly aids us with granular model observability that is not found in a single DBT test and offers zero downtime secret and environment management, ensuring the environment remains consistently up. Moreover, it provides frictionless local development by allowing data engineers to run Astro dev start, spinning up a Docker context that mimics the production Airflow one-to-one, allowing for testing DBT models against a local development schema, pushing branches into GitHub, and letting CI/CD automatically deploy updated DAGs to Astro by Astronomer cloud.

Astro by Astronomer offers multiple features that depend on projects. In our case, I would say the best features include developer tooling and automation. Astro by Astronomer provides the Astro CLI, creating local parity where developers can work locally in a Docker container mimicking a production environment, which is fantastic. Additionally, Astronomer Cosmos is an open-source framework natively integrated into Astro by Astronomer that automatically converts DBT projects into fully observable native Airflow DAGs, allowing every DBT model to be tested and become an individual node in the Airflow UI without needing manual wrappers. This feature saves a lot of development time.

Furthermore, it has Auto AI, which assists with DAGs, troubleshooting task failures, and generating automated migrations across different Airflow version upgrades while enabling integration with various AI and MCP servers. It also provides no-coder blueprint authoring, functioning as a drag-and-drop workflow. On observability and data lineage, it presents an excellent pictorial landscape of data flows and lineage management. Cost optimization is also vital, with dynamic auto-scaling based on workload without needing to worry about scaling nodes under the hood. It offers zero downtime, and it includes a fantastic Terraform provider from a DevOps perspective.

The feature that has made the biggest difference for our team among all these offerings depends on the project at hand. I would say it is the cloud-native hosting that treats Airflow as a virtual machine. When using open-source Airflow or basic cloud providers such as AWS-managed Airflow MWAA or GCP Cloud Composer, the vendor simply spins up EC2 or Kubernetes instances and installs Airflow on top, leaving the rest to you. You still have to debug slow DAG parses, manage complex CI/CD containers, and build custom lineage connections. Astro by Astronomer operates at the pipeline and code level, which is vital. This allows Astro by Astronomer, designed by Astronomer, to streamline how data engineers write, test, and observe data workflows, enhancing scaling, cost efficiency, and maintaining zero downtime with automated upgrades.

Astro by Astronomer has positively impacted my organization by significantly improving developer productivity, reducing downtimes, and alleviating overhead associated with upgrades. The acceleration in developer velocity and time to market is evident, as our cost optimization efforts have led to a nearly 45% reduction in cloud spending, translating to a high return on investment.

Astro by Astronomer is doing great, and while improvement opportunities exist, particularly around the cost model, which can occasionally feel steep, I generally find it quite favorable. For smaller teams, the cost can be quite high; self-hosted Airflow or AWS MWAA options are cheaper. For larger companies, it seems acceptable. Aside from that, I do not have any other concerns, but perhaps adding advanced governance such as streamlining RBAC directly into the UI could be beneficial, which may not currently be available.

I have been using Astro by Astronomer for approximately five years.

Astro by Astronomer is definitely stable, which is one of the key reasons we chose it.

The scalability of Astro by Astronomer is very good.

Customer support for Astro by Astronomer is very good and top-notch.

I definitely see a return on investment, as the time to market has notably decreased. Fewer employees needed is not a point for discussion since we are already a lean team; the current number of employees is optimal for our work with Astro by Astronomer. When I mention a 45% reduction in cloud spend, it was measured against a previous solution we used, which was managed Airflow web services. Now, with Astro by Astronomer, our costs are significantly lower. However, it is not a definitive metric, as various factors come into play. It is essential to note that not the entire team has fully adopted Astro by Astronomer yet, which is vital for context.

My experience with pricing, setup cost, and licensing has been very good.

The accuracy of Astro by Astronomer's AI capabilities is very good, and its reliability is also very good. The advice I would give to others looking into using Astro by Astronomer is that it is a good product with great support and excellent AI integrations, making it suitable for industry scale. I would rate this product as an eight out of ten.