Apache Airflow® with Astro by Astronomer - Annual Plans

A specific example of how I've used Astro by Astronomer in one of my recent projects is in the flagship project that we have, a framework that is in charge of orchestrating all the different ingestions that are done for the lakes the company has. I orchestrate the ingestion from different sources and then orchestrate the processing to take it to the different layers: bronze, silver, and gold. I would say it is the core of orchestration between layers within the Lakehouse.

It is the core of my day-to-day work. We have more than 50 pipelines running for different lakes across different sources in different lakes, according to domains within the company. Using Airflow is the core to be able to perform all the batch processes and migrations from sources until reaching tables or the semantic layer. It not only orchestrates ingestion; it also orchestrates processing with DBT, modeling with DBT, and processing with EMR and Glue Job.

Besides the ease of use and quick access from the cloud, I find the CLI especially valuable in Astro by Astronomer. The CLI allows me to do everything in a very automated way. Also being able to deploy in a much faster way and having auditing over the pipelines is valuable. I understand that Astro by Astronomer also has an Astro IDE to be able to build pipelines from the browser with a prompt. That is a tool that I think is quite interesting, though I have not tried it yet.

Astro by Astronomer has had a positive impact on my organization mainly in terms of time savings and error reduction, since I seek to standardize all the processes within my framework. Based on how I handle ingestion, the general DAG that I have is standard for all the lakes that exist in the company.

I can go deeper into how I have measured that time saving and error reduction. Before, each lake worked with the DAGs independently. That implied rework in each of the lakes. Now I have a single framework that is replicated in all the DAGs, in all the lakes. This translates into simply replicating and not having each one be independent. There is substantial time saving there. Also at the error level, when some type of error occurs, due to the experience I have been having within the framework, it is known that it can be reflected and automated for the rest of the lakes or remediated for the rest of the lakes.

I think there is an aspect of the tool that has caused me some difficulty, which is the scheduler, and I think it could be more intuitive or efficient. The scheduler's latency, which I know is not sub-second and is not suitable for sub-second latency, is more for batch and minutes is recommended; in seconds it can become intensive. In the future, having something a bit more in seconds would be interesting. I also think there is an issue that it tends to be a bit slow; sometimes the tool does not reflect changes quickly, especially when I want to see something refreshed automatically.

I decided to switch from native Airflow to Astro by Astronomer because of the official support.

My main use case for Astro by Astronomer depends on the project use cases. For something like big data projects on ETL pipeline and ELT pipelines, it truly depends on projects to projects. I would say first you can go with something modern data warehouse and lake house orchestrations. Why do we choose Astro by Astronomer? Because it has built-in data lineage which tracks the data flow down to the table and column level, making it very easy for a data owner to know how the data is flowing. Secondly, it provides multi-tenant and multi-cloud pipeline integrations, allowing integration with any cloud vendors whether it is AWS, Azure, or GCP. Third, I would say that it has wonderful production machine learning and AI pipelines, where MLOps teams can use Astro by Astronomer to orchestrate end-to-end machine learning workflows with distributed frameworks such as Ray, Spark, or SageMaker to evaluate metrics. It creates complex DAGs for you. Lastly, on CI/CD driven analytics engineering, engineering teams moving towards data operations use Astro by Astronomer to automate testing and deployments in my current organization. Developers spin up the local Airflow environment using the Astro CLI, write some test code, and push it to GitHub or GitLab which automatically deploys to production via CI/CD. Astro by Astronomer, a commercialized version of Airflow, works on local dev and production, making it easy for people to test in the dev environment.

A quick specific example of one of these use cases is that in our recent team, we run Snowflake, Databricks, and BigQuery as well as DBT. Astro by Astronomer serves as the central orchestration pipeline or orchestration plane, triggering the DBT transformations and syncing the ingest parts, such as from Kafka or Fivetran, while running quality checks on the data and notifying downstream tools, such as BI tools including Looker or Tableau. We chose Astro by Astronomer for its built-in lineage, which helps track the data flow from the table and column level. This is a use case of Astro by Astronomer that we use in multiple teams within Zalando, where we rely on different cloud software or platform as a service, with Astro by Astronomer forming the central orchestration layer.

Regarding my main use cases, a small example of our e-commerce daily revenue and customer reporting occurs at 2:00 AM. The daily team pulls raw data, aggregates financial metrics, computes churn risk, and refreshes executive Looker dashboards in BI dashboards. The workflow begins when S3 ingestions come in, which go to Snowflake. When the data arrives in Snowflake, we have triggers written on Astro by Astronomer. From there, the DBT pipeline takes over, writing it into staging for clean and deduplicated data, with quality tests validating success or failure, including any malformed data issues. This way, we utilize Astro by Astronomer precisely, rather than writing messy bash operators to execute DBT runs as a single opaque block. Astro by Astronomer, developed by Astronomer Cosmos, alleviates this by allowing us to manage individual DBT models and test them in discrete observable Airflow tasks. It makes life easier by adhering to Astro CLI standards, which are module-based and consistent. It greatly aids us with granular model observability that is not found in a single DBT test and offers zero downtime secret and environment management, ensuring the environment remains consistently up. Moreover, it provides frictionless local development by allowing data engineers to run Astro dev start, spinning up a Docker context that mimics the production Airflow one-to-one, allowing for testing DBT models against a local development schema, pushing branches into GitHub, and letting CI/CD automatically deploy updated DAGs to Astro by Astronomer cloud.

Astro by Astronomer offers multiple features that depend on projects. In our case, I would say the best features include developer tooling and automation. Astro by Astronomer provides the Astro CLI, creating local parity where developers can work locally in a Docker container mimicking a production environment, which is fantastic. Additionally, Astronomer Cosmos is an open-source framework natively integrated into Astro by Astronomer that automatically converts DBT projects into fully observable native Airflow DAGs, allowing every DBT model to be tested and become an individual node in the Airflow UI without needing manual wrappers. This feature saves a lot of development time.

Furthermore, it has Auto AI, which assists with DAGs, troubleshooting task failures, and generating automated migrations across different Airflow version upgrades while enabling integration with various AI and MCP servers. It also provides no-coder blueprint authoring, functioning as a drag-and-drop workflow. On observability and data lineage, it presents an excellent pictorial landscape of data flows and lineage management. Cost optimization is also vital, with dynamic auto-scaling based on workload without needing to worry about scaling nodes under the hood. It offers zero downtime, and it includes a fantastic Terraform provider from a DevOps perspective.

The feature that has made the biggest difference for our team among all these offerings depends on the project at hand. I would say it is the cloud-native hosting that treats Airflow as a virtual machine. When using open-source Airflow or basic cloud providers such as AWS-managed Airflow MWAA or GCP Cloud Composer, the vendor simply spins up EC2 or Kubernetes instances and installs Airflow on top, leaving the rest to you. You still have to debug slow DAG parses, manage complex CI/CD containers, and build custom lineage connections. Astro by Astronomer operates at the pipeline and code level, which is vital. This allows Astro by Astronomer, designed by Astronomer, to streamline how data engineers write, test, and observe data workflows, enhancing scaling, cost efficiency, and maintaining zero downtime with automated upgrades.

Astro by Astronomer has positively impacted my organization by significantly improving developer productivity, reducing downtimes, and alleviating overhead associated with upgrades. The acceleration in developer velocity and time to market is evident, as our cost optimization efforts have led to a nearly 45% reduction in cloud spending, translating to a high return on investment.

Astro by Astronomer is doing great, and while improvement opportunities exist, particularly around the cost model, which can occasionally feel steep, I generally find it quite favorable. For smaller teams, the cost can be quite high; self-hosted Airflow or AWS MWAA options are cheaper. For larger companies, it seems acceptable. Aside from that, I do not have any other concerns, but perhaps adding advanced governance such as streamlining RBAC directly into the UI could be beneficial, which may not currently be available.

I have been using Astro by Astronomer for approximately five years.

Astro by Astronomer is definitely stable, which is one of the key reasons we chose it.

The scalability of Astro by Astronomer is very good.

Customer support for Astro by Astronomer is very good and top-notch.

I definitely see a return on investment, as the time to market has notably decreased. Fewer employees needed is not a point for discussion since we are already a lean team; the current number of employees is optimal for our work with Astro by Astronomer. When I mention a 45% reduction in cloud spend, it was measured against a previous solution we used, which was managed Airflow web services. Now, with Astro by Astronomer, our costs are significantly lower. However, it is not a definitive metric, as various factors come into play. It is essential to note that not the entire team has fully adopted Astro by Astronomer yet, which is vital for context.

My experience with pricing, setup cost, and licensing has been very good.

The accuracy of Astro by Astronomer's AI capabilities is very good, and its reliability is also very good. The advice I would give to others looking into using Astro by Astronomer is that it is a good product with great support and excellent AI integrations, making it suitable for industry scale. I would rate this product as an eight out of ten.

I have been using Astro by Astronomer CLI as a local development environment for about a year. During this period, it has been the main tool I use to develop, test, and validate my Apache Airflow projects before deploying them to production. In production, I use Apache Airflow together with Astronomer Cosmos to orchestrate dbt pipelines.

My main use case for Astro by Astronomer is using the CLI as a local development environment for Apache Airflow projects. I use it to develop, test, and validate DAGs before deploying them to production, ensuring everything works correctly. In addition, in my day-to-day work, I use Astronomer Cosmos to integrate and orchestrate dbt pipelines with Apache Airflow, running data transformations on Amazon Athena with Apache Iceberg tables.

In my day-to-day work, I use Astro by Astronomer CLI to develop and test new Apache Airflow DAGs locally before publishing them to the production environment. For example, when I need to create a new data pipeline with dbt using Astronomer Cosmos, I first validate all the orchestration locally with Astro by Astronomer CLI, check that the dependencies, tasks, and integrations are working correctly, and only then do I deploy it to the production environment. This reduces errors and makes the development process much faster and more reliable.

In addition to using Astro by Astronomer, I also use Astronomer Cosmos to integrate dbt with Apache Airflow. This makes creating and maintaining DAGs much easier because Cosmos automatically generates the dbt tasks, respecting the dependencies between models, making orchestration simpler and more organized. In our environment, dbt runs happen in containers on Amazon ECS, using AWS Fargate. This model allows us to start resources only during pipeline execution and shut them down afterward, avoiding having dedicated servers running all the time. As a result, we were able to reduce infrastructure costs, maintain a scalable environment, and execute transformations efficiently, especially for workloads that do not need to be active continuously.

The main value of Astronomer Cosmos is simplifying the development and operation of pipelines with Apache Airflow. Astro by Astronomer CLI offers a very consistent local development experience, while Astronomer Cosmos makes it easier to integrate dbt with Airflow by automatically generating the DAGs and respecting the dependencies between models. In addition, this approach integrates very well with Amazon ECS and AWS Fargate to run the dbt jobs. This allows us to scale on demand and pay only for the resources used during pipeline execution, reducing infrastructure costs without sacrificing reliability and ease of maintenance.

A differentiator I consider very important is that Astronomer Cosmos is very flexible and integrates easily with other modern data engineering technologies. In my case, the combination of Astro by Astronomer CLI for local development, Astronomer Cosmos to orchestrate dbt projects, and Amazon ECS with AWS Fargate to run the jobs has brought a very efficient workflow. Besides facilitating the development and maintenance of pipelines, this architecture allowed us to reduce infrastructure costs, because the dbt containers are started only when needed and shut down at the end of execution. This offers a good combination of productivity, scalability, and operational efficiency, especially in environments that run on-demand workloads.

The positive impact of Astro by Astronomer has mainly been on team productivity and the standardization of development. Astro by Astronomer CLI made it much simpler to create and validate Apache Airflow pipelines in a consistent local environment, reducing configuration issues and speeding up testing before deployment. In addition, with Astronomer Cosmos, we were able to integrate dbt with Airflow in a much more organized way, automating the creation of DAGs.

A practical example is that we were able to significantly reduce the time needed to develop and validate new pipelines because all developers work in the same local environment using Astro by Astronomer CLI. This reduced configuration issues and decreased rework during deployments. Another important result was the reduction in costs for running dbt. Instead of keeping dedicated infrastructure running all the time, we started running the jobs in containers on Amazon ECS using AWS Fargate, which are started only when needed and shut down at the end of execution. Although I cannot share exact numbers for confidentiality reasons, we observed a relevant infrastructure cost saving, as well as a more scalable and simpler environment to operate. We also noticed a reduction in failures related to the integration between Airflow and dbt, thanks to the use of Astronomer Cosmos, which automates the creation of DAGs and ensures that dependencies between models are respected.

I believe one area for improvement for Astro by Astronomer would be to further expand the documentation and examples for more advanced scenarios, especially involving integrations with AWS, Amazon ECS, AWS Fargate, and Astronomer Cosmos. Although the documentation is good, some more complex use cases require additional research or testing until you find the best approach. It would also be interesting to offer more templates and ready-made best practices for modern architectures with dbt, Airflow, and Kubernetes, making it easier for teams that are just getting started. This would reduce the learning curve and further speed up the implementation of production environments.

The main point would be to provide more content and reference architectures for large-scale corporate environments, especially involving Airflow, dbt, Kubernetes, and AWS. This would help teams adopt best practices more quickly and reduce the time spent on architecture decisions. Otherwise, I consider the experience very positive, and Astro by Astronomer platform meets the needs of development and orchestration of data pipelines very well.

I have been working in technology for about twelve years, and specifically as a data engineer for approximately five years. During this time, I have worked at different companies and on different projects, always focused on data platforms, architecture, data processing, and cloud solutions. I currently work on the evolution and support of a data platform using technologies like Apache Airflow, dbt, AWS, and Astronomer Cosmos.

The interview was good and well structured. The questions covered the main aspects of the tool, such as user experience, benefits, improvement points, and business impact. If I could suggest some improvements, I would avoid very similar questions. At times, there was repetition, such as asking if I wanted to add something right after practically every answer. I would give more room for technical examples. Since the audience is in technology, questions about architecture, integrations, implementation challenges, and best practices would generate richer reviews. I would try to reduce administrative questions at the end—name, company, reference, contact—putting them in a form instead of asking all of them by voice. I would allow slightly more natural answers, without interrupting the interviewee between one question and the next. Overall, I found the experience positive, objective, and easy to follow.

In the flow, Cosmos unites data and ideas, and simplicity grows. I would rate this experience a nine out of ten.