Apache Airflow® with Astro by Astronomer - Pay As You Go

My main use case for Astro by Astronomer is to orchestrate data pipelines in the cloud and orchestrate an end-to-end data flow for a Lakehouse or Data Lake for batch processes.

A specific example of how I've used Astro by Astronomer in one of my recent projects is in the flagship project that we have, a framework that is in charge of orchestrating all the different ingestions that are done for the lakes the company has. I orchestrate the ingestion from different sources and then orchestrate the processing to take it to the different layers: bronze, silver, and gold. I would say it is the core of orchestration between layers within the Lakehouse.

It is the core of my day-to-day work. We have more than 50 pipelines running for different lakes across different sources in different lakes, according to domains within the company. Using Airflow is the core to be able to perform all the batch processes and migrations from sources until reaching tables or the semantic layer. It not only orchestrates ingestion; it also orchestrates processing with DBT, modeling with DBT, and processing with EMR and Glue Job.

Among the best features offered by Astro by Astronomer, I would say first that it offers a CLI with a service also in the cloud, this CLI to be able to do specific developments.

Besides the ease of use and quick access from the cloud, I find the CLI especially valuable in Astro by Astronomer. The CLI allows me to do everything in a very automated way. Also being able to deploy in a much faster way and having auditing over the pipelines is valuable. I understand that Astro by Astronomer also has an Astro IDE to be able to build pipelines from the browser with a prompt. That is a tool that I think is quite interesting, though I have not tried it yet.

Astro by Astronomer has had a positive impact on my organization mainly in terms of time savings and error reduction, since I seek to standardize all the processes within my framework. Based on how I handle ingestion, the general DAG that I have is standard for all the lakes that exist in the company.

I can go deeper into how I have measured that time saving and error reduction. Before, each lake worked with the DAGs independently. That implied rework in each of the lakes. Now I have a single framework that is replicated in all the DAGs, in all the lakes. This translates into simply replicating and not having each one be independent. There is substantial time saving there. Also at the error level, when some type of error occurs, due to the experience I have been having within the framework, it is known that it can be reflected and automated for the rest of the lakes or remediated for the rest of the lakes.

I think Astro by Astronomer could be improved by having more clarity in cost topics and how the cost associated with the tool works.

I think there is an aspect of the tool that has caused me some difficulty, which is the scheduler, and I think it could be more intuitive or efficient. The scheduler's latency, which I know is not sub-second and is not suitable for sub-second latency, is more for batch and minutes is recommended; in seconds it can become intensive. In the future, having something a bit more in seconds would be interesting. I also think there is an issue that it tends to be a bit slow; sometimes the tool does not reflect changes quickly, especially when I want to see something refreshed automatically.

I have used Astro by Astronomer for a year, but I have used the entire Airflow suite for more than five years.

I consider Astro by Astronomer to be stable.

I consider the scalability of Astro by Astronomer to be good. It adapts well to my organization's needs when data volumes or users increase.

I would rate the technical support of Astro by Astronomer as unknown since I have not had any relevant experience with the support team. I cannot rate the technical support of Astro by Astronomer because I have not had experience.

I did use another solution before Astro by Astronomer, which was native Airflow.

I decided to switch from native Airflow to Astro by Astronomer because of the official support.

I have not seen a return on investment with Astro by Astronomer. I cannot share any relevant data, such as staff reduction, money savings, or time savings.

Before choosing Astro by Astronomer, I evaluated other options. I evaluated MWAA, AWS's own service.

My advice to other professionals who are considering implementing Astro by Astronomer is to first evaluate the tool, check all the documentation, the CLI, and the courses they have available and then test from there if it is the tool they are looking for. I would rate this product a 7 out of 10.

Public Cloud

Amazon Web Services (AWS)

My main use case for Astro by Astronomer depends on the project use cases. For something like big data projects on ETL pipeline and ELT pipelines, it truly depends on projects to projects. I would say first you can go with something modern data warehouse and lake house orchestrations. Why do we choose Astro by Astronomer? Because it has built-in data lineage which tracks the data flow down to the table and column level, making it very easy for a data owner to know how the data is flowing. Secondly, it provides multi-tenant and multi-cloud pipeline integrations, allowing integration with any cloud vendors whether it is AWS, Azure, or GCP. Third, I would say that it has wonderful production machine learning and AI pipelines, where MLOps teams can use Astro by Astronomer to orchestrate end-to-end machine learning workflows with distributed frameworks such as Ray, Spark, or SageMaker to evaluate metrics. It creates complex DAGs for you. Lastly, on CI/CD driven analytics engineering, engineering teams moving towards data operations use Astro by Astronomer to automate testing and deployments in my current organization. Developers spin up the local Airflow environment using the Astro CLI, write some test code, and push it to GitHub or GitLab which automatically deploys to production via CI/CD. Astro by Astronomer, a commercialized version of Airflow, works on local dev and production, making it easy for people to test in the dev environment.

A quick specific example of one of these use cases is that in our recent team, we run Snowflake, Databricks, and BigQuery as well as DBT. Astro by Astronomer serves as the central orchestration pipeline or orchestration plane, triggering the DBT transformations and syncing the ingest parts, such as from Kafka or Fivetran, while running quality checks on the data and notifying downstream tools, such as BI tools including Looker or Tableau. We chose Astro by Astronomer for its built-in lineage, which helps track the data flow from the table and column level. This is a use case of Astro by Astronomer that we use in multiple teams within Zalando, where we rely on different cloud software or platform as a service, with Astro by Astronomer forming the central orchestration layer.

Regarding my main use cases, a small example of our e-commerce daily revenue and customer reporting occurs at 2:00 AM. The daily team pulls raw data, aggregates financial metrics, computes churn risk, and refreshes executive Looker dashboards in BI dashboards. The workflow begins when S3 ingestions come in, which go to Snowflake. When the data arrives in Snowflake, we have triggers written on Astro by Astronomer. From there, the DBT pipeline takes over, writing it into staging for clean and deduplicated data, with quality tests validating success or failure, including any malformed data issues. This way, we utilize Astro by Astronomer precisely, rather than writing messy bash operators to execute DBT runs as a single opaque block. Astro by Astronomer, developed by Astronomer Cosmos, alleviates this by allowing us to manage individual DBT models and test them in discrete observable Airflow tasks. It makes life easier by adhering to Astro CLI standards, which are module-based and consistent. It greatly aids us with granular model observability that is not found in a single DBT test and offers zero downtime secret and environment management, ensuring the environment remains consistently up. Moreover, it provides frictionless local development by allowing data engineers to run Astro dev start, spinning up a Docker context that mimics the production Airflow one-to-one, allowing for testing DBT models against a local development schema, pushing branches into GitHub, and letting CI/CD automatically deploy updated DAGs to Astro by Astronomer cloud.

Astro by Astronomer offers multiple features that depend on projects. In our case, I would say the best features include developer tooling and automation. Astro by Astronomer provides the Astro CLI, creating local parity where developers can work locally in a Docker container mimicking a production environment, which is fantastic. Additionally, Astronomer Cosmos is an open-source framework natively integrated into Astro by Astronomer that automatically converts DBT projects into fully observable native Airflow DAGs, allowing every DBT model to be tested and become an individual node in the Airflow UI without needing manual wrappers. This feature saves a lot of development time.

Furthermore, it has Auto AI, which assists with DAGs, troubleshooting task failures, and generating automated migrations across different Airflow version upgrades while enabling integration with various AI and MCP servers. It also provides no-coder blueprint authoring, functioning as a drag-and-drop workflow. On observability and data lineage, it presents an excellent pictorial landscape of data flows and lineage management. Cost optimization is also vital, with dynamic auto-scaling based on workload without needing to worry about scaling nodes under the hood. It offers zero downtime, and it includes a fantastic Terraform provider from a DevOps perspective.

The feature that has made the biggest difference for our team among all these offerings depends on the project at hand. I would say it is the cloud-native hosting that treats Airflow as a virtual machine. When using open-source Airflow or basic cloud providers such as AWS-managed Airflow MWAA or GCP Cloud Composer, the vendor simply spins up EC2 or Kubernetes instances and installs Airflow on top, leaving the rest to you. You still have to debug slow DAG parses, manage complex CI/CD containers, and build custom lineage connections. Astro by Astronomer operates at the pipeline and code level, which is vital. This allows Astro by Astronomer, designed by Astronomer, to streamline how data engineers write, test, and observe data workflows, enhancing scaling, cost efficiency, and maintaining zero downtime with automated upgrades.

Astro by Astronomer has positively impacted my organization by significantly improving developer productivity, reducing downtimes, and alleviating overhead associated with upgrades. The acceleration in developer velocity and time to market is evident, as our cost optimization efforts have led to a nearly 45% reduction in cloud spending, translating to a high return on investment.

Astro by Astronomer is doing great, and while improvement opportunities exist, particularly around the cost model, which can occasionally feel steep, I generally find it quite favorable. For smaller teams, the cost can be quite high; self-hosted Airflow or AWS MWAA options are cheaper. For larger companies, it seems acceptable. Aside from that, I do not have any other concerns, but perhaps adding advanced governance such as streamlining RBAC directly into the UI could be beneficial, which may not currently be available.

I have been using Astro by Astronomer for approximately five years.

Astro by Astronomer is definitely stable, which is one of the key reasons we chose it.

The scalability of Astro by Astronomer is very good.

Customer support for Astro by Astronomer is very good and top-notch.

I definitely see a return on investment, as the time to market has notably decreased. Fewer employees needed is not a point for discussion since we are already a lean team; the current number of employees is optimal for our work with Astro by Astronomer. When I mention a 45% reduction in cloud spend, it was measured against a previous solution we used, which was managed Airflow web services. Now, with Astro by Astronomer, our costs are significantly lower. However, it is not a definitive metric, as various factors come into play. It is essential to note that not the entire team has fully adopted Astro by Astronomer yet, which is vital for context.

My experience with pricing, setup cost, and licensing has been very good.

The accuracy of Astro by Astronomer's AI capabilities is very good, and its reliability is also very good. The advice I would give to others looking into using Astro by Astronomer is that it is a good product with great support and excellent AI integrations, making it suitable for industry scale. I would rate this product as an eight out of ten.

Hybrid Cloud

Amazon Web Services (AWS)

I have been using Astro by Astronomer CLI as a local development environment for about a year. During this period, it has been the main tool I use to develop, test, and validate my Apache Airflow projects before deploying them to production. In production, I use Apache Airflow together with Astronomer Cosmos to orchestrate dbt pipelines.

My main use case for Astro by Astronomer is using the CLI as a local development environment for Apache Airflow projects. I use it to develop, test, and validate DAGs before deploying them to production, ensuring everything works correctly. In addition, in my day-to-day work, I use Astronomer Cosmos to integrate and orchestrate dbt pipelines with Apache Airflow, running data transformations on Amazon Athena with Apache Iceberg tables.

In my day-to-day work, I use Astro by Astronomer CLI to develop and test new Apache Airflow DAGs locally before publishing them to the production environment. For example, when I need to create a new data pipeline with dbt using Astronomer Cosmos, I first validate all the orchestration locally with Astro by Astronomer CLI, check that the dependencies, tasks, and integrations are working correctly, and only then do I deploy it to the production environment. This reduces errors and makes the development process much faster and more reliable.

In addition to using Astro by Astronomer, I also use Astronomer Cosmos to integrate dbt with Apache Airflow. This makes creating and maintaining DAGs much easier because Cosmos automatically generates the dbt tasks, respecting the dependencies between models, making orchestration simpler and more organized. In our environment, dbt runs happen in containers on Amazon ECS, using AWS Fargate. This model allows us to start resources only during pipeline execution and shut them down afterward, avoiding having dedicated servers running all the time. As a result, we were able to reduce infrastructure costs, maintain a scalable environment, and execute transformations efficiently, especially for workloads that do not need to be active continuously.

The main value of Astronomer Cosmos is simplifying the development and operation of pipelines with Apache Airflow. Astro by Astronomer CLI offers a very consistent local development experience, while Astronomer Cosmos makes it easier to integrate dbt with Airflow by automatically generating the DAGs and respecting the dependencies between models. In addition, this approach integrates very well with Amazon ECS and AWS Fargate to run the dbt jobs. This allows us to scale on demand and pay only for the resources used during pipeline execution, reducing infrastructure costs without sacrificing reliability and ease of maintenance.

A differentiator I consider very important is that Astronomer Cosmos is very flexible and integrates easily with other modern data engineering technologies. In my case, the combination of Astro by Astronomer CLI for local development, Astronomer Cosmos to orchestrate dbt projects, and Amazon ECS with AWS Fargate to run the jobs has brought a very efficient workflow. Besides facilitating the development and maintenance of pipelines, this architecture allowed us to reduce infrastructure costs, because the dbt containers are started only when needed and shut down at the end of execution. This offers a good combination of productivity, scalability, and operational efficiency, especially in environments that run on-demand workloads.

The positive impact of Astro by Astronomer has mainly been on team productivity and the standardization of development. Astro by Astronomer CLI made it much simpler to create and validate Apache Airflow pipelines in a consistent local environment, reducing configuration issues and speeding up testing before deployment. In addition, with Astronomer Cosmos, we were able to integrate dbt with Airflow in a much more organized way, automating the creation of DAGs.

A practical example is that we were able to significantly reduce the time needed to develop and validate new pipelines because all developers work in the same local environment using Astro by Astronomer CLI. This reduced configuration issues and decreased rework during deployments. Another important result was the reduction in costs for running dbt. Instead of keeping dedicated infrastructure running all the time, we started running the jobs in containers on Amazon ECS using AWS Fargate, which are started only when needed and shut down at the end of execution. Although I cannot share exact numbers for confidentiality reasons, we observed a relevant infrastructure cost saving, as well as a more scalable and simpler environment to operate. We also noticed a reduction in failures related to the integration between Airflow and dbt, thanks to the use of Astronomer Cosmos, which automates the creation of DAGs and ensures that dependencies between models are respected.

I believe one area for improvement for Astro by Astronomer would be to further expand the documentation and examples for more advanced scenarios, especially involving integrations with AWS, Amazon ECS, AWS Fargate, and Astronomer Cosmos. Although the documentation is good, some more complex use cases require additional research or testing until you find the best approach. It would also be interesting to offer more templates and ready-made best practices for modern architectures with dbt, Airflow, and Kubernetes, making it easier for teams that are just getting started. This would reduce the learning curve and further speed up the implementation of production environments.

The main point would be to provide more content and reference architectures for large-scale corporate environments, especially involving Airflow, dbt, Kubernetes, and AWS. This would help teams adopt best practices more quickly and reduce the time spent on architecture decisions. Otherwise, I consider the experience very positive, and Astro by Astronomer platform meets the needs of development and orchestration of data pipelines very well.

I have been working in technology for about twelve years, and specifically as a data engineer for approximately five years. During this time, I have worked at different companies and on different projects, always focused on data platforms, architecture, data processing, and cloud solutions. I currently work on the evolution and support of a data platform using technologies like Apache Airflow, dbt, AWS, and Astronomer Cosmos.

The interview was good and well structured. The questions covered the main aspects of the tool, such as user experience, benefits, improvement points, and business impact. If I could suggest some improvements, I would avoid very similar questions. At times, there was repetition, such as asking if I wanted to add something right after practically every answer. I would give more room for technical examples. Since the audience is in technology, questions about architecture, integrations, implementation challenges, and best practices would generate richer reviews. I would try to reduce administrative questions at the end—name, company, reference, contact—putting them in a form instead of asking all of them by voice. I would allow slightly more natural answers, without interrupting the interviewee between one question and the next. Overall, I found the experience positive, objective, and easy to follow.

In the flow, Cosmos unites data and ideas, and simplicity grows. I would rate this experience a nine out of ten.

Public Cloud

Amazon Web Services (AWS)

Our main use case for Astro by Astronomer in our organization is to schedule jobs, including PySpark pipelines and Managed Flink. The DAG for Flink is Cron-based, while the DAGs for PySpark jobs are asset-driven and event-driven.

An example of a workflow I schedule with Astro by Astronomer is a Cron-based job that runs every day at 6:00 PM in the evening. This DAG triggers an MSF application to run, which is pointed to Kafka. We ingest data from Kafka using Flink and store it in a Lakehouse, which is Iceberg.

Our scheduling process is based on Astro by Astronomer. Currently, we have approximately 150 DAGs and several PySpark jobs.

The best feature Astro by Astronomer offers for our use case is asset-driven event-driven scheduling. We don't need to wait or guess when our ingestion process from Flink will be done. As long as the ingestion process is complete, it automatically triggers. There are tasks that emit assets, and we can start our downstream jobs by receiving events from the assets.

This has changed my workflow compared to before. It's faster and more efficient, and we no longer need to wait or guess at what time the first ingestion process will finish. As long as it's finished, it will trigger the event and materialize an asset, which automatically starts the downstream processes.

Astro by Astronomer has positively impacted my organization because it continuously follows the latest version of Airflow as an open-source project. As long as there is a new release of Airflow, they try to catch up and release another version of Astro by Astronomer. This is a significant advantage since we mainly use assets everywhere. The development of assets is still in progress and not fully mature yet, but we chose to use it in order to progress with it and follow the improvements every time. Astronomer is trying to catch up with the latest features and releases.

There are some downsides to the event-driven scheduling features since it is new. We have faced several problems that we have managed to tackle. For example, Airflow may batch many events in one execution run, which can be problematic. Ideally, there should be one event and one execution run so that you don't miss any events. I think this will be fixed later on.

The feature can be better in other ways. We faced an issue with paused DAGs since they are asset-driven and asset-triggered. Due to regulatory requirements, we had to pause some DAGs. The problem is that the events on the asset accumulate, so when we unpause or activate the DAG, those events are still there. The next incoming event will batch and group all the events together, which causes us to miss events because there is no one-to-one relationship between events and executions.

It would be beneficial if we could handle these assets and events more selectively. For example, I would want to run only the last event or perhaps only one event among all the accumulated events. This functionality is not currently possible. I don't know if there is an open ticket to solve this, but it would be great if they could implement this capability.

Astronomer provides support from AWS. We recently had a problem and received assistance from their support team.

My advice to others looking into using Astro by Astronomer is that you must first understand how Airflow works. For example, if you want to add libraries or endpoints you need, you should know that Astro by Astronomer is essentially a managed version of Airflow. You don't need to create a scheduler, web server, or database, which is a good point as it eases your deployment process. However, you must understand how Airflow works and its components. When you have an issue or problem, you need to know how to monitor it. In my case, the AWS support team helped me check what was happening when I faced any issues. At minimum, you need to understand what's happening and what components are running behind the scenes.

Astro by Astronomer is a great tool for scheduling data pipelines. The deployment process means you don't need to worry about how it will be managed. For example, if you use Kubernetes to deploy your Airflow, it will be challenging at the beginning. Astro by Astronomer addresses this problem by managing infrastructure for you. I would recommend using it. I have given this review a rating of seven out of ten.

Public Cloud

Amazon Web Services (AWS)

I have been using Airflow daily for the last six years as my main product and primary tool to orchestrate data ingestions and process data.

I use Astro by Astronomer to orchestrate data ingestion and to call GCP tools or AWS tools, such as EMR jobs or Dataflow jobs. It also calls Cloud Functions, Lambda, and everything necessary to ingest and process data, including DBT processes.

Currently, I am trying to use Astro by Astronomer for my local development to create a Docker image faster and more easily, allowing me to have my own environment to practice and to use as a sandbox for learning and trying different approaches.

Astro by Astronomer has many integrations out of the box, making it easy to use and to access documentation. Astronomer is a good company to support that.

Astro by Astronomer offers many integrations out of the box with good documentation and a certification workflow.

Astro by Astronomer makes it easier to create retries and send emails, which is very useful.

Astro by Astronomer is consistent and dependable because you have governance over logs and task runs. I can understand the whole workflow for Airflow and use CI/CD to change code and create and modify DAGs.

Astro by Astronomer needs something to debug it, not only to print or log the system but an IDE to put a breakpoint and develop it inside Airflow without waiting for the Airflow job to compile the files and code.

I would like to have the possibility to put a breakpoint inside Airflow and show the variables, the call stack, and anything else on the stack to understand better while developing.

Astro by Astronomer is a complete tool, and with a small change it will improve significantly.

I have been using Airflow daily for the last six years as my main product and primary tool to orchestrate data ingestions and process data.

Astro by Astronomer is very stable.

Astro by Astronomer's scalability is very good. Sometimes we experience performance bottlenecks, depending on the time we are running DAGs simultaneously.

Astro by Astronomer is my first solution for workflow orchestration.

It is a simple product to install quickly, has a lot of documentation, and it is very similar to Airflow. It has the same trunk code.

Astro by Astronomer is a standard tool for data engineering, so I did not analyze any other tools to use.

Because I have been using Astro by Astronomer from the start of the project, it is a handy tool to orchestrate ingestion as a data engineer. Therefore, it has not changed our process but is a part of it.

For me, Astro by Astronomer deserves a rating of 10 because I believe I use it correctly—Airflow as an orchestrator, not as a runner.

If you use Airflow as it was designed, you will experience the full improvements the tool was made for.

Now, the company has installed MWAA instead of using Astro by Astronomer.

I do not have a relevant metric to show, but common sense tells me it is a good tool to have governance and simple maintenance for many processes.

For me, the community around Airflow is growing every day and is very well documented. I am following Marc Lamberti regarding the new features on LinkedIn, and for now, it is sufficient and very useful.

The certification workflow created by Marc Lamberti is great and fantastic because it covers many topics for Airflow and is a good exercise and place to understand Airflow better. It is also good to demonstrate to the community that I am proficient in Airflow.

Public Cloud

Amazon Web Services (AWS)

I use Astro by Astronomer in the main case of data pipelines orchestration and data quality.

It is useful because more than one person in the team can work on the data pipeline, seeing if the stages and the tasks were run successfully. If not, they can fix to run properly and to manage, build, or maintain a data pipeline that exists, not only one that was built for our team but was given from a business team, for example in the same company or from another team that was merged with our current team at our company.

I think that the best features from Astro by Astronomer are the data observability from the whole pipeline and the ability for more than one person from the same team to work in the same data pipeline or manage many data pipelines as we want to have. With orchestration, observability, data quality, reduced costs, and an integrated framework with other tools such as Databricks, Azure, AWS, or GCP and data tools, we can cover all the stages from the data pipeline from the beginning, from using APIs, from ingestion, from treating the data, and from the last mile that we can call as dashboards or machine learning studies or data scientist models that we can apply to our data from the last mile in the data pipeline. Astro by Astronomer can help us with that.

The biggest impact for our team is that we can integrate the tools and reduce costs and facilitate the maintainability of our process. We can observe the data, correct and fix problems when an issue appears, and report to our leader and the business teams the current status of the whole ingestion process of the data, guaranteeing the data quality and the reduced costs of our process.

I think that Astro by Astronomer is a good example of how data orchestration is important to the company when we think about big data and ingestion from many external companies or external institutions that bring to us data that is important to provide final products that can provide to the company the best decision-making in the future for the company regarding revenue, costs, and guidelines for the company that want to grow at a very fast pace.

Astro by Astronomer could impact the company in a way that the data teams and the business teams can learn in a facilitated and easy way how to manage data pipelines, how to manage tools from cloud to observability and to data visualization, and can manage the whole process and act in an easy and fast way when it is needed by our members in our team and in our company. It is important to have a tool that can show to us in an easy way the whole process and where we should act when an issue or some problem is occurring and how we can handle that and provide the perfect solution to reduce the impact in the company.

Especially in data quality, we have some kind of tool that we managed to have before Astro by Astronomer that we got the status of our pipelines from many different tools that was difficult to manage, and it was difficult to have the observability for the main pipelines that we had at that time. With Astronomer, we can synthesize in the same platform all the data pipelines and provide the best rules and the best technical cases to apply data quality to our huge amount of data that we ingest daily and have reduced costs and improve our time spending in fixing problems in the data pipelines and the data infrastructure that we have in our company.

Astro by Astronomer can be improved always by integrating and being in attention with the capabilities that are emerging in the technology as well. For example, AI models that we are building here in the company and how we can provide the observability for these processes that are emerging, talking about machine learning products, model products, GenAI products, chatbots, and agent AIs and all the processes that we are building and that we can and want to have observability and maintainability of this process in a very synthesized and very fast pace way of solving problems and acting when it is needed by our team.

It is so important to have the documentation updated and have some kind of tutorials in text and examples or some kind of AI product that can help a new member of the team or the actual members of the team to be updated with the improvements that Astronomer is doing to the product. It is important to keep the team updated to the products and how we can suggest some improvements thinking about new technologies that are emerging in the tech world nowadays such as AI and all the products that emerge from this theme, including agent AIs, GenAIs, chatbots, and machine learning models.

I think that there is always a margin of improvements thinking about how we can provide updated documentation, updated tutorials, use AI models to help our team to keep updated with the new technologies, improve the interface, improve how we can access the interface such as in a web app, mobile app, or desktop, and find new ways to keep updated with our data products such as pipelines and data quality in these pipelines. We always have some kind of improvements to do in the product and certainly in the future Astro by Astronomer will keep updated with all these new capabilities that the product can have to integrate not only the data pipelines and orchestration, but the cloud architectures, principal institutions such as Oracle, AWS, Azure and GCP, and the last mile of the data ingestion such as dashboards, data modeling, and machine learning models or agent AIs and AI products as well.

It is always important to have a good knowledge of data governance nowadays because not only is it good to have data products in the last mile as I mentioned before, but it is important to have the data governance in our team to know what we are producing, how we are storing this data, how we can keep our data safe in our environment to be attending the requirements of the LGPD and the laws that are in the countries that we provide our products and to know how we can manage this amount of data with observability, maintainability, and how to manage the data to specific teams in the company that need to access the data but actually do not have the high level of technical information that a data engineer can have in the daily work as a data professional in the company.

For now it is good enough, but it is always important to keep updated with the latest models and the improvements that the AI tech is providing in the tech world. It is important to know that AI is improving and Astro by Astronomer can keep updated with this. For now, to answer the question about accuracy and reliability of the outputs, I think it is good enough and provides feeds and correct answers to address the questions that the data teams are interested in knowing by the tool that we are using.

I have been using Astro by Astronomer for three years.

It is stable.

It is good enough. We can build and have many kinds of data orchestrations and provide observability and maintainability in a simplified way to our team members.

It is good enough for us. They provide to us important support when needed to build new products or to manage products that were already built.

I have used open source solutions and I use some kind of native solutions by other tools such as cloud services that have orchestration, Databricks as well, and DBT have documentation in SQL. Astro by Astronomer can simplify and unify all these features in a low cost way. I switched because I knew that Astro by Astronomer was a relevant and proven or tested solution to data orchestration in many companies. The benchmark was made and the tool was chosen.

I evaluated other options with integrated data orchestration in external tools such as GitHub Actions or some kind of open source products to orchestrate data. Astro by Astronomer was chosen because our benchmark with other companies, especially big ones, proved to be reliable to our needs talking about data quality and data orchestration in a huge amount of data ingested daily.

It is important to understand how the product works and how you can understand all the features and get the maximum value that the product can bring to you talking about a huge amount of data, big data pipelines, and using CLI and APIs to connect the cloud environments and to have proper observability, maintainability, and connect with security and safety environments to your data. Finally, you can have data governance with low costs compared to other strategies in the data architecture and have observability in the whole process. You can handle situations, issues, and problems that can emerge in a daily work of data ingestion. I would rate this product an 8 out of 10.

Hybrid Cloud

Amazon Web Services (AWS)