Apache NiFi is used mainly for ETL to get data from multiple sources and then load it into a data lake. For example, data from Ab Initio is extracted and then loaded to an S3 bucket. From the S3 bucket, that data is read again and then loaded into other data layers. Different data layers, such as raw and raw silver, all use Apache NiFi.
Certified Apache NiFi - Calculated Systems
Calculated SystemsExternal reviews
8 reviews
from
External reviews are not included in the AWS star rating for the product.
Data flows have transformed and now support reusable ETL pipelines across diverse sources
What is our primary use case?
How has it helped my organization?
Apache NiFi has positively impacted the organization by making development really easy, allowing efficient design and development, and enabling code reuse which has reduced the development effort. Integration with Git is also really good for sharing code across teams. A reduction in development effort of about 30% has been observed.
What is most valuable?
The best features Apache NiFi offers include the ability to connect to any type of sources, which is a big advantage since most connectors are already available and do not need to be created. In transformation, it has a wide variety of transformations that can be used across big data and any kind of data, including JSON formatted data.
The connectors used most often include connecting to the Oracle database as the main focus and then connecting to log data, which have greatly benefited the team.
What needs improvement?
Improvements in the user interface to make it easier to use would be beneficial, and adding more security features would make Apache NiFi more secure and robust. Documentation and support could also be enhanced, as most support is usually received from users rather than from the product owners.
For how long have I used the solution?
Apache NiFi has been used for more than five years.
What do I think about the stability of the solution?
Apache NiFi is stable.
What do I think about the scalability of the solution?
The performance of Apache NiFi is really good. Based on the workload, more nodes can be added to make a bigger cluster, which enhances the cluster whenever needed. Apache NiFi's scalability is good and it is auto-scalable, which is pretty impressive.
How are customer service and support?
Initially, customer support was contacted more often, but after understanding Apache NiFi better, not many issues have been faced. The customer support is really good, and they are helpful whenever concerns are posted, responding immediately.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
Before Apache NiFi, StreamSets was used, and it was unsatisfactory as it consumed all the server resources and did not release them. After finding Apache NiFi, it became a preferred solution.
What was our ROI?
Apache NiFi provides a good return on investment, although specific cost details are not known since that is not an area of involvement. Feedback has been given to stakeholders that Apache NiFi is beneficial in development and does great work, indicating a positive return on investment.
Which other solutions did I evaluate?
Before choosing Apache NiFi, other options such as StreamSets, Teradata, Informatica Big Data version, and Glue were evaluated, but Glue was not chosen due to its high cost.
What other advice do I have?
To train or onboard new team members to use Apache NiFi, the code has been modularized and processes have been documented really well, so when new team members are onboarded, they are asked to review that documentation to understand the processes and the modules that have been created. In a couple of days, once they go through all that material, they are up to speed.
In terms of flexibility and ease of use, Apache NiFi is more open compared to other ETL tools that have been used, such as Informatica and Teradata. It is open source with many contributors and can handle various data sources, including log data, structured, unstructured, and semi-structured data, unlike traditional ETL tools.
Tools such as Prometheus and Grafana are sometimes used to keep an eye on Apache NiFi server, and DataDog is also used along with it. Scaling Apache NiFi workloads is managed through auto-scaling.
To handle data security and compliance when using Apache NiFi, LDAP authentication is utilized, all clusters and nodes are kerberized, and single sign-on is used to authenticate. In transit, SSL encryption is used, and at rest, AES encryption is used, which is more than enough for the needs.
Apache NiFi is kept up to date by keeping an eye on new features that have been released, discussing them internally to assess if they need to be incorporated into development. If there are any gaps in the current version, an upgrade to the new version will be attempted.
Apache NiFi is a pretty good tool that meets most ETL needs, and in terms of performance and security, it is really good. After using it for quite some time without any issues, it is recommended as the number one tool for ETL. The overall review rating for Apache NiFi is 8 out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Daily workflows have integrated diverse logs and have delivered flexible data orchestration
What is our primary use case?
I have been using Apache NiFi virtually daily, as it is part of my main responsibility in my current role.
My main use case for Apache NiFi involves integrating various data sources and performing transformations to load them into mostly our NoSQL database, Elasticsearch, but sometimes into other databases as well.
For integrating and transforming data, we receive a lot of logs generated with our AWS services that the company wants to collect, particularly for our security team to review those logs and ensure they can conduct their security checks and reviews to confirm there is no abnormal behavior. We use Apache NiFi to capture those logs sent to many S3 buckets, collect those logs, decompress them with Apache NiFi, perform any necessary transformations, and send them to Elasticsearch so that end users, often from the network team or security team, can then use Elasticsearch and Kibana for data analysis.
My advice for others considering Apache NiFi is that if you are willing to, you can use it on-premises; it offers great customizability. While it is specifically designed for streaming data, it can also accommodate batch data. Moreover, it is useful for various out-of-the-box solutions, including unique uses such as email notifications, showcasing flexibility in data orchestration, ETL, and other applications.
What is most valuable?
Apache NiFi offers great flexibility in terms of whether you want to be a low-code user or a high-code user, especially if you are a Python or Java developer, thanks to the recent addition of custom-built processors in the latest versions of Apache NiFi where you can use Python or Java to create your own processors versus using the great selection of out-of-the-box processors already available in Apache NiFi to do almost anything. If you are willing to put together a complex web of processors, you can do almost any data transformation you want, but the customizability with making your own processors, again with Python or Java, has been a huge benefit for performing both what Apache NiFi is specifically made to do and some more out-of-the-box solutions, such as creating some kind of email notification system as well. This kind of use with Apache NiFi has existed even before the implementation of custom processors. You could create scripts, even putting them in Python in Apache NiFi using the execute script methods, and this has existed before, but now it has even better functionality with the latest version of Python rather than just a Jython type of hybrid. Those are some of the best things that it offers.
The flexibility of Apache NiFi has helped me in my daily work, especially because instead of utilizing a bunch of Apache NiFi processors, which we do use for most of our processes, it can be much easier to combine transformation logic within Python processors since the majority of our team prefers Python programming as our choice of language. This integration allows us to put it all in one place. We can integrate Apache NiFi with our Python processors that we host on a Git repository, which integrates very well, and we can manage the same scripts and make changes efficiently. It is great coming from a Python developer mindset shared amongst the team.
Apache NiFi has positively impacted my organization as it continually improves functionality and throughput with each iteration over the past three years. One of the big tradeoffs with open source is that how well it functions is largely dependent on the user, but that means you can adapt it to whatever custom use case you have. We have been able to consolidate several different authentication methods through just Microsoft, and Apache NiFi has been helpful in facilitating that. Additionally, due to its many ways of extracting data from different sources, we can develop specific solutions ourselves, allowing us to integrate various data sources. Thanks to the open-source customizability, we can adapt Apache NiFi to our built cluster, which has numerous benefits, particularly since we are managing many of our processes. This approach saves us significant costs compared to moving to something more managed or on the cloud, as managing open-source technologies ourselves ultimately reduces expenses.
Regarding cost savings, I do not have a strict idea of how much we have saved since the company was already using Apache NiFi when I joined, but I am certain comparisons have been made against other ETL or data orchestration tools that are popular among different cloud providers such as AWS or Azure. The cost savings must be significant, particularly given that we are handling terabytes and petabytes of data daily, trying to find software that allows this in an affordable manner. It is clear that substantial savings exist, as long as we manage our own clusters and bugs effectively. The tradeoff with managed services is that they handle much of this, ensuring uninterrupted service, but these come at a cost. Conversely, with open-source software management, we incur no costs as we handle everything ourselves.
What needs improvement?
I believe Apache NiFi could be improved with easier, out-of-the-box provided monitoring solutions. While Apache NiFi has an API that generates logs, it would be beneficial to have simpler access to that data saved historically. It would assist in easily retrieving data for historical analysis and storing it elsewhere without the hassle of setting up APIs and delving into documentation. Just having a more streamlined approach to collecting this data would be greatly advantageous.
I would suggest continuous improvements regarding the custom developer-built processors, as many times the errors that arise are not useful. We often seem to struggle with a combination of implementing our own error handling or analyzing logs, as the information does not always align or proves unhelpful. Continuous enhancement in this area would be wonderful, so we do not need to decipher which error is more accurate or which report gets us nearer to the actual problem. For instance, I encountered a situation where flow files would not process; they were retried but returned to the queue before the Python processor due to ambiguous errors. It eventually turned out that the issue was the flow files' size being too large for the Python processor, which we only discovered by splitting the flow files, at which point the issue resolved. The initial error did not indicate it was related to memory or size limitations but appeared as a parsing error or something similar.
For how long have I used the solution?
I have been working in my current field for about three years.
What do I think about the stability of the solution?
Apache NiFi is now more stable than before.
What do I think about the scalability of the solution?
Apache NiFi's scalability is good. You can scale it up as long as you have the machines and servers available. If you have room for more instances, scaling up is fairly straightforward, provided you manage configurations effectively.
How are customer service and support?
Apache NiFi's customer support is good.
How would you rate customer service and support?
What was our ROI?
I have definitely seen a return on investment through time savings. Working with Apache NiFi allows us to manage it more efficiently, transitioning from spending hours or days resolving issues to requiring much less intervention now. Thanks to improvements on both our side in how we run processes and enhancements to Apache NiFi, we have reduced the time commitment to almost not needing to interact with Apache NiFi except for minor queue-clearance tasks, allowing it to run smoothly. At this point, we have certainly saved hundreds of hours.
What other advice do I have?
The customizability of Apache NiFi helps even with unique use cases, as I mentioned before, given that Apache NiFi can be used in this capacity. While there are better applications or software options available, when you are trying to keep it simple and finding ways to utilize a couple of processors for a unique solution, you can do that in Apache NiFi. For example, we have several notification-type pipelines we have built in Apache NiFi, such as reading from a SQL database to identify users who have not completed training and then sending them an email reminder to complete that training. We have that running regularly, week by week. Another instance involves a processing data flow that scans for specific data found in logs, which triggers an email notification to the relevant team letting them know that a unique identifier has appeared, allowing them to handle the situation.
I encountered some odd cases such as increasing concurrent threads on a processor, which should work similarly to copying several processors, yet functional throughput varies. It seems that using a distributed processor yields better throughput than just increasing the concurrent threads on one processor, which has been odd but is a workaround we had to adopt to boost throughput. Resolving such quirks could elevate the rating further.
I rate Apache NiFi an eight out of ten. I choose eight because, as open-source software, there is always room for improvement, but the tradeoff between learning how to use the software and the savings it provides, along with its customizability, ranks it pretty high. It is effective for what it does and continues to improve, so it could score higher if there are significant enhancements in custom-built processors and ongoing improvements in functionality.
Which deployment model are you using for this solution?
On-premises
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Plug-and-play data flows have transformed how our team builds and runs large daily ingestions
What is our primary use case?
I have been using Apache NiFi for a couple of years now. My experience with Apache NiFi is that I have used it for a customer of ours who used that system to orchestrate ingestion processes from different sources into AWS. My main use case for Apache NiFi is data ingestion, so connecting to sources like APIs or different data lakes to get the data into S3. I manage more than 50 ingestion flows currently; some of them are quite small, dealing with a few megabytes a day, while others are multiple gigabytes a day and those are run daily, so we have a daily refresh of the sources.
What is most valuable?
Apache NiFi's best features include its plug-and-play solution, which means you don't need a lot of insight or knowledge to use it. The simplicity and plug-and-play approach of Apache NiFi has helped our team by allowing our customer to scale and build different ingestion flows, which previously needed additional development effort because custom solutions were required; now we just plug and play.
The features of Apache NiFi, including the integration capabilities to connect to different sources and monitoring with all the observability features, work really well for our team in obtaining the needed information. Apache NiFi has positively impacted our organization by making us a lot more productive; I would say development has significantly improved.
Development has improved with a reduction in time spent being the main benefit; before we needed a matter of days to create the ingestion flows, but now it only takes a couple of hours to configure.
What needs improvement?
I don't have any frustrations about how Apache NiFi can be improved; I'm pretty happy. I don't think there are needed improvements for Apache NiFi, particularly regarding the user interface, documentation, or performance.
For how long have I used the solution?
I have been working in my current field for five years now.
What do I think about the stability of the solution?
Apache NiFi is indeed stable.
What do I think about the scalability of the solution?
Apache NiFi's scalability works for us.
How are customer service and support?
I haven't interacted with their customer support.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I have evaluated other options before choosing Apache NiFi, and I have used other options as well.
How was the initial setup?
Apache NiFi is deployed in our organization using public cloud infrastructure. I purchased Apache NiFi through the AWS Marketplace. There are no improvements needed for Apache NiFi deployment on AWS, as the experience with that was pretty smooth.
What about the implementation team?
I don't manage pricing, setup cost, or licensing for that customer, but I think they are pretty satisfied with it as well.
What other advice do I have?
My advice for others looking into using Apache NiFi is that it's a good solution. I would rate this review 9 out of 10.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Low-code orchestration has bridged on-prem and cloud workflows and now streamlines data sharing
What is our primary use case?
My main use case for Apache NiFi is orchestration; I kickstart the job and then pass on the handler to Databricks or AWS to run the ETL pipeline.
A specific example of a workflow where Apache NiFi plays a key role is when there is an on-premises Hadoop system and a cloud component. Because of the company's policy regarding firewalls, the data cannot be directly moved through services such as Kinesis or DMS integrating directly with on-premises resources. Apache NiFi works as an orchestrator and a middle tool to get the parameters to trigger the job and then pass on the handler to the cloud services. Because of the firewall, Apache NiFi comes into the picture. Another use case for Apache NiFi is once the data is created in S3; I can extract a subset of the data and send it as an SFTP for outside recipients.
I have another scenario regarding my main use case with Apache NiFi; there is a use case for synthetic data, and we are using synthetic data generative AI software to synthesize data in the cloud environment. Now for users who are not on the cloud and want to access the synthetic data, Apache NiFi is used to pull the data back from the cloud.
What is most valuable?
The best features Apache NiFi offers include drag and drop capabilities; I would not say it is no-code, but it is pretty low-code.
The drag-and-drop interface of Apache NiFi has helped my team by cutting down on development time, allowing us to focus more on the mapping part and testing part of it.
In terms of features, if a job requires a predetermined mapping for the metadata, Apache NiFi comes in handy to map for the metadata, which is needed before an external file can be validated and pushed onto the cloud for processing. This is better than traditional systems where we would have to sit down and map individual attributes and their data types to create that metadata interface.
Apache NiFi has positively impacted my organization by definitely bridging the gap between the on-premises and cloud interaction until we find a solution to open the firewall for cloud components to directly interact with on-premises services.
What needs improvement?
Regarding improvements in Apache NiFi, there is scope; it should gear more towards an AI mechanism, especially for metadata generation. If I want to integrate Python code or embed Python code within the Apache NiFi parameters or workflows, it should come with AI integration that can assist in generating code based on user requirements. Gearing more towards a no-code mechanism will really enhance Apache NiFi's productivity, as customers would want that.
About needed improvements, I think integration with other tools would really help in the age of AI. Apache NiFi should have APIs or connectors that can connect seamlessly to other external entities, whether in the cloud or on-premises, creating a plug-and-play mechanism.
For how long have I used the solution?
I have been using Apache NiFi for roughly one and a half years.
What do I think about the stability of the solution?
The biggest challenges I have faced while using Apache NiFi revolve around reliability; I have seen Apache NiFi crashing at times, which is one of the issues we have faced in production. It does not happen often, but there have been instances where the NiFi nodes crashed, and sometimes we encountered issues retrieving data where it was unable to establish connectivity.
To troubleshoot or resolve the Apache NiFi crashes and data retrieval issues, my team primarily replicates the same scenario in development to see where the issue lies. On one occasion, the failure was linked to an authentication mechanism change at the enterprise level.
In my experience, Apache NiFi is stable.
What do I think about the scalability of the solution?
Apache NiFi's scalability has not been an issue yet; I would say it is pretty stable. Depending on the workload we process, it remains stable since at the end of the day, it is just used as an orchestration tool that triggers the job while the heavy lifting is done on Spark servers.
How are customer service and support?
I have not needed to reach out for help regarding customer support for Apache NiFi.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
What about the implementation team?
We have a dedicated team that handles version upgrades and maintenance for Apache NiFi.
What was our ROI?
I would say there has been time saved to an extent, but not in terms of resources when evaluating return on investment.
Which other solutions did I evaluate?
I was not involved in evaluating other options before choosing Apache NiFi.
What other advice do I have?
On a scale of one to ten, I would rate Apache NiFi an eight. I chose eight out of ten because there is scope for improvement; in this age, folks want more automation, which is why I kept it at eight. My advice for others considering using Apache NiFi is that it is a good tool, but there are also other options to consider before making a decision. I gave this review an overall rating of eight out of ten.
Data ingestion has accelerated and now supports flexible API integration and custom transformations
What is our primary use case?
Apache NiFi is used to orchestrate ingestion processes. For example, Apache NiFi ingests data from external sources such as external databases or external APIs. Custom transformation is then applied, and data is written inside the data lake.
How has it helped my organization?
Apache NiFi speeds up ingestion pipelines development. Ingestion pipelines that usually took a week to develop can now be developed in a couple of days.
What is most valuable?
Apache NiFi has extensive integration capabilities and integrates with many sources. It supports custom transformations, making it a very flexible tool that can be leveraged to perform most computation needs.
For transformation with Apache NiFi, JSONs are processed and denormalized to map information onto different tables. For source integration, the most valuable aspect was the ingestion from external APIs.
What needs improvement?
Apache NiFi is a very good tool, but there is room for improvement.
For how long have I used the solution?
Apache NiFi has been used on different projects for a couple of years.
What other advice do I have?
Apache NiFi should be considered if a scalable and flexible tool is needed for building ETL pipelines and reducing time to production. This review has a rating of 8.
Data workflows have accelerated project delivery and reduce costs for analytics teams
What is our primary use case?
Apache NiFi is used for real-time and batch ingestion on data warehouse platforms. For example, Apache NiFi ingests all analytics from the e-commerce website into the data warehouse in the AWS Redshift database.
How has it helped my organization?
Speeding up projects with Apache NiFi has helped the organization by resulting in cost savings. A 30% reduction in cost was noticed as a specific metric regarding those savings.
What is most valuable?
The best feature of Apache NiFi is the simplicity of the tools because it is a drag-and-drop tool. The simplicity of Apache NiFi's tools helps by speeding up all the implementation process. Apache NiFi is also used to speed up projects in order to gain more projects in less time.
What needs improvement?
Apache NiFi is a good product as it is currently.
For how long have I used the solution?
Apache NiFi has been used for a long time, five years in different projects.
What do I think about the stability of the solution?
Apache NiFi is stable.
What do I think about the scalability of the solution?
The scalability of Apache NiFi is good because it is simple to scale up the resources.
How are customer service and support?
The customer support for Apache NiFi is fine. I would rate the customer support of Apache NiFi a 10 on a scale of 1 to 10.
How would you rate customer service and support?
Positive
Which solution did I use previously and why did I switch?
A custom solution implemented with Python was previously used before switching to Apache NiFi. The decision was made to switch from the custom Python solution to Apache NiFi to simplify all the deployment.
How was the initial setup?
Apache NiFi was purchased through the AWS Marketplace.
What was our ROI?
A return on investment has not been observed, and it is not possible to share these metrics.
What's my experience with pricing, setup cost, and licensing?
The experience with pricing, setup cost, and licensing was fine, as the integration with the AWS Marketplace was very good. The pricing in Italy is considered a little bit high, but the product is worth it.
Which other solutions did I evaluate?
Other options were not evaluated before choosing Apache NiFi.
What other advice do I have?
Apache NiFi receives a rating of 9 out of 10. This rating of 9 out of 10 for Apache NiFi was chosen because of the documentation and the support of the product. The advice for others looking into using Apache NiFi is to test the solution with a POC and then go to production in a quick way.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Visual workflow offers clarity and boosts data pipeline construction
What is our primary use case?
I am implementing the ETL workflow using Apache NiFi to prepare data and upload it to the cloud. Our use case involves importing data from on-premise and private servers to build a data hub and data mart. The data mart is then published on the cloud.
How has it helped my organization?
We primarily use Apache NiFi for data preparation tasks.
What is most valuable?
The visual workflow aspect of Apache NiFi is an invaluable feature as it operates on a no-code platform that allows for easy drag-and-drop pipeline construction. Compared to Airflow, which requires programming before visual representation, Apache NiFi offers clarity in pipeline activities. This feature greatly aids in understanding what the pipeline is doing.
What needs improvement?
The logging system of Apache NiFi needs improvement. It is difficult to debug compared to Airflow, where task details and issues are clear. With Apache NiFi, I have encountered processes that die without any traceable error, which might relate to the inadequate logging system.
For how long have I used the solution?
I have been working with Apache NiFi for about six months.
What do I think about the stability of the solution?
Sometimes, when I run Apache NiFi, processes crash without any clue, which might relate to the logging system. The process can die, and the logs do not show any detail to identify the problem, impacting stability.
What do I think about the scalability of the solution?
For scalability, I would rate it an eight. We can run parallel pipelines simultaneously without issues unless memory is full. Scarcity of memory is the only constraint, but processing capabilities allow us to handle much simultaneously.
How are customer service and support?
The technical support from the official Apache team is rated a three out of ten. Issues often require self-resolution or community help, as the support isn't effectively managed.
How would you rate customer service and support?
Negative
Which solution did I use previously and why did I switch?
I have used Airflow before, which required programming first and then visual representation of the workflow.
What about the implementation team?
There is another team responsible for setting up Apache NiFi, so I'm not involved in the deployment process.
What's my experience with pricing, setup cost, and licensing?
Apache NiFi is open-source and free. Its integration with systems like Cloudera can be expensive, but Apache NiFi itself presents the best pricing as a standalone tool.
Which other solutions did I evaluate?
Prior to Apache NiFi, I used Airflow, which differed mainly in its approach to programming and workflow visualization.
What other advice do I have?
Overall, I rate Apache NiFi an eight out of ten. I am quite happy with it.
The tool enables effective data transformation and integration
What is our primary use case?
I use NiFi as a tool for ETL, which stands for extract, transform, and load. It is particularly effective for integration methodologies.
The tool is useful for designing ETL pipelines and is an open-source product. Data is often stored in different forms and locations. If I want to integrate and transform it, NiFi can help load data from one place to another while making transformations.
I can handle stream or batch data and identify various data types on different platforms. NiFi can integrate with tools like Slack and perform required transformations before loading to the desired downstream.
It is primarily a pipeline-building tool with a graphical UI, however, I can also write custom JARs for specific functions. NiFi is an open-source tool effective for data migration and transformations, helping improve data quality from various sources.
What is most valuable?
NiFi works on data and file levels, streamlining real-time data processes. It is highly effective for handling real-time data by working with APIs for immediate and continuous data extraction. For real-time data tasks, this front-end UI-based tool is superior to back-end platforms.
What needs improvement?
There are some areas for improvement, particularly with record-level tasks that take a bit of time. The quality of JSON data processing could be improved, as JSON workloads require manual conversions without a specific process.
Enhancing features related to alerting would be helpful, including mobile alerts for pipeline issues. Integration with mobile devices for error alerts would simplify information delivery.
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
Scaling works well, allowing cluster expansion. However, I have never encountered very large clusters, so it's uncertain how well it supports extensive scaling.
How was the initial setup?
The initial setup is fast, especially for communication stabilization. Although the product is open source, it functions as a cluster. For single-node environments, installation is simple. For company-wide or enterprise-level clusters, the initial stages may present issues with authentication and access. Stabilization, such as port communication, may not be immediately effective.
What other advice do I have?
I recommend the product for its data privacy features. It allows secure data handling because the data is stored on my nodes. However, a skilled technician is necessary due to the reliance on Java, especially for back-end operations and error debugging.
Enterprise versions may offer easier troubleshooting. As an open-source solution, good support is crucial.
I rate the overall product as eight out of ten.
Useful to transfer data from one service to another and is user-friendly
What is our primary use case?
We use the tool to transfer data from one service to another. It helps us to migrate data from one department to another.
What is most valuable?
Apache NiFi is user-friendly. Its most valuable features for handling large volumes of data include its multitude of integrated endpoints and clients and the ability to create cron jobs to run tasks at regular intervals.
What needs improvement?
The tool should incorporate more tutorials for advanced use cases. It has tutorials for simple use cases.
What do I think about the stability of the solution?
I rate the tool's stability an eight out of ten.
How are customer service and support?
I have relied on the documentation available on Apache NiFi's website for support.
How was the initial setup?
I tried to install the tool on my work laptop, and while it worked initially, it started to run slowly after some time. The department that handles the company's databases uses Apache NiFi on proper servers. I tried using it on my laptop to see if it worked, but it ran very slowly and consumed many resources from my machine.
What's my experience with pricing, setup cost, and licensing?
I used the tool's free version.
What other advice do I have?
I rate Apache NiFi an eight out of ten.
Good monitoring, metrics capabilities and provides ability to design processors with a single click
What is our primary use case?
As a DevOps engineer, my day-to-day task is to move files from one location to another, doing some transformation along the way. For example, I might pull messages from Kafka and put them into S3 buckets. Or I might move data from a GCS bucket to another location.
NiFi is really good for this because it has very good monitoring and metrics capabilities. When I design a pipeline in NiFi, I can see how much data is being processed, where it is at each stage, and what the total throughput is.
I can see all the metrics related to the complete pipeline. So, I personally like it very much.
What is most valuable?
The good thing about Apache NiFi is that it has a concept called a flow file, and there's something called a flow file processor. The processor is the building block of your entire job. They have close to 500 processors for each purpose.
For example, for reading from Kafka, Ni-Fi has a processor called "consumer Kafka". To write to S3, they have a processor called "put S3". Now, if I read from Kafka and write my own application, I'd need to ensure the library I'm using tracks my messages. I'd also need to handle any failures by rereading messages and ensuring acknowledgment. But all this complexity is already handled by Apache processor.
They have around 500 processors, with a community investing significant effort into developing them. I can design your processor with a single click, export the entire workflow, and import it. The format is actionable, so NiFi is immediately set up.
It's also distributed in nature so that I can scale it across nodes based on the workload. These nodes share their state. If one node goes down during processing, that data might be lost, but any subsequent data is safe. Such occurrences are rare.
In essence, if you want a quick solution, Apache NiFi is a strong contender. There are other solutions like AirFlow and some paid pipeline options.
AirFlow is open-source but can be complicated. For ETL or ERT solutions, there are pricier options. But if I need a pipeline that I can monitor step by step, Apache NiFi is a good choice. It integrates with Prometheus metrics, allowing me to embed them in my workflow.
There's also a processor for integration with Slack, and I can receive notifications when the workflow is completed or fails.
Another feature I appreciate is "back pressure," which NiFi handles automatically. It maintains its own queue and addresses back-pressure issues. If, for instance, an upstream entity isn't fast enough, items get stored in a queue, managed internally by NiFi's back pressure algorithm.
What needs improvement?
There is room for improvement in integration with SSO. For example, NiFi does not have any integration with SSO. And if I want to give some kind of rollback access control across the organization. That is not possible.
So I have to create a separate username and password, and then I have to share it with the individual team. So, that is the pain point to be at the enterprise level.
For how long have I used the solution?
I have been using it for one and a half years.
What do I think about the stability of the solution?
I would rate the stability a seven out of ten because there are a lot of processes that need to be implemented.
What do I think about the scalability of the solution?
It's scalable. It can easily scale on multiple nodes. Depending on the workload, it also handles that internally; like the workers, they coordinate with each other, and they share the workload with each other. So, it's pretty good in terms of scalability.
How was the initial setup?
The initial setup is very easy, especially for users who are familiar with EDL or EMT.
NiFi is one of the easiest tools on the market to learn and use. It is also a quick-win solution, which is good for first-time users who are developing data pipelines for EMT. NiFi makes it easy to track and trace the status of your pipelines, so you can be sure that they are working properly.
What other advice do I have?
If I were to advise someone, I would ask the user what endpoints they want to touch. If I want to read something from Kafka and I want to put this thing on the S3 bucket, what is the alternative I have?
I have Kafka Connect, where I can connect Kafka with one Kafka, and I can put it into an S3 bucket. Is this scalable? No. Is this monitoring No.
We can't monitor it. We can't scale it. It's going to be a complete black box. The person who knows Kafka Connect, or Kafka, can understand what is happening there while using Kafka Connect. But if I compare it, I literally don't need to understand what Kafka is.
I know, "Okay, this is Kafka. These are the endpoints, and this is the URL I have to point to." That's it. My job is done. I will create a complete flow pipeline within, let's say, thirty minutes or something without having any current knowledge. I can read, I can Google it, and I can just implement it.
For people who are new to big data technologies like Kafka and BigQuery, I would give this solution an eight out of ten.
Let's say you need to build a solution to read from Kafka and write to an S3 bucket. You could use Kafka Connect, but if your requirements change and you need to start reading from a database instead, Kafka Connect will not work. With Apache NiFi, you can easily modify your flow pipeline to start reading from the database instead.
showing 1 - 10