Log analytics has become faster and data resilience supports growing security workloads
What is our primary use case?
At my previous company, which was a security analytics tool, my main use case for CrateDB was to ingest any kind of logs from email, applications, firewalls, DNS, Microsoft, and other technology tools. We used to ingest logs, which are small text files with information of what occurred, and after a process of going through a Kafka queue, they would be stored in CrateDB as a long-term storage option. Later, we would retrieve this data to look for anomalies in a UI-based platform.
CrateDB fit well into our pipeline and use cases in general terms. In some situations where the customer was very big or the data volume was huge, there might be a little delay because we were using CrateDB installed by us in AWS servers. Sometimes those servers were not powerful enough, so there was some delay, but after restarting it, all worked pretty well and I think it was a great solution for our use case.
CrateDB positively impacted my organization by reducing the time needed for processes. Sometimes with other data tools, like Snowflake, it would take a long time to store and retrieve all the logs quickly. Its scalability was also impressive, as it was easy to start with one server and then horizontally scale to multiple nodes to retrieve data. These aspects stood out for our use case and helped my company gain more customers during my time there.
What is most valuable?
One of the best features CrateDB offers is that it never lost data. Even with intermittent connection issues due to data volume, the data was never lost and could always be recovered. It was very fast to retrieve long queries, for example, we used to query in the UI very quickly, even with complex queries. CrateDB was fast to parse all that data and fix it for us, as well as display it on our UI platform. From a performance point of view, the speed of read and write was probably the best capability that CrateDB had, especially under stress situations and how it was able to work around them.
CrateDB's speed and reliability made a big difference for us, especially when there were big customers where the data was in gigabytes or terabytes per day. It could ingest all that data quickly and never failed in writing or reading it. This performance made a difference for our customers when choosing a security analytics tool because of CrateDB's speed with large data volumes. A tough scenario we encountered was when we had to restart the servers when CrateDB was unresponsive, but this process did not take long since they were in AWS. If the data volume was very high, it occasionally needed a restart because it could not be read perfectly fine, but generally, the performance and way it worked were very great and I did not have any complaints about it.
Integrations were great, as we used CrateDB with Kafka and other big data analytics tools like Hadoop. This compatibility between different technologies in an ETL scenario was key for us. The integrations were very important and they worked well with the mentioned technologies—Kafka, Hadoop, Logstash, and others.
What needs improvement?
One area for improvement in CrateDB could be the command line interface, as sometimes it was not very easy to understand. However, if you are technically adept, it was not a tough challenge; it was just a matter of getting used to the platform, the CLI, and the commands needed for execution.
Documentation could be better because there was not as much available compared to other storage options. Nonetheless, we were able to find the needed information, and there were colleagues with similar experiences who helped.
For how long have I used the solution?
I have been using CrateDB for almost three years at my previous company, which was a security analytics software vendor.
What do I think about the stability of the solution?
CrateDB is stable. In ninety percent of the times, it was quite stable, but as always with varying data volumes, there were occasional instances where we had to restart the servers, though this was rarely necessary.
What do I think about the scalability of the solution?
CrateDB's scalability was good; we were able to deploy it on different servers and achieve horizontal scaling when needed, especially with high customer data volumes.
How are customer service and support?
I did not have to work with customer support directly, so I do not have any complaints. We managed to fix most of the issues ourselves without needing their involvement.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
I have used Snowflake in other situations, but I mainly have experience with CrateDB. CrateDB was the first solution we chose and the one we started using right away without evaluating other tools at that time.
How was the initial setup?
Integrations were great, as we used CrateDB with Kafka and other big data analytics tools like Hadoop. This compatibility between different technologies in an ETL scenario was key for us. The integrations were very important and they worked well with the mentioned technologies—Kafka, Hadoop, Logstash, and others.
What about the implementation team?
We installed CrateDB ourselves and did not purchase it through the AWS Marketplace.
What was our ROI?
I can assert that we saw a return on investment through time saved for sure. I do not have estimates for money saved or employee reduction, as we did not experience any shortage on that front. We did save time from the configuration and setup point of view since it was fairly easy for those with technical experience in Ubuntu or other Linux environments.
What's my experience with pricing, setup cost, and licensing?
We were happy with the pricing, setup cost, and licensing of CrateDB, and I do not have any complaints. Everything was great.
Which other solutions did I evaluate?
CrateDB was the first solution we chose and the one we started using right away without evaluating other tools at that time.
What other advice do I have?
I think CrateDB did great in our use case, as it was a great solution for storing and retrieving data quickly. At the end of the day, there is a lot of parsing and steps along the way, and CrateDB was fast enough for our needs. At my previous company, they still use it up to today, and I think they are pretty happy with how it works and the kind of performance it provides.
I would advise others looking into using CrateDB to have some technical experience in the background before starting to use it to avoid running into issues during setup.
I do not have specific statistics on time saved, but for customer growth, I know we achieved an increase of thirty percent in our current customer volume once we switched to CrateDB.
My company was just a customer of CrateDB and there were no other kinds of partnerships with them.
CrateDB is deployed in various ways depending on the project and customer needs. In most cases, it is in AWS, which is a public cloud. In other cases, it is on-premises, installed on the servers of a company or even on my own company's servers.
I would rate this review an eight overall.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)