We use it for logging and troubleshooting.

How has it helped my organization?

Every team immediately created their own Splunk dashboard, and all the product owners were ecstatic about this. We were able to create a catalog of dashboards and have a holistic view at all levels. We could understand our business much better. Real-time errors, which were buried in emails before now, surfaced up on dashboards. Even our executives could understand this, and it changed the way teams thought about alerting and reporting. It allowed us to send out real-time notifications to integrate with Opsgenie, and it changed the way IT works.

What is most valuable?

The dashboards are the most valuable feature. We like the ability to drill in and see what queries are under the dashboard, build new visualizations, edit the querying, and see the reports. The dashboards are very intuitive and similar to SQL. They are easy to set up and get running.

What needs improvement?

The query language is pretty slick and easy, but it is not consistent in parts. Some of it feels a little esoteric. Personally, some of my engineers are coming from SQL or other languages. Some things are a little bit surprising in Splunk and a little bit inconsistent in their querying, but once you get use to it and once you get use to the field names and function names, you can get the hang of it. However, if it was a bit more standardized, it might be quicker to get it up and running.

I would like additional features in different programming models with the support for writing queries in SQL or other languages, such as C#, Java, or some other type of query definitions. I would also like a better UI tool for enhancements of advanced visual query editors.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

It is pretty stable, though it has gone down from our usage. We do need to keep an eye on our query volumes. Right now, it is too easy for a user to write a query, run it, make it available in polling mode (real-time mode), and bring down the server. Some more safety alerting would help and be beneficial.

We do have to educate developers on how to not blow it up. It is a little to easy to write an expensive query and overly stress the system. This could be improved. Overall, once you have people who know what they are doing, it is very stable.

What do I think about the scalability of the solution?

Our environment is on-premise, and it is big. We have a couple hundred users. However, it was slow and unavailable at times before we trained all the engineers on how not write a long, constantly polling query.

How is customer service and technical support?

Our internal tools team did work with the Splunk support team extensively. I was not directly involved, but from my point of view, they were able to fix and resolve issues within a day or less, so they have been okay

How was the initial setup?

It is early days right now to evaluate the integration and configuration of Splunk in our AWS environment. We are just starting to integrate it with regular stuff. While I think it is okay so far, I really do not have enough information.

What was our ROI?

Most of our return on investments have been through faster error resolutions. Our meantime to recovery has dropped for issues. We can often fix things before the customer notices them. Whereas, when logging was done custom by each team in non-standard ways, it would take days to resolve issues that are now resolved in sometimes minutes.

Which other solutions did I evaluate?

We knew we were going to go with Splunk. It was the leader and the one we liked. We didn't consider any others since Splunk met our needs.

We chose Splunk because of the ease of the UI, querying, and creating dashboards. It has a standardized query language, which a lot of the IT staff were already familiar with it. It was the market leader from our prospective for our needs.

What other advice do I have?

Go with Splunk. A lot of people know how to use it because they have experience with it. It works well. While it has some pain points, it provides reports and data visibility.

It integrates great with Opsgenie, PagerDuty and Slack. We love the Slack integration, as works great with the Slack alerts.