It is mostly centralized logging, a whole bunch of BI metrics, and an aggregation point, which we have adulterated for some PCI data.

It does meet our use case for the most part.

What is most valuable?

We like the dashboard creation and the ease with which we can harness the APIs to create custom BI dashboards on the fly. This adds most value for us. The nature of some of our microservices that I have run on the cloud are mixed workloads, wherein with the flow of data, it can change over time. In order to adjust for this, and cater to the needs of some of our internal customers, BI dashboards need to be created, tweaked, and modified. Also, doing this by hand is next to impossible. Therefore, we have strung all of this through a programmatic pipeline, which s something which we like because it is easier for us to harness it utilizing the API.

What needs improvement?

For on-premise, it's more about optimization. With such a heavy byte scale of data that we are operating on, the search for disparate data sometimes takes about a minute. This is understandable considering the amount of data that we are pumping into it. The only optimization that I recommend is better sharding, when it comes to Splunk, so that data retrieval can be faster.

With the AWS hosted version, we have not hit this bottleneck yet, simply because we are not yet at the multiple terabyte scale. We have hit with the on-premise enterprise version. This is a problem that we run into every so often. We don't run into this problem day in and day out. Only during the month of August through October do we contend with this issue. Also, there is a fair bit of lag. We have our ways to work around it. Between those few months, we are pumping in a lot of data. It is between 8 to 10 terabytes of data easily, so it is at a massive scale. There are also limitations from the hardware perspective, which is why it is an optimizing problem.

For how long have I used the solution?

More than five years.

What do I think about the stability of the solution?

On the cloud, we are pushing through less than half a petabyte of data. So far, it has been fairly stable because it runs on all the underlying AWS infrastructures. Therefore, we have had no issues at all. In terms of availability or outages that we've experienced, there haven't been any. We've been fairly happy with the overall landscape of how it works on AWS.

What do I think about the scalability of the solution?

On cloud, we absolutely like it. Splunk AMIs make it easy for us to spin up a Splunk cluster or add a new node to it. For our rapid development and scale of deployments in terms of microservices and the number of microservices that we run, we have had no problems here.

On-premise requires a lot of planning, which happens on a yearly basis. We have Splunk dedicated staff onsite for on-premise to help us through this.

We have 450 people making use of Splunk in our organization, and there was a bit of knowledge transfer needed on how to write a Splunk query. So, there is a bit of a learning curve. Once you get over it, it is fairly simple to use. We also have ready-made Splunk queries to help people get started.

How is customer service and technical support?

We do deal with technical support on an ongoing basis. They can definitely do better from a technical point of view. Their only purpose working onsite is to make sure that our massive set of Splunk clusters are online, and the clusters are tuned well enough to work well.

We would expect the technical support people onsite to be subject-matter experts of Splunk. We have seen in a few areas where we have been left wanting more, wherein some of our engineers happen to know more than them in terms of some of the query optimizations, etc. This is where we think there is a fair amount of improvement that can be done.

What about the implementation team?

We wrote the automation to bootstrap everything onto AWS, which was fairly easy. As long as we had all the hooks going into AWS, and we had the SDK. So, we did not have too much trouble getting the bootstrap up and running.

What was our ROI?

Some of the insights that we have obtained as a part of using Splunk have greatly helped us in increasing our revenue in terms of selling our products.

We have seen a decent ROI. For the month of October 2018, when we had a product launch, we were able to query and generate BI dashboards on the fly. This was huge, and not possible two and a half to three years back because it was more of a manual process. Now, with APIs being available, it is very simple to tweak or write a small piece of glue code to go ahead and create a new dashboard for a business unit to make near real-time decisions to focus more on other geographies when launching the product.

Which other solutions did I evaluate?

I wasn't there when the evaluation was done. When I came on board, this product was handed down to me, and we have not evaluated any other solutions or products since then.

What other advice do I have?

Make sure it fits your use case. Be clear about what you want to achieve, get out of the product, and how you want to integrate it. Once you tie the solution into your systems, it is not trivial or easy to walk away from. Therefore, due diligence needs to be made to understand what your requirements are before choosing a product. Some companies may not even want to host, and prefer to go the managed services route.

We have it integrated with every product that I can think of.