Saves test execution files for easy access and provides a execution model for running JMeter or YAML scripts across different infrastructure configurations
What is our primary use case?
I used it for a couple of projects, but I don't actively use it now.
We use it for performance testing, volume testing, stress testing, and endurance testing.
How has it helped my organization?
We use it for EPS, web HTTP HTML, SQM, RabbitMQ, and sometimes ActiveMQ. So, it handles various testing scenarios for me.
It has been most effective in managing large-scale tests. It saves the test execution files to the repository. The tool also has a distributing/executing model. You can create a JMeter script or YAML and then execute it using different infrastructure-related configurations.
Moreover, it can be integrated with tools like ALM.
What is most valuable?
Running from the cloud with load distribution, it exhibits load from different geo-regions. The best feature is generating loads from different cloud regions.
I find these features useful for my particular use case because I can't execute or generate the load within my infrastructure. With the cloud, I can rent on a pay-per-use model and execute the load with a massive number of users.
What needs improvement?
Sometimes, when we execute tests, the results calculated by BlazeMeter, specifically the response times for failed transactions, are incorrect. We've already reported this issue. If this could be fixed, BlazeMeter would be a much better tool compared to LoadRunner.
Currently, it incorrectly calculates response times for failed transactions, it provides data that isn't useful. We have to manually aggregate the data to get accurate values.
In future releases, I'd like to see BlazeMeter integrate with mobile applications and allow testing on real devices.
By testing on real devices, we could gather metrics related to CPU usage, memory, and battery consumption. This would give us a better understanding of how the application performs on actual devices and help us ensure there are no battery drain issues, high internet usage, or excessive CPU or memory usage. This would allow us to confidently certify that the application is optimized for real-world device performance.
For how long have I used the solution?
I work as a centre of excellence, so I've been working with JMeter and BlazeMeter for almost nine to ten years.
For only BlazeMeter, it would be five to six years. BlazeMeter doesn't have versions, but JMeter is currently on version 5.6.3.
What do I think about the stability of the solution?
I would rate the stability a ten out of ten. There haven't been any outages, so I'm satisfied with the stability.
What do I think about the scalability of the solution?
I would rate the scalability a nine out of ten. It scales well. When the load generators get overloaded, it automatically distributes the load to new instances.
We don't very actively use it right now, but we have used it in the past two years. If we have the opportunity and the client is looking for a cost-effective tool, we would definitely choose BlazeMeter.
How are customer service and support?
My experience with the customer service and support have been very good.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We used LoadRunner. The cost was the main reason for the switch.
How was the initial setup?
I would rate my experience with the initial setup a nine out of ten, with ten being easy. It was straightforward, I didn't had any issues.
- Deployment Model: It's a SaaS model, so it's already available for use. We only need to create and upload our scripts.
- Integration with existing CI pipelines: Once a new build is ready, we have automated pipelines that trigger a load test on the deployed build. It then provides a result indicating whether it's a go or no-go based on the configured SLAs.
What about the implementation team?
Our organization maintains BlazeMeter, not me. I'm a performance architect who uses BlazeMeter to assess application performance.
What was our ROI?
The cost is low, so there's a definite return on investment compared to LoadRunner.
There is no direct ROI because we still pay for the product. But think of it this way: if I spend $100 on LoadRunner but only $10 on BlazeMeter, then the ROI is essentially the $90 saved. That's how we look at it.
What's my experience with pricing, setup cost, and licensing?
I would rate the pricing a three out of ten, where one is very cheap, and ten is very expensive.
Which other solutions did I evaluate?
We looked at NeoLoad. Here, also the cost was the primary factor.
What other advice do I have?
Overall, I would rate it a nine out of ten. For me, it's a good product.
It's a good tool for automation testing and performance testing, especially if you're looking for a high-performing, highly scalable, and cost-effective solution.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)