Monte Carlo Data + AI Observability Platform

My main use case for Monte Carlo is data monitoring, monitoring jobs that have failed, and I have also used testing fairly extensively. I usually use the pre-boxed testing that Monte Carlo offers, but on a couple of other use cases I have developed custom testing as well.

I cannot remember the exact details of how I use custom tests in Monte Carlo because the last time I did it was probably several years ago. I think that it was generally testing that was not more generic, such as uniqueness testing or volume monitoring, but I honestly do not know if I can remember or give the exact details of what I was doing.

As far as doing anything very unique or different that I have tried in Monte Carlo, I would say I am more of a standard user of it. We have it connected to one of our Slack channels that monitors our jobs and any anomalies that we have within our data products. I primarily use it to keep an eye on some of my models that I am responsible for.

I have found a lot of value in volume anomalies with Monte Carlo. I do not remember what it is called exactly, but I have really appreciated that feature because it helps identify potential issues that might be hard to identify otherwise, such as if the data source has changed without my awareness, even though the schema did not change. Monte Carlo can still detect something that changed upstream and is not as obvious. Things such as that have been very helpful with any kind of volume anomaly detection.

As far as the volume anomaly detection in Monte Carlo, this is something fairly recent. I helped develop brand new models for our Genesis phone call system data. Something had changed upstream where some of the API calls were doing something a little differently. There was no schema change, but the volume increased dramatically overnight, which prompted discussions with the upstream data warehouse team to identify what those changes were so we could accommodate them in the analytics layer.

I think improved reliability with Monte Carlo has been valuable. Our team monitors the channel that feeds us any alerts on our models that we are responsible for. While we have not gone through every single alert yet, it has been very helpful for each of us to keep an eye on the models that we personally are responsible for. The alerting has been great, as well as some of the metrics, such as being able to identify tests that are failing. It has been helpful to identify those immediately rather than having to dig into the data and take the time to do the data discovery to find out what issues there are.

Having used Metaplane and Elementary and worked with those other tools, I think Monte Carlo had a gap or something that was not as strong, specifically around custom feature development. I mentioned that I could not give specific examples, and I do not think I could do that now, but Metaplane specifically had a strong area with that. Additionally, the UI in Monte Carlo can be a little overwhelming sometimes. That is another thing to add to my comments, but overall, I think it is fairly easy to use.

I think that all those areas in Monte Carlo, the specific examples mentioned such as documentation, have been fine, as well as the support as far as I know. I have not had to work specifically with issues regarding support. I have not had to work hard with the product itself as far as getting support, but overall it has been good. I have been able to find help with issues in the past through the developer documentation. For me, the only thing that feels a little overwhelming when using Monte Carlo is the AI, which is something I appreciated about Metaplane because it felt more user-friendly and did not feel so overwhelming.

I have been using Monte Carlo for probably three to four years collectively. I am using it currently at HubSpot and I have used it at two other previous positions.

We have tried different tooling, and I was part of POC work for trying out different tools such as Elementary and other related tools. We found that Monte Carlo ended up being the best for us because it was able to do pretty much everything that the other competing tools were able to do for us, as well as being pretty entrenched already in our data architecture. Another reason I think we ended up staying, which I think says a lot about Monte Carlo in general, is that they were able to work with us on pricing and assure that we were happy customers. As a large organization, we were pretty happy with the pricing and the negotiation that Monte Carlo was willing to do.

I would probably give Monte Carlo an eight out of 10. I think it is very usable and beneficial overall for data monitoring. I really appreciate things such as the time since last update, which is easy to find, row count changes, and even some of the orchestration, such as execution time, which has been really cool. There are some really cool monitors that come out of the box. Monte Carlo has some other cool things such as field lineage and table lineage that have been pretty cool to use. I can use it right within that tool and do not have to go to Atlan to look at that. Setting up personal dashboards has been probably a little harder to do compared to Metaplane or Elementary, but at least the functionality is there. That is kind of the reason I would give Monte Carlo an eight out of 10.

I think Monte Carlo's AI capabilities are fine overall. Monte Carlo seems to be doing what it needs to be doing as far as keeping the data secure in its product, even when using AI. I do not have much to say about that.

I personally have not used the AI tool within Monte Carlo much, so I probably cannot say I have a definitive opinion on that. I believe I might have used it once in the past and it helped me figure out what I needed to know as far as using some of the tooling in it, but I do not remember what I was trying to use at the time.

The first thing I would suggest to somebody brand new to Monte Carlo is to spend the time understanding monitors in general. That is for me the bread and butter of what the tool can do. I would suggest spending time in the developer documentation, understanding what each monitor does, some of the out of the box monitors such as volume testing, row count changes, and similar things. I think utilizing some of the custom fields or custom testing is really powerful if you know what you are doing and know how to implement that in a smart way. While the capability is there, I would suggest people really understand those two things specifically.

I would recommend Monte Carlo overall. I gave it an eight out of 10 because I think it does its job in helping us monitor our data products and making sure that nothing is breaking that we are not aware of, as well as being actively aware of any anomalies that might exist.