We use it primarily for the commercial support, and there are some plugins which make integration and scripting a little easier on the commercial side. Our primary use case is being able to call somebody and know a lot about Jenkins. However, we don't pretend to be Jenkins experts, since we are not Jenkins coders. We can just call somebody and have assistance instantly, which is the reason we chose to use CloudBees. The support is there when you need it.

We have not been using it that long. We have only been using it for a little more than six months. It was a migration from Jenkins to CloudBees, which was seamless.

How has it helped my organization?

Prior to using CloudBees, if we couldn't figure something out, we would try to figure it out. We would waste a couple hours, or weeks, on some particularly hairy issues. It slowed us down. We had more people focusing on our build management systems than we had focusing on using it, in some cases.

CloudBees drastically improved our workflows. If something didn't work, we could have it working again. We weren't spending time troubleshooting and Googling for stuff. E.g., "We know this guy had a similar issue. We will contact him and see what he did about it."

When we hit a barrier, we hit a barrier. We had to almost guess and feel our way in the dark at how we could get around it. Now, we still hit barriers. While there are some cases where we submit an issue, and they say, "Give us a couple days to look at the particularly issue, because it is a hairy issue." It always comes back with an answer. Importantly, it's access to an entire company of knowledgeable people.

While we can't abuse these relationships, we can call them and know that they aree there. We pay for it. and they are okay with us calling them. It's a different thing than, "Well, I know Bob, so I will call Bob." I can't call Bob every week.

The benefit is support and the speed that it provides for troubleshooting.

What is most valuable?

All its technical features are valuable, but we could do 90 percent of those with Jenkins. The difference with CloudBees is when one of those features doesn't work, we can't figure it out, or it doesn't work the way we expect to there is somebody that we can contact who can make it work.

What needs improvement?

CloudBees has a lot of features. It has a core product with some special build stuff and purchase, which is great, but where they can improve is around the messaging. We use it for support, but a lot of our clients could possibly use some of the other features around it.

As an example, the Kubernetes skew stuff. A lot of people are just using the plugins for Jenkins, but it's not clear to a lot of our clients why they need CloudBees, because it's a hidden away thing. E.g., you are using the internet, but you don't usually think about what it takes to keep it on. In a lot of cases, the people making the purchasing decisions don't understand why Jenkins isn't enough. "Why do we need that? My team knows Jenkins." Great, your team knows Jenkins, but they don't know all of Jenkins. They know the pieces they use, which is fine. They shouldn't know all of Jenkins, because that's not what they're meant to be doing. They're meant to be using it for coding, etc.

I still find the messaging really confusing. It's not immediately clear what the value-add is yet. It's not clear enough to pitch it in an easy, prepackaged way. If you already know their ecosystem, it's easy to figure out. However, if you don't, you have to learn their ecosystem first. The guys who are making the purchasing decisions are usually not the ones who will take the time to learn the ecosystem. If we had to justify the solution less, then we could just slide them some information which would be simple to consume. This would make our lives a lot easier.

For how long have I used the solution?

Less than one year.

What do I think about the stability of the solution?

I have not noticed any stability problems. No crashes come to mind, but we have not been using it for an extremely long time. We may eventually see issues, but I don't think so. We have found them to be stable, and previously, we found Jenkins to be stable. I imagine things will continue to be stable.

What do I think about the scalability of the solution?

In terms of the scalability aspect, some of our customers do a release every two weeks. They roll up on the small side of approximately 30 different containers into a release and deploy to a couple environments, then rollout a week later. Other clients are doing 50 releases a day, because every time that they modify code, it builds something new. They are deploying into an environment per developer, an integration environment, and end-to-end.

This is partly a test of the scaling's usability. However, we haven't seen an issue. Our biggest environment has about 50 active users with about a couple hundred different services being built. We've occasionally had to resize a VM and give it more resources, etc., but we have not noticed any blockers where we can't move faster. Sometimes, we have to give it more horsepower, but beyond that, I can't think of a case where we've hit a wall.

I think we have hit the upper maximum on what we will ever need in terms of scaling. Their license model makes it easy to break out from that. If we need to break up projects into multiple build systems, that's fine too. I don't see any hidden scalability issues.

How is customer service and technical support?

A lot of our team who speaks with their technical support are heavily technical. Traditional support is used to dealing with the managers or an IT department. Development doesn't usually call support, because they tent to want to fix it and discover it themselves. I have noticed a lot of our developers are comfortable calling CloudBees, or speaking to them in general, whether it be on Slack, email, or calls. They tend to work alongside you, so it feels like a colleague helping you with your issue, more than it does just some random person. At heart, they are a group of developers too.

Everyone who has spoken with them has said that they interfaced well with our development team. The main people who are calling them on a case-to-case basis have found this really nice. We have great interaction with them: It's not an us versus them. When we call them, they are temporarily part of our team. They come in, and they help us, then they go away after. However, they are right there when we need them. This is a completely different model than a lot of other software solutions.

Which solutions did we use previously?

Because it is Jenkins, ultimately in the core, all of our Jenkins plugins with it worked. That was a big boon. Usually, when you switch to a product, you have re-evaluate everything. We didn't. We just moved our plugins over. Everything just worked, and where it didn't, they helped us fix it.

How was the initial setup?

The initial configuration and integration was seamless. We migrated from Jenkins. As we had originally done our built in Jenkins, except for the fact that it was a slight version upgrade, it was seamless to move over to CloudBees. All of our integrations continued to work.

What's my experience with pricing, setup cost, and licensing?

Their support model is a recurring model. It is not a one-off fee. We pay on a yearly basis. We are already gearing up to justify our user license count for next year. When it comes time for renewal, we will have to tell all of our projects and customers why that cost is there. While we will be able to justify it, I feel like if they could add more information that it would make it easier for us.

I can't tell you if we purchased it direct or from the AWS Marketplace.

Which other solutions did I evaluate?

We had already done an evaluation of Jenkins versus other products, such as CircleCI and a bunch of the other ones, and we had already settled on Jenkins. When we moved to CloudBees, at that point, it was a question of open source, which we're fine with, versus commercial support. We needed the commercial support and had already settled on Jenkins. We were comfortable with Jenkins features and just needed the support.

What other advice do I have?

We use it for everything from from building a container to deploying a container, and in some cases, for kicking off environment creation. It integrates well with other products in our AWS environment. The core concept is fully scriptable. We do use it to integrate, and it is infinitely integratable with everything in AWS.

It depends on what product that you want on CloudBees, but if you want the support or core stuff, it's absolutely a no-brainer to just go to CloudBees. You don't have to change much at all.

It's like when you have a cell phone, and you buy the same brand on a newer version. You do the WiFi sync, then all your pictures, contacts, and everything else comes over automatically. This is the same thing. If you're on Jenkins already, and you're happy with it and not looking to move products, just needing support and additional features, CloudBees is a no-brainer. I wouldn't even evaluate other products. I would just look at them.

If you're in the mood to just look at products similar to this, I would suggest installing a few of the big commercial ones. If you're working at CloudBees, you're obviously into the commercial side of it. This means that you will be looking at things, such as CloudBees, CircleCI, and a couple of the other big ones, depending on what features you need.

The unfortunate things with build systems and a lot of code bases is you can't fully implement them as a test, because of the amount of hours, especially if you're switching (though not in all cases). If you're switching it can be a huge time sync to implement to evaluate. If you pick a simple project, there are a bunch of GitHub projects which are meant to sample applications for learning build systems. Install one of those and use all the systems on that one and get a feel for what they offer you.