4 Dos and Don’ts When Using the Cloud to Experiment

“The best way to show that a stick is crooked is not to argue about it or to spend time denouncing it, but to lay a straight stick alongside it” ― D.L. Moody

In my last post, I described how the cloud is making experimentation easier, cheaper, and less risky for companies of all shapes and sizes. The more that companies realize this, the more having a culture of experimentation becomes table stakes to stay competitive in today’s market. Experimentation breeds innovation, and there’s never been a better time to execute on new ideas.

So where do you start? Here are four things to consider and four things to avoid when building a culture of experimentation in your organization.

1. DO manage expectations

Not every experiment will deliver the results you envision, but every experiment is an opportunity to learn and improve your operations. If your organization is not used to the concept of “failing to learn,” start small and make sure everyone knows which projects you consider experiments. Manage your stakeholder’s expectations by being clear about the purpose of the experiment, what you hope the outcome will be, how you’re going to measure and test the result, and what you hope to learn from it. I’ve found that most executives appreciate experimentation with uncertain outcomes if they know how the organization will be better off for having tried it and learned something.

DON’T start with a project where everyone is set on a specific outcome

If you’re acting as a change agent trying to create a culture of experimentation, don’t experiment too early in your Journey with a project where your stakeholders demand a specific outcome. I wouldn’t advise you start experimenting with your end-of-year billing run, for instance. A CEO I once worked for told me that it’s ok to fail, except when it isn’t. Be satisfied with incremental progress and slowly increase the number of experiments you run, but don’t outpace the organization.

2. DO encourage your teams to propose experiments

Every organization has its own way of determining which projects get technology resources. Unfortunately, some organizations now treat the technology or IT department as a cost center and have pushed ideation too far away from those implementing it. Good ideas can come from anywhere, of course, and most technology professionals have a unique perspective that will surface when it comes to outside projects. This is especially true in organizations that are just getting started with the cloud — the individuals using the cloud for their projects are in the best position to propose experiments that leverage capabilities that are unique to the cloud to benefit the business. Help champion your team’s proposals and position your staff to have an influence on which projects get invested in with the executive team.

DON’T pursue an experiment until you know how to measure it

You want to spend time on the right experiments and ensure the lessons learned from them will improve your operations and your products. Before you let your team to move forward with an experiment, you should agree on what they will measure during the experiment, and how. If you’re testing a new feature on your website, what are the metrics that will make it successful? Page views? Number of clicks? Abandonment? This small-but-important act of due diligence will force your teams to think through why they’re proposing an experiment in the first place. It will also force your team to prioritize the right experiments.

3. DO consider DevOps to institutionalize experimentation

A DevOps culture can be a powerful way to codify experimentation into your organization. Marrying run-what-you-build with automation can drastically reduce the time it takes to release changes, allowing you to release more often and quickly rollback changes that don’t work. Mature DevOps organizations also develop A/B testing frameworks that allow them to experiment on slightly different user experiences with different user cohorts in parallel to see what works best.

DON’T doubt your team

Doubt is one of the most powerful ways to discourage your team and open the door for failure. As you learn to properly scope experiments, measure them, and iterate on them quickly, you should find that you’re able to adapt your approach before you need to doubt it. Making sure your teams are thinking about the right way to measure experiments and asking hard questions is healthy, but you should be looking to help the team work through issues rather than doubt their ability to deliver. People tend to follow leaders that believe they will succeed.

4. DO encourage the whole organization to participate

As you start to deliver results faster through experimentation, other areas of the organization will become attracted to your methods. Involve those people. Try a hackathon that includes different business areas, have your stakeholders help define how you’ll measure experiments, and ask the organization for areas they’ve always wanted to experiment with. While not every company chooses to give its employees time to experiment, those that do typically tout it as a competitive advantage. At the very least, these kinds of inclusive activities will improve employee morale and retention. In my brief tenure so far at Amazon, I’ve found that anyone able to think through and articulate an experiment in writing typically gets the opportunity to try it. This is a special part of our culture and a great tool for attracting and retaining innovators and builders.

DON’T let experiments slow or halt delivery

Don’t let your teams off the hook to deliver just because something is an experiment. While it’s ok to fail and learn, it’s not ok to underdeliver on the experiment. Software still needs to ship to be tested; often times it needs to be measured with real traffic in production. Just because it’s an experiment doesn’t mean you should start measuring it late or let its quality suffer. You’re still running a business, after all.

What is your organization to enable a culture of experimentation? I’d love to hear about it.

Keep building,
–Stephen
@stephenorban
orbans@amazon.com

Note: Creating a culture of experimentation is the third of seven best practices I’m writing about in my new Enterprise Cloud Journey series. The remaining six are: provide executive support, educate staff, engage partners, create a center of excellence, implement a hybrid architecture, and implement a cloud-first policy. Stay tuned for more on each of these.