New York Public Library’s Cloud Journey

“A library is the delivery room for the birth of ideas, a place where history comes to life.” -Norman Cousins

I had the pleasure meeting Jay Haque a few years ago when he started leading the New York Public Library’s (NYPL) cloud Journey. I was on a similar Journey at Dow Jones, and we had the opportunity to share our stories. A few weeks ago, I was delighted to see Jay comment on one of my posts, and after hearing a bit more about how his Journey at NYPL had progressed, I thought his perspective on best practice would benefit the broader market. It turns out that I wasn’t alone, and today Jay and his team were named the winners of the 2016 AWS City on a Cloud Innovation Challenge. Thank you Jay and I’m looking forward to seeing you and the NYPL continue to transform the way you deliver technology for your customers!

(Stories like this one that are based on real experience tend to be the most valuable, and I’d love to host more of these kinds of posts. If you have a story you’d like to share, let’s talk about posting on my blog!)

–Stephen
@stephenorban
orbans@amazon.com

***

When Stephen asked me to write about my experience with incubating a cloud center of excellence, I revisited his “7 Best Practices for Your Enterprise’s Journey to the Cloud” post and found that my company’s experience aligns with these practices.

The New York Public Library’s Journey speaks directly to the concept of scale. A large team or a big project is not necessary to start the enterprise cloud Journey, you can start with a small, focused effort, geared at establishing your practice and poising it to scale. Significant top-down support with overwhelmingly strong vision and “air cover” along with financial backing are any technologists’ dream come true, but we know that doesn’t always happen. In absence of this, start somewhere, no matter how small.

Our Journey started with the simple idea of building a configuration management platform in the cloud to prove the redesign of our primary website and the release of our famed Digital Collections site would benefit tremendously from it. We realized quickly that the cloud, and specifically AWS, accelerated project completion times through increased automation, reduced costs with right-sizing, provided redundancies, and enabled seamless scaling capabilities that would otherwise be cost prohibitive. Our Journey had humble beginnings and the resulting success has been extraordinary; the use of AWS by our world-class development, product, project and DevOps teams won us the honor of being named the winner of the 2016 AWS City on a Cloud Innovation Challenge in the Best Practice category.

Working with Control Group, an AWS partner, we devised a plan that not only delivered the necessary infrastructure for a web property but also a configuration management platform, which would accelerate future projects. The core idea here is that, while the web property will one day be retired, the management platform will scale as needed and will automate the infrastructure builds for a limitless number of sites. The project would be completed in no more than 12 weeks with less than 50 percent allocation of the team time.

To give the clearest sense of how our project aligned with Stephen’s seven best practices, I present our experience in the same format. (If you haven’t read the post, I highly suggest you do by going here.)

1. Provide Executive Support

Top-down executive support gives your initiative purpose and weight. In terms of scale, start with what you can get, even if it’s only one executive or one project. Stay focused on building further executive support as you rack up “wins.” This will help you garner support organically.

When we first discussed the Journey, the Library had two major SaaS projects underway, a newly appointed CTO, limited human resources, and faced potential budget cuts. As you might imagine, the appetite to start a large-scale, focused cloud initiative was low.

Still, we knew we had to start somewhere or risk falling behind industry trends. Our CTO backed building a configuration management system in the cloud as a low-risk, low-resource utilization project that would be completed quickly. Once that project illustrated the value of the cloud, interest from the CTO and other executives grew considerably and would be the foundation of support for the rest of our Journey.

To obtain the necessary support of the project, I focused on the following five factors:

1. Leverage an existing project. We positioned the configuration management platform as a means of accelerating future, high-priority projects. By building the proposed platform, we would increase execution velocity on future projects requiring infrastructure.
2. Start small. This enabled the remaining three factors to play in our favor:
3. Maximize the probability of success and minimize risk. The project was small enough that failure was unlikely. In the event that it did fail, the cost and business impact would have been low.
4. Execute fast. Small projects that drag on can be a nuisance. The spotlight was on showing immediate results and moving progressively towards larger projects.
5. Talk scalability. Though the project was small, we highlighted how various components of it would scale for future successes.

2. Educate Staff

Technologist are always learning — it’s the very nature of what we do. In terms of scale, be mindful of the capacity, or available time, to learn. Then, motivate your team and provide any resources you can. Focus on learning the core elements that are going to provide the greatest immediate return. For us, orchestration, configuration management, and code became the centerpiece of everything, so we focused on learning these things first.

At the start of the configuration management platform project, the team was well aware of our ambitious goals. The weeks leading up to kicking off our first AWS project were filled with discussions about the service: how and where we might leverage it, partner presentations, and learning about what other organization were doing. This built a sense of excitement and strong willingness to learn. As the project began, sufficient “air cover” was garnered by the allocation of resource time through our PMO practice and CTO level support. Finally, in absence of AWS offered courses, we partook in partner provided workshops.

3. Create a Culture of Experimentation

In leveraging a partner to help kickstart our Journey, we focused on fostering a culture of experimentation.

The partner presented technical ideas well in advance of implementation, giving the systems engineering team sufficient time to explore and test-drive the technology in our AWS sandbox environment. The sandbox provided a flexible environment that allowed the systems engineers to quickly build and tear down stacks and experiment with various solutions to complex problems. We explored instantiation of our web infrastructure using various methods including using AMIs, complete orchestration using Puppet, and syncing code from external repositories or S3. This level of experimentation, along with deep-dive discussion with our partner, provided a forum to meld our ideas with emerging best practices, ensuring we effectively weighed competing technologies.

The culture of experimentation seeded in those early days of our Journey is still very much in place now. Engineers and developers with varying specialties are encouraged to experiment with all aspects of the cloud. We architected and incubated a media-transcoding pipeline using Elastic Transcoder during a one-day internal hackathon. We could not have realized this level of autonomy in experimentation without AWS, which provides a means for organizations to experiment without a significant financial investment.

4. Engage Partners

We could not have made ground as quickly as we did without Control Group. In terms of scale, staff augmentation infused our small systems engineering team with proven AWS expertise. Control Group offers customers established best practices for automating system builds and deployments in the cloud that they also use within their own software development operation. Their practical experience with the AWS, both as a service provider and a user, helped accelerate our Journey by drastically reducing our learning curve.

Partner involvement in the early stages of our Journey supported efforts to educate our team, internal experimentation, and the application of best practices in the work we did. This close relationship enabled continued success.

5. Create a Center of Excellence

As our initial AWS project was nearing completion, we focused on the “what’s next” element. In terms of scale, we focused on going BIG. This would require substantial support.

The development and systems engineering teams discussed how to build future projects on AWS. This was the beginning of understanding how roles would change, how we would work together, and realizing how much this way of working would impact overall velocity. We maintained partner support as we moved our primary website into AWS and followed by independently working on the release of the highly anticipated Digital Collections site on the platform. The Digital Collections site was large and complex enough to truly test our ability to operate in the cloud. Our systems engineering and development teams had amassed sufficient expertise in the months leading up to the release and having Control Group at arms reach gave us full confidence in our ability to architect the right solution for the high-demand web site.

We’re still very much working on our CCoE, and even on the concept of DevOps. We’re learning what the competing forces are and asking hard questions about how to balance them. For example, we want velocity, but don’t want to sacrifice integrity of our systems. We want standardization, but not too much overhead. These questions are all asked in terms of optimization: we’ve gone from months to deliver infrastructure to just days, and yet we’re continually asking “how can we do this better and faster?”

6. Implement a Hybrid Architecture

This was an absolute necessity for the New York Public Library, as we have a significant on-premises infrastructure that we must continue to operate. Our primary focus is on the building new products in AWS and moving those sites that we can (easily) onto the platform. The organization’s appetite to move legacy content to the cloud is growing; however, the resources necessary to execute a move are at times better utilized towards moving new products forward. In terms of scale, we must pick our migration targets wisely, and consider all our options with legacy systems. Migration is one; deprecation is another. In the meantime, we actively support a hybrid architecture.

7. Implement a Cloud-First Policy

We are partially here. We are very good at building web-centric systems on AWS, and these are almost always built in the cloud. In fact, there must be compelling technical reason for us to deploy a new web solution on-premises. We’ve still got some learning to do on how to deploy other types of business applications in the cloud, and even these cases, we tend to ask if a SaaS product is a viable alternative. The road to a cloud-first policy is paved with wins from the Journey; the cost and efficiency benefits speak for themselves and enable the realization that cloud-first is the optimal strategy.

Summary

NYPL was just starting its cloud Journey when Stephen and I first met to share our stories. We’ve come a long way since then and the model for success has become clearer in that time. Stephen’s masterful articulation of this in the seven best practices is beneficial to any organization, at any stage, of its Journey. I hope this post was helpful to those considering how and where to start their Journey. I would love to hear about your own experience as it will add to the collective library of success stories, and every story is a learning opportunity.

Jay Haque
Director of DevOps & Enterprise Computing | New York Public Library | Follow NYPL: NYPL Archives
jayhaque@nypl.org | Follow me: @jayhaque