Patterns and Anti-Patterns at USCIS

In previous blog posts I have described anti-patterns of enterprise IT that we often observe in enterprises we work with, and patterns of successful enterprise IT that are their antidotes. To make these patterns and anti-patterns more concrete, I will explain how they have played out at my old organization, US Citizenship and Immigration Services (USCIS), where I was the CIO. USCIS just happens to be the organization I am most familiar with—I think that if you examine almost any large enterprise you will find similar challenges—and, if they have succeeded in digital transformation, similar remedies. You can refer back to the original post for full explanations of these anti-patterns and patterns.

Part One: The Anti-Patterns at USCIS

Big FANGS: In 2006, USCIS began its Digital Transformation project. The purpose of the project was to make all of immigration processing electronic, with the goal of speeding up processing, becoming more efficient, improving national security, reducing costs, improving service to applicants and partner organizations…and just about everything else that could be considered good. Until that point, all of the processing had been paper based, beginning with the 90+ forms that applicants used to apply for our various benefits (green cards, naturalization, and much more). The lifecycle cost estimate for the project was $3.6 billion—though please don’t take the number too literally, since a government LCCE is something of a fiction (I’ll explain in a later post). A better measure of cost was that after the first $1 billion had been spent, nothing usable had been created. This was only one of many huge government IT projects.

Risky and Sticky: The DHS investment oversight process and the Software Development Lifecycle (SDLC), which we were obliged to follow, required writing and approving on the order of 100 documents and involved 13 gate reviews. Most of the documents were long and required signatures from a number of senior executives. Each stage’s gate review was conducted by about 20 senior leaders. Once the plan was approved, the project team was expected to follow it exactly, and an independent group of auditors—the Independent Verification and Validation contractor—was to verify that the requirements had been completed as specified.

Hubris HIPPO: We collected several one-inch binders of requirements for the first of five releases primarily by talking to the managers of the different functions and lines of business. Then we got all of the senior leaders together for a week(!) to review all the requirements and sign off on them as final. It later turned out that many of the assumptions about user behavior were flawed (or at least wishful thinking) but that was very difficult to discern from reading the technical specifications.

Railroad Baron: Because IT created the legacy systems that the project was planning to replace, the agency decided to run this project outside of IT. IT was to provide “enabling services” to assist the project. A contractor was engaged to be the “solution architect”—that is, the system integrator who would design and build the new IT system, in part by subcontracting to what turned out to be 54 other companies. The IT department was to “enable” the solution architect contractor by providing services that would be “billed” to the project through a transfer pricing mechanism.

Bloated Bear: The project was planned to take five years. Product owners were drawn from the business to oversee each different set of work. Because this was their big opportunity to modernize all of their IT capabilities, the product owners included all the features that their part of the business had wanted for years. Product owners had a strong incentive to not declare work as “complete” until it accomplished everything that their co-workers thought was important. As a result, the work continued to expand, even though it was constrained by the original set of requirements.

Fashion Straightjacket: The architecture of the solution was based on integrating 29 COTS products, which turned out to not integrate very well. At the core of the design was Siebel—a general-purpose case management system—because the agency had already purchased licenses and, after all, this system was just a case management system for immigration cases. Most of the time in the first release was spent trying to integrate Siebel, Documentum, and the Websphere portal server.

Key Under the Doormat: Because the initial release did not include functionality for many edge cases, it could actually only process a very small portion of the agency’s workload—virtually everything in immigration turns out to be an edge case. The network and data center proved to be unreliable, and the failover network switches turned out to not actually failover when there was a failure. Ordering new hardware when it was necessary to scale took up to nine months, given the complexities of the government’s procurement process.

Part Two: The Patterns at USCIS

Despite all of these challenges, the Transformation project arguably became a showcase for successful digital transformation (unfortunately, only after a considerable amount of time and spending). The project is now releasing functionality to production about three times a week. It can demonstrate that it is successfully accomplishing the business objectives it set out to accomplish. The system it is building has demonstrated high availability and resilience. It has reduced lead times for processing certain benefit types dramatically. And it has innovated techniques for transforming government IT that are being used at other agencies as well.

Here are the patterns:

Shrink: By introducing DevOps and working in the cloud, the project began deploying small chunks of functionality about three times a week. In place of the (unlimited) set goals (efficiency, national security, customer service, etc.) the agency substituted seven very specific and measurable objectives. As a result, bloat could be reduced—only features relevant to those seven objectives would be included—and made their progress measurable.

Strangle: The project was moved into the IT department to take advantage of existing assets and processes; for example, it no longer had to manage its own incident response process or build its own data warehouse. Teams began building the simplest possible architecture to accomplish each requirement, and then evolving and refactoring it later. A containerized microservice architecture was established.

Confirm: User-centric design experts were now deployed to work with users and test user experience (UX) ideas. Teams began working closely with business users, sometimes deploying out to remote locations, to continue to refine features as they saw how users were using them. Monitoring was set up to validate progress against the seven objectives.

Cohere: The seven objectives were shared goals of the business and IT. One goal, for example, was to reduce the movement of paper from one location to another. The team solved this through a combination of making the most popular lines of business all-electronic and streamlining business processes to reduce the number of times paper was handed off.

Omit: Not every part of the business would be moved off of paper, using an 80/20 rule to eliminate most of the paper movement. Only features that would affect the seven core objectives were allowed into scope. The project prioritized application forms to automate based on the impact they would have on the seven goals, so that it was always possible to stop the initiative as it reached diminishing returns.

Shorten: Oversight processes were both been shortened and made more effective. Instead of the 100 documents, the project now uses a very brief decision-support document that is about four pages long but focuses on the important information. All gate reviews were eliminated, except for a brief ceremony held every quarter. Administrative staff was radically cut—virtually all employees were actually building product. Despite the simplicity of the process, senior decision makers actually acquired considerable control over the project: reviewing it every month, providing feedback and direction, and re-visiting the original investment decision.

Embed: Security, QA, compliance, and performance testing all became embedded in the system development processes and gatekeeping was eliminated. Controls and policies were automated and are applied continuously both as the system is being developed and in production. The project was continuously audited and monitored.

These patterns are being applied across a wide range of IT projects at USCIS. In a notable example, the project to modernize the agency’s E-Verify system was re-organized around a set of five concrete business objectives, all of which were fulfilled within about 2.5 years rather than the four years planned, returning the remainder of its budget. Innovative teams have created a “network in a backpack” for processing at remote refugee camps; national security triggers; tablet-supported interviews; applicant identity management systems; and improved scheduling processes for biometric collection and interviews. A set of rapid-response IT teams was deployed to a number of field locations around the country to quickly execute small projects with fast and decentralized governance processes, using a standardized toolset with embedded controls. The teams have even held hackathons where they mobilized the entire agency community to deliver up to eight new applications in each two-day event.

The patterns are powerful—pay close attention while you are using them. If you blink, you might miss the rapid deliveries of business value!

–Mark
@schwartz_cio
A Seat at the Table: IT Leadership in the Age of Agility
The Art of Business Value
War and Peace and IT: Business Leadership, Technology, and Success in the Digital Age (now available for pre-order!)