5 Steps to Building a Culture of Security

(image www.bluecoat.com)

In an earlier blog post, I discussed the importance of building a culture of security rather than thinking of security as just the job of the CISO’s team. In this post, I’d like to discuss some ideas on how to build such a culture, drawing on my experiences at USCIS.

As CIO, I was the “Authorizing Official” for USCIS, the person who had decide if each system, was secure enough to be launched. I did this by granting an Authority to Operate (ATO) after consulting with my CISO and security team. The government ATO process was deliberately designed to give agencies flexibility: as a business executive, I could make risk-based trade-offs to ensure that security was treated in a way that was practical and that balanced security goals with mission accomplishment.

Unfortunately, this approach conveys the misleading idea that security is opposed to mission accomplishment and that trade-offs must be made. As I explained in the earlier post, I believe that security is an essential aspect of mission (or business) accomplishment and there are rarely as many trade-offs involved as people seem to believe.

Although we took security very seriously at USCIS (after all, we were a component of the Department of Homeland Security), I found that our focus was largely directed toward compliance with the details of the Federal Information Management Security Act (FISMA). Yes, we had talented security professionals and rigorous vetting and approval processes, but most employees didn’t see the connection between their day-to-day work and the security of the nation’s information systems. Security was viewed as a “check the boxes” burden that might derail the launch of a system at the last minute. Code produced by developers was only tested in its late stages by security folks who might then disrupt the flow of work by demanding remediation of what seemed like trivial issues. If the long list of controls mandated by FISMA was in place, then the security folks were expected to get out of everyone’s way and allow systems to make it to production.

With some variations, this type of gatekeeping security is quite common at enterprises that I talk to. And—to be clear—I believe that a focus on compliance and checklists of controls are an absolutely critical part of security. Atul Gawande’s book The Checklist Manifesto: How to Get Things Right shows that checklists have improved outcomes substantially in healthcare and other fields, and is applicable to information security as well. As I said in my previous post, maintaining good security hygiene—analogous to what doctors do when they wash their hands before an examination—is arguably the core factor in maintaining security. The problem is not the checklist approach to ensuring security. The problem is the manual, after-the-fact approach to applying those checklists.

At a high level, the approach to building a culture of security that we implemented at USCIS looked like this:

1. Consistently communicate the connection between security and mission objectives.
2. Set up practices to “build security in,” and fast feedback mechanisms to correct mistakes.
3. Establish norms for security hygiene and set high quality standards.
4. Adopt a zero-known-defect approach.
5. Continuously vet security, both in development and production.

It is important to notice that these five mechanisms do not involve substantial spending or require much time once they are established. They are not really about trade-offs between security and product delivery. I’ll dig in deeper to each of these five mechanisms below.

1. Consistently Communicate the Connection Between Security and Mission Objectives

We all have a responsibility to our customers and other stakeholders to maintain the security of our systems. Customers trust us with their personal data. Shareholders trust us with the stock price of the company. In the case of USCIS, the public trusted us to maintain the integrity of the country’s immigration system. These responsibilities rest with everyone in the organization.

The CFO should be clear that security matters for financial well-being. The CMO and head of sales should be clear that without the company’s ability to manage the integrity of its systems they cannot deliver on their implicit or explicit promises to customers. The COO should understand that operations includes operating the company in a reliable way, with integrity and consistency.

Security is not a burden on anyone. It is a requirement for doing what they do.

Everyone in the company should be answering these questions to themselves: “Is it OK for us to let our systems be compromised? For our applicants’ data to be stolen? For a denial of service attack to make us stop providing services?” If you don’t believe security is important, then you are misunderstanding your job.

Instead of just meeting with my security folks to decide whether to grant an ATO, I insisted on a meeting with all of the key stakeholders, including business sponsors, product owners, and development teams. I asked questions to determine the security posture of the system and made sure everyone could see and understand any issues that were exposed. I forced people to commit to actions they said they would take to improve security. And I supported the security team in their findings and made it clear to everyone that security was non-negotiable.

2. Set Up Practices to “Build Security In” and Fast Feedback Mechanisms to Correct Mistakes

Our periodic audits revealed the bad news: employees will be fooled by social engineering attacks. No matter how much we trained them, employees would give up their passwords when asked by “a technician from the helpdesk.” A dedicated adversary could design spearphishing attacks that would fool people into clicking on links. People will write down their passwords, because if we make passwords strong enough to be effective and insist that they be different for every system, then they are too hard to remember.

That’s why we moved entirely to multi-factor authorization. It is not a panacea, but it does reduce the reliance on passwords and makes life much more difficult for bad guys. It makes it easy, in other words, for people to do the right thing. We introduced automated security tests into our development pipeline that gave developers immediate feedback if they had introduced a common security vulnerability—and would teach them what the vulnerability was and how to avoid it. We created reusable code that implemented good security practices (identity and credential management, auditing and logging, and so on) and could be included easily in new systems. We installed encryption software on everyone’s laptops.

When our penetration testers found a vulnerability, we gathered everyone together and had the testers present their findings so everyone could understand how the pen tester had fooled them and avoid a similar incident in the future.

3. Establish Norms for Security Hygiene and Set High Quality Standards

You wash your hands when leaving the restroom. If you are a developer, you validate your input. Washing your hands makes it less likely you will ingest germs. Validating your inputs makes it less likely you will succumb to a SQL injection hack or a buffer overflow. It is just something you do, and if you don’t, you become a social pariah.

Washing your hands will not keep you safe from hereditary neurological diseases, but the most common illnesses are not hereditary. Similarly, there are complex hacks that ordinary hygiene will not keep you safe from, but by far the great majority of hacks exploit the simple things we forget to do or the careless mistakes we make.

Good security hygiene is virtually free and extremely effective. It is mostly a matter of building new habits, but it prevents the vast majority of everyday security exploits. Sensitive documents should be shredded. User accounts should be given the minimum privileges possible. When employees leave the company, their accounts should be immediately terminated.

At USCIS, it was initially disturbing to employees when we required them to insert their badges into a slot in their keyboards when they wanted to log on. Eventually it became a habit and no one thought about it. They also had to get used to locking their screens when they walked away from their desks. But virtually all of the security disasters in the government in recent years have had to do with the theft of credentials and these simple measures helped prevent more of these disasters.

4. Adopt a Zero-Known-Defect Approach

I don’t understand the idea of allowing a known security vulnerability into production. What good does it do to bolt most of your doors, but leave one swinging open? It only takes one door for a thief to get into the house! This is not a risk/cost trade-off. You have wasted your other security spend if you leave a door open.

In a continuous delivery environment, code must pass all of its tests before it can be promoted. It is a simple green light / red light condition. All the more should security be treated this way. This might seem like a tremendous burden. But except in the case of a legacy system that has many outstanding security defects, good practices with automated regression test suites guarantee that, at any given moment, the only place there can be a defect is in the small amount of code that was just checked in.

At USCIS, when I reviewed each legacy system—with all of the major stakeholders in the room—I would ask what known vulnerabilities there were. I insisted on a plan for remediating each of them or establishing compensating controls, and a very aggressive timeline for doing so. All stakeholders came to understand that known vulnerabilities were not acceptable.

5. Continuously Vet Security, Both in Development and Production

The old process at USCIS involved reviewing each system every two to three years to make sure it was still secure, and then issuing it a new ATO. Instead, we started enrolling each system into an ongoing authorization process, where the system was continuously tested and assessed by automated tools while it was running. If any vulnerability was found, there was an immediate escalation process to deal with it. Essentially, we extended our pre-release security testing process into post-release; the same things that would have prevented an ATO when the system was first launched would now trigger an immediate escalation and remediation after launch as well.

This made security a proactive matter rather than something we thought about only when forced to. We not only wanted to release secure systems, but to prove to ourselves at every instant that we were as secure as we could be. Another way to look at it is that there is urgency not just when a system is under attack, but equal urgency when a flaw is detected that might make it vulnerable to attack.

Conclusion

The five techniques described here all contributed to establishing a culture where security was valuable and considered something we owed to our applicants and stakeholders. They involved breaking old habits and creating new ones, but they were not expensive or time-consuming. And most of all, they helped do away with the problematic idea that there is a trade-off between being secure and satisfying customers.

–Mark
@schwartz_cio
A Seat at the Table: IT Leadership in the Age of Agility
The Art of Business Value