Building a Culture of Security

It is no longer sufficient to leave security to a team of specialists who watch over the enterprise’s risk posture and control it through a set of constraining policies. It is not enough to guard the boundaries of the enterprise’s network with firewalls, or to simply implement sets of controls specified in a compliance framework. Security is no longer a secret ingredient added to the stew just before it is served. On the contrary: security has become everyone’s job, an act of preparing all the ingredients beforethey’re added to the stew. Its management has become a strategic concern of the enterprise. And the way forward is for the enterprise to build a culture of security, an awareness of risks and controls and a set of norms and practices that align with keeping the enterprise secure.

It is traditional at this point in an article on security to tell frightening stories of companies humbled in the face of the vulnerabilities they left to be exploited by bad actors. I will not, for several reasons. The threats are in the news every day, and we are well aware of them already. More importantly, we must get used to thinking of security as a positive thing, a way of building, acting, and making decisions that is…well, just something we do, naturally, as builders and enterprise executives. Again, that is to say: enterprises must create a culture of security, rather than reactively responding to specific threats as they are encountered.

As soon as an enterprise deploys an IT capability, innumerable attempts will be made to hack it [1]. Our systems are constantly under attack by bad actors with automated tools looking for an easy way in. And that is not to mention the dedicated, talented hackers, the advanced persistent threats, who are willing to spend time and money to break into high-value targets they have selected. But the threats to our systems come not only from bad actors. IT systems can also be defeated by bad data, unexpected surges in usage, untested edge cases often involving concurrent operations, cascading failures, and speed issues that multiply geometrically. In his book Humans Vs Computers, Gojko Adzic cites many examples of unexpected inputs causing bizarre behavior. For example, consider poor Jennifer Null who was unable to buy airplane tickets and Christopher Null whose bank couldn’t send him statements—all because of their unfortunate last names! In order for our systems to securely perform their jobs, they must also be scalable, resilient, available, well-tested, performant, and tolerant of failures and unexpected inputs like people named Null. To be rugged is to be resilient in the face of the unexpected.

There is good news here. Security is, in many ways, free—in the same sense that quality has been said to be free. In the same sense that basic hygiene is (more or less) free—washing your hands, for example. In the sense that security is cheaper to build it in rather than to pay the consequences of having to add it later. Security is a type of quality. It is about assuring that IT capabilities will continue to work as designed when they are placed in real conditions—that is, under attack and faced with the unexpected.

Just as there is no real trade-off between quality and speed, there is no trade-off between security and speed. Interestingly, by far the greatest majority of exploits could be stopped by simple security hygiene. There is a very small set of weaknesses that account for the vast majority of break-ins (SQL injections and buffer overflows, for example, for those readers who are technical). Many software systems incorporate other third-party software, particularly open source, but use old versions that are known to have security vulnerabilities. There is no excuse for this: tools are available to identify these known issues, and a good automated regression test suite and deployment process will allow frequent patching.

Ask any CISO what his or her greatest security fears are and you will probably hear two: compromised credentials and failure to patch often enough. Add to this the top application vulnerabilities—SQL injections and cross-site scripting—and you have accounted for the vast majority of actual break-ins. But today’s good practices provide inexpensive ways to avoid these vulnerabilities without slowing down the delivery process or imposing undue burdens on users. Security is primarily not a matter of fancy, geeky engineering—it is a matter of following good practices as an everyday way of doing business. It is a matter of hygiene.

The best model that I have found for a hygiene-based, culture-driven approach to security is that of the Rugged Software, or Rugged DevOps, movement, which advocates building secure and resilient software because, simply, it is the right thing to do. According to the founders of the Rugged movement, “Rugged organizations produce rugged code—designed to withstand not just today’s threat, but future challenges as well.”[2] The key to ruggedness, they say, is cultural:

We believe that the key to producing secure code is to change your software development culture. We have to get beyond looking at the technology and look at the software development organization that created it. We believe this evolution has to start with the people, process, technology, and culture of that organization.[3]

Although they talk about “code” and “software development,” these principles apply to the entire enterprise. An enterprise that treats security and resilience as a “nice to have” quality—something that only their security specialists need to worry about—as an extra cost, as a burden, or that doesn’t think of it at all can never be rugged. That enterprise is doing an injustice to their customers, whose data is at risk; to their companies, who will suffer unnecessary costs and damage to their reputations; and to their investors, whose investments will lose value. And if the enterprise is a government agency, they may be doing damage to their nation and citizens.

Security and resilience should be—and are—concerns for all enterprise executives, managers, and employees. It should be fundamental to a company’s culture. Imagine the irony of a company that tries to build a culture of “putting the customer first” or even just “increasing market share” but does not make it a principle to protect the customer’s data! A cavalier attitude toward security and resilience is not (and should not be) acceptable.

Culture includes those norms of the company that are reinforced by everyone’s behavior, and by how they view everyone else’s behavior. What does a culture of security look like?

Drawing again from The Rugged Handbook:

“Rugged” describes software development organizations [that] have a culture of rapidly evolving their ability to create available, survivable, defensible, secure, and resilient software. Rugged organizations use competition, cooperation, and experimentation to learn and improve rather than making the same mistakes over and over. Rugged organizations also actively seek out threats and create defenses before they are a problem.[4]

Here are some tenets I have put together (drawing heavily from The Rugged Handbook) that I think apply to a rugged organization:

• A target: We understand that we are constantly under attack, both deliberate attacks and accidental ones. We incorporate that fact into everything we do.
• Education: We value education on security (technical or non-technical, depending on our roles). We try to keep abreast of developing threats, accept advice from our security specialists, and try to understand our security policies and rules.
• Hygiene: We always practice good security hygiene. It is part of doing things “right.” We do not give out our passwords. We do not share user accounts. We do not leave sensitive personal information lying on our desks when we go home at night. We use secure coding practices.
• Continuous Improvement: If we do leave sensitive information lying on our desks at night, we take feedback from the person who notices it and don’t do it again.
• Zero-Defect Approach: When it comes to security, we do not accept any known vulnerabilities. If we discover a problem, we fix it immediately. We do not “triage” security defects—deciding that some are worth fixing now and others are not.
• Reusable Tools: Because we take a strategic approach to security, we look across all of our IT systems and build tools and processes that can be shared between them. These might include reusable logging and monitoring, enterprise-wide user provisioning, and standardized onboarding and offboarding processes for employees.
• Unified Team: All parts of our organization collaborate to make security strong and systems resilient.
• Testing:We not only test our systems rigorously (primarily with automated tests) while they are being developed but also continue testing them while they are in production. We test failure scenarios (e.g., Chaos Monkey, Game Days) and our ability to respond to them.
• Threat Modeling: We try to think like a bad guy, just as we try to get inside our customers’ minds. We brainstorm possible routes an attacker might take to defeat our controls, and we test to make sure they cannot.
• Peer Reviews: Each technologist should be thinking not only about possible defects in their work, but also about possible security vulnerabilities. Code should always be reviewed by a peer, who should also be looking for vulnerabilities.

At USCIS (where I was CIO), which is a part of the Department of Homeland Security, we thought a lot about security. But at the time I joined, we did not really make security part of everyone’s everyday processes. Sure, we required all employees to pass an annual training course on security awareness. We had extremely skillful security engineers and penetration testers dedicated to keeping our systems safe. And we periodically performed social engineering audits, where our auditors would call employees, pretending to be from the IT helpdesk, and ask for their passwords. The auditors would wander the office at night looking for passwords that had been written down, possibly posted on sticky notes, and would look for scraps of personal information of our customers that had been left on people’s desks.

But for most employees, security was looked at as a burden. Teams that developed code hoped that the security folks would give their systems a thumbs-up so that they could be deployed to production. “Secure” often meant satisfying those security testers and passing compliance requirements. Systems were launched with known vulnerabilities that were listed in a tracking system to be addressed later. Allowing these vulnerabilities to exist was called “a business decision” based on “an analysis of risk.”

In an upcoming post I will discuss how we changed that culture and the extent to which we were successful in doing so. For now, this post is an outline of what a culture of security might look like. In future posts, I will discuss how to get there. But let us carry one thought with us into the next set of posts: security is simply one kind of quality.

–Mark
@schwartz_cio
A Seat at the Table: IT Leadership in the Age of Agility
The Art of Business Value

[1] According to this articlefrom CNBC, businesses in the UK received an average of 1,000 attacks per day. That’s nothing compared to the Department of Energy’s systems, which receive 10 million attacks on the average day, according to Forbes, or the Utah State systems which include some NSA computers and reportedly receive an amazing 300 million attacks per day, according to The Hacker News.

[2] The Rugged Handbook, https://www.ruggedsoftware.org/wp-content/uploads/2013/11/Rugged-Handbook-v7.pdf.

[3]The Rugged Handbook.

[4]The Rugged Handbook.