When you’re moving at the speed of innovation, mistakes are bound to happen. In this Security Leaders interview, Bill Shinn, Senior Principal in the AWS Office of the CISO shares his advice on how to correct human errors without losing sight of the humanity of your workforce.
Part of this interview is also available in an audio format. Listen to the podcast by clicking your favorite player icon below, and subscribe to AWS Conversations with Leaders podcast to never miss an episode.
Join Clarke Rodgers, Director of AWS Enterprise Strategy for part two of his conversation with Bill Shinn, where we discuss the human side of security. Inevitably, people on the security team will make mistakes, but great leaders know how to turn errors into growth opportunities. See the video above or read the conversation in detail below to learn what mechanisms security leaders can adopt to catch and correct mistakes early and often.
In case you missed it, make sure to watch part one of Bill and Clarke’s conversation for more information on the role of security in product development.
Bill Shinn on the human side of security at AWS
Clarke Rodgers (00:14):
So, you joined AWS as a solution architect?
Bill Shinn (00:16):
I did, as the first security solution architect. I started as the first specialist solution architect in security and talked to all the customers because every customer wanted to talk to us about security. And the early conversations helped me trust the cloud, helped me understand how you secure the cloud, and it was very much security of how AWS is secured.
And then the architecture work was helping customers build on top of us with the solutions we had available in our services and open source solutions and things customers would build. And, along with our partners, allowed customers to meet their security and risk objectives. And then we take that feedback of things we can build and make it easier to be secure in the cloud.
So, over the course of those first five years, I worked with a lot of our large, globally significant financial institutions to understand their needs, to listen, to help work internally with our legal teams and regulatory teams to help shape kind of how we approach those really regulated markets.
It was also pretty timely. There were some changes to HIPAA and HITECH laws and regulations around that time. And so, we were being asked to sign business associate agreements from our healthcare customers. We had a lot of really early genomics workloads. We had some kind of disruptive type electronic medical record. We had a lot of medical imaging storage. There's a lot of PHI that people wanted to put on the cloud. And so we needed to do the work to be able to support those workloads as well. So I was kind of with a small team of people in sales and some of our security assurance team and legal. And that was a real opportunity to learn about the healthcare industry.
And so I think over those five years I've done a significant amount of executive briefings to CISOs and CIOs of our customers and was working really closely with AWS Security. But I think I wanted to be a little closer to where that work occurs internally. And so, having worked with Steve Schmidt, our CISO at the time, for years kind of took what I was doing, and I had built up a field technical community too. I'm currently still the worldwide technical leader for our security technical field community. That's the professionals out in our sales field, like solution architects that are focused on security to help our customers.
And so I kind of wanted to lead that, but from within AWS Security. So, I talked to Steve and we created the Office of the CISO. And for a little while there, it was just me and the team grew to a small number of people that it is now. But customers want to talk to the people that are doing the security work.
Defining the role of the AWS Office of the CISO
Clarke Rodgers (02:41):
So, let's go a little bit deeper into the role of the office of the CISO.
Bill Shinn (02:45):
Clarke Rodgers (02:45):
So there seems to be a customer facing element to it. But then there's also an internal facing side of it. Can you talk about maybe perhaps how a customer conversation ends up with an internal conversation and what the outcome could look like?
Bill Shinn (03:01):
Yeah. That happens a lot, a couple of different ways. One is a customer could come with an escalated request for information. So, something that we maybe haven't needed to share before. And they want to know something about the internals of AWS. And our office is really there to make sure that that answer meets their needs, that we're digging deep enough with our service teams internally to give them the information they need to make informed risk decisions.
And that left/right flow of information is of part of our office's responsibility. I think it's also to represent the voice of the customer in how we think about security. And we certainly have other mechanisms for that as well. But I think the field community that I lead is a fantastic source of where the trends are going, of what customers need, not just from a product and services perspective, but how we should operate as a partner in the industry.
And so that function of taking feedback from customers and providing it to our assurance functions. Also, customers expect our services to behave in a certain way. And so those customer security expectations, we kind of work to put that into our proactive security work internally. The other, and this happens directly from the field to the service teams and doesn't necessarily require us, but there is a lot of product feature requests and shared innovation with customers that comes through our function. And we work with our other service teams outside of AWS security when that has an element of risk management and security.
Clarke Rodgers (04:25):
And then security needs to be treated as a first-class citizen as part of all of this?
Bill Shinn (04:28):
Absolutely. One of the things that I’ve found, maybe as I get a little older, but also as I've been in the security industry, I always felt like there was a bit of, “If you can't write shell code, you're not a security professional.” And I think people have finally realized that there's a diversity of jobs in the security and risk management business. There's compliance, there's security assurance, there's training. Even IT audit I think is kind of a piece of that world. There's obviously engineering, there's a lot of operations functions that require a certain skill set. And so, having all those functions be at the table, I think is helpful. It helps you move faster, honestly, because they're going to have their stakeholders and the earlier they're involved, the better.
Nurturing and growing security talent at AWS
Clarke Rodgers (05:15):
So, let's go down that path for a minute. Once a security professional has joined AWS, whether it's in AWS Security or as a sales solution architect or whatever the case may be, what kind of growth opportunities do they have once they're here?
Bill Shinn (05:30):
There are so many roles in security, Clarke. I think we have the traditional functions you'd find in most security programs, at least within AWS Security. We have to manage vulnerabilities, we have security assurance and all of our compliance programs. And so that's a pretty sizable bit of work. There's opportunities I think, to move around. We encourage people to try different things.
Especially that's one of the things I like about AWS is the longevity of certainly the leadership team, but a lot of people in AWS Security have been here a long time. And for security professionals, I think the average tenure of a CISO is pretty low. But our leadership has been here for greater than a decade now. But we encourage people to move around, try new things. Sometimes we have people that work in proactive security or our application security team that does security reviews for services and features, and then they'll go work on a service team for a while.
Clarke Rodgers (06:19):
Bill Shinn (06:19):
And learn about what it's like to be a builder. And we're all kind of builders, especially the security folks, but the people in the service teams, their operational readiness requirements are very high. The rigor of engineering in the service teams is extremely detailed. And we strive for perfection. We rarely, rarely have the same issue twice and we deep dive on these things. But I think going back and forth like that gives a real, I think an empathy and a better understanding of what our builders need to do to be successful and fast. And then they take that knowledge and bring it back to AWS Security. That's a pretty common opportunity.
The security world is so big, even in the sales organization, we have people move back and forth, to AWS Security. They want to talk to our customers more. So, we certainly make opportunities for our internal security people to talk to customers, but it's not their primary job. And sometimes they go to our customers and then they work to see what it's like to be an AWS customer for a while and then they come back.
So, I think there's a huge demand for security talent in the industry right now. There's a lot of open positions. We want our people to feel like they can learn here. And so, we create a lot of opportunities. We have internal conferences, we run our own external security conference called re:Inforce. But I think there's ample opportunities for people to grow their careers here.
Clarke Rodgers (07:33):
So, it's clear that humans make up security teams, humans make mistakes. What kind of mechanisms do you put in place or does AWS Security put in place to catch these mistakes and make sure that they are one-offs and people learn from them and never happen again, so to speak?
What can security leaders do to minimize human errors?
Bill Shinn (07:54):
The more we can automate the better. And even with automation, if you don't predict the right canaries or the right failure modes, those robots can fail too. I don't like the term “insider threat.” It's kind of a trend in the security space. I think if you're talking about your employees and your coworkers, and yes risks exist, but there is a human factor there. But let's make it human, right — not assuming that everyone around you is a bad actor. I think the security industry, and I think I mentioned this a little bit earlier, I think it's evolved to be assuming best intentions I think, between what we can safely assume are good faith actors.
I think most people when they come to work, they want to do their job and they want to keep data safe and they want to be customer obsessed and earn trust with our customers. I assume that they do, otherwise they wouldn't have been hired, probably. And it's a different problem if we can't assume that safely. Going from that position of, not implicit trust, but assuming good intentions and good faith.
Clarke Rodgers (08:56):
Bill Shinn (08:58):
People make mistakes. I've made some good ones. But how you learn from it and how you react, it's similar to being a parent. It's how you react, right? It can be blameless. If it's more than several times and the same mistake, it's kind of a performance issue. But if it's a mistake that they didn't have the appropriate training, they didn't have the appropriate tools to do their job, they were under pressure and deadlines, there's always, not an excuse, but an explanation for why that mistake was made.
And it's not always that person's fault if they weren't supported in the correct way. If the tools don't work, the user experience is not simple and is too high of a bar to be secure in the way they do their business, it's our responsibility to work with those business functions and help them invent and simplify a way to be more secure and reduce the number of mistakes that are even possible.
So, I think for engineers, a lot of that is automation — putting things through pipelines at the appropriate test coverage. For business analysts and users, a lot of it is data classification controls and making sure that they know what data they're working on and what the responsibilities are for that data. Things like email mistakes, sending the right thing to the wrong person, or our technical support and customer support functions, they deal with a lot of customers and so, it's very important to be precise that you know who you're working on. It's a confused deputy problem, where you're doing something in one account or you're doing something for one customer and you suddenly think it's for another or you fat finger an email, those kind of things.
We have protections in place, because those are natural mistakes of people working at pace. So I think we learned, and as most of our customers learned, you have to build the scaffolding around to make it safe and easier to do their job. Also, having a culture of escalation here is so non-threatening, it is encouraged to escalate appropriately. Both things that you think are coming that people need to be aware of so they're ready for it and not surprised and also, when you see something, say something; it's just that simple. We make it extraordinarily easy to contact us at AWS Security. There's one webpage, everybody knows what it is. You push a button, it asks you what the issue is and it routes it to someone who's going to take very deep ownership of that until it's either resolved or triaged or we find someone who needs to actually fix it and take action. And there's estimated completion dates and follow up.
And the person that reported an issue either, "Hey, I got this email from an internal campaign to do some customer focused research than they're offering a hundred dollars gift card" or something. "And I think it might be phishing." Like great, it might be, but if it's not, we'd rather have them report that potentially. And then we can go back to the team that sent the email and say, “You might want to word this a little differently and people are very aware of phishing now, so be careful and don't include links,” and those kind of things.
But yeah, I think that that blameless culture, that culture of escalation, they're mechanisms that ensure that security remains the top priority here.
Clarke Rodgers (12:00):
Bill, thanks so much for joining me today.
Bill Shinn (12:02):
Sure. Thanks a lot, Clarke, good to see you.
About the leaders
Sr. Principal, AWS Office of the CISO
Bill Shinn spends his time helping security teams understand privacy, security and compliance as they move their business to AWS. Prior to AWS, Bill spent over 12 years managing and leading information security operations and architecture initiatives at some of the largest U.S. financial institutions, including U.S. Bank and JPMorgan Chase.
AWS Enterprise Strategist
As an AWS Enterprise Security Strategist, Clarke is passionate about helping executives explore how the cloud can transform security and working with them to find the right enterprise solutions. Clarke joined AWS in 2016, but his experience with the advantages of AWS security started well before he became part of the team. In his role as CISO for a multinational life reinsurance provider, he oversaw a strategic division’s all-in migration to AWS.
Take the next step