Building a Better Security Team: Emotional Intelligence in DevSecOps

Clarke Rodgers:
Welcome to the Executive Insights podcast, brought to you by AWS. I'm Clarke Rodgers, Director of Enterprise Strategy, and I'll be your guide through a series of conversations with security leaders.

Today's guest is the very fashionable Hart Rossman Vice President of Global Security Services at AWS. Join us for our discussion on emotional intelligence, compliance, engineering, and incident response. Enjoy.

Hart, thank you so much for joining me.

Hart Rossman:
Oh, thanks for having me, Clarke.

Clarke Rodgers:
It's been a couple of years. You were one of our first guests on Conversations with Security Leaders. A lot of things have happened with your career. Please get us up to date.

Hart Rossman:
Well, I mean, first of all very clearly I've leveled up my wardrobe.

Clarke Rodgers:
You have indeed.

Hart Rossman:
I've come a long way from layered t-shirts back in 2022. And then since then, we've also formed the Global Services organization, which is really how we've brought together all of our field delivery services across AWS to better accelerate customer outcomes.

And in the process we created a dedicated security organization, which I look after called, very imaginatively, Global Services Security.

Clarke Rodgers:
Has a nice ring to it. And what does that encompass? If I'm a customer, what am I coming to you for?

Hart Rossman:
Yeah, so we do two things. First and foremost, we're focused on helping customers build, deploy, and operate securely in AWS and also build, deploy, and operate security solutions in AWS. So that's everything from compliance, engineering, incident response, threat detection, cryptography, identity, whatever it might be. Then the other thing we do is we look after security internally in the field at AWS. And so we help Amazonians who are looking to raise the bar for security in the field.

Clarke Rodgers:
Oh, interesting. So what would that look like? When you say an Amazonian in the field, is that at a customer site or is it Amazonian building a service and then you help them-

Hart Rossman:
Actually, it's both. Right?

So from what we call engagement security, we might help solution architects, our salespeople, a consultant in the field, get the right security outcome for their customer and doing it in the right way to protect both the customer and the employee. Then the other thing we'll do is we'll help our builders build faster at a higher security bar. So a good example of that is that gen AI has become a bit popular-

Clarke Rodgers:
I've heard.

Hart Rossman:
...over the last 18 months or so. And we wanted to make sure that across the field, our solutions architects, our cloud support engineers, our consultants, could build quickly and securely, really innovative gen AI solutions. So we collaborated internally with AWS security and our service teams to essentially create these golden paths to quickly, effectively, efficiently, and securely allow our builders to innovate with this new technology.

Clarke Rodgers:
That's very cool. I meet with a lot of CISOs. You meet with a lot of CISOs. I've seen a general trend over the last 18 months of going from how do I secure gen AI inside of my environment? So maybe I'm buying a third-party tool, maybe I'm using Bedrock, whatever the case may be, how do I make that as secure as possible? Transitioning to how do I use gen AI tooling for a security outcome?

Hart Rossman:
Yeah.

Clarke Rodgers:
Are you seeing that? And if so, how is your organization helping customers?

Hart Rossman:
Yeah, we absolutely are. I think there's a couple of interesting angles to that. First, is that I think a lot of these things, it's difficult to protect what you don't understand. So step one is just encouraging these security minded organizations to use the technology. Get comfortable with it, kick the tires.  Have some frivolous use cases that are valuable, like build a recipe book or something out of it. Write recipe chat bot, whatever it might be.

Clarke Rodgers:
Just to get comfortable with it.

Hart Rossman:
Just to get comfortable with it and just sort of think deeply about the types of use cases, the types of data that you might apply. At AWS, we've published the security scoping matrix for gen AI workloads that helps you in a very disciplined way think through what are the right outcomes from business and security standpoint and then allow you to apply the right controls and technology around that. That's kind of one element of it.

The other is, as you're pointing out how do we use the technology specifically to get good security outcomes? And when this was becoming popular, the AWS Cert was looking around to sort of understand how can we best help customers in this space?

One of the things we quickly keyed in on is that there was a lot of information out there about how to pen test or how to do red teaming or AppSec reviews of LLMs of gen AI. There really wasn't any publicly available information on how do you do incident response if an LLM was involved or if an LLM might even be part of the cause of the issue. So we dug really deep. We did some experimentation, we developed some run books and playbooks of it.

Because we think it's valuable for customers, we published it. We made available these automated run books and playbooks. We published a methodology for responding to security issues when gen AI might be involved. We've gotten really, really great feedback from customers about that. And so then we thought, well, we ought to be using this more ourselves. So we've worked with a couple of teams across AWS security and we have an internally built security responder chatbot essentially. And what it allows our responders to do is when they have an inbound ticket, they can ask this Chatbot questions that help them prioritize, help them triage and help them discover resolution paths or courses of action much quicker than if they have to follow a traditional investigative workflow.

Clarke Rodgers:
Can you share an example of what they might ask the chat bot?

Hart Rossman:
It sort of depends on the nature of the investigation, but I will share that one of the most interesting things is we put it out there in production and we thought these folks are battle-hardened veterans. They're going to ask it like a question, treat it a little bit like a tinker toy and then move on. It turns out that on average they ask 11 questions of the bot.

One of the reasons for that is because early use, they were able to so quickly get to an effective course of action that they feel it's good use of their time to work with the AI rather than pursuing a traditional investigative approach. Now that we've seen that, one of the things we really want to do is say, "Okay, well can we essentially guarantee them to get the right answer in three to five questions instead of maybe as many as 11?"

Clarke Rodgers:
Staying on the incident response, we recently announced AWS security incident response service.

Hart Rossman:
We did.

Clarke Rodgers:
Tell me all about it.

Hart Rossman:
We've been doing incident response on the customer side as shared responsibility for quite some time. It's something that customers have asked for help on. And we've seen a number of patterns and best practices and realized that we could do a lot more for our customers. If we could help them get security resolutions faster by providing the right technology and most importantly, the right collaboration tools.

Incident response is a team sport. For a typical customer, when they've got an escalating security issue, they've got a number of factions within their organization. They've got security, they've legal, they've PR communications, they've got their product team, network support. They've got all these folks, and then they might want to work with their cloud provider, somebody like AWS.

So we looked at these patterns over time and said, "We can create a self-service security incident response service that allows customers to get better outcomes quicker and gives them the option to bring in partners or the AWS Cert if they feel like they need some additional help."

The three features of the service at launch is first and foremost auto-triage and monitoring. If we know that this is kind of a daily activity, let's remove the undifferentiated lifting from the incident response team, which is often the smallest, least funded part of the security organization. We wanted to make it easier for them to focus on the high value tasks or activities. So you can turn on the auto-triage, auto-monitoring feature. It does prioritization, it does suppression, it does auto-containment for a number of typical issues that they might see based on their configuration and then allows them to prioritize tasking case workflow for the things that are truly worthy of a human's attention. That part of it then allows them again, to bring in a partner or third party or tag in the AWS Cert if they'd like some additional help.

Clarke Rodgers:
That's fantastic. I hope nobody actually has to use it, but it's great that the feature is there. Can you talk a little bit more about the partner angle of this?

Hart Rossman:
At the heart of everything in security, it's collaboration, it's a team sport. Sadly, the adversaries collaborate and work together. And so as defenders, we've got to do an even better job of that.  When we launched the service, we really wanted to make it partner-centric because it's important for the customer experience, for them to get the fastest security outcome possible. One of the things we observed working with customers over time is that as the collaboration builds, as you're responding or investigating, handling authorization, bringing in third parties and giving them access to things can be very tricky.

The case management system and workflow is designed from the beginning to integrate with partners and provide a flexible collaboration system to bring in third parties so you get the right expertise at the right time and you're not held back waiting for that identity ticket to get approved somewhere or trying to figure out how to get external counsel in through the firewall. Anything that slows down an effective containment and remediation is just the enemy of a good response.

Clarke Rodgers:
That's absolutely amazing and congratulations to you and your team on that release.

Hart Rossman:
It's been a lot of fun building and I think the most fun is seeing customers be able to do incident response with some of the team tooling and technique that the AWS Cert and our partners use.

Clarke Rodgers:
Incident response has been an area that has been a challenge for all sorts of customers. So hopefully this will help that. Another area where customers often have to focus is in compliance.

There's been an increased focus on compliance engineering. Can you talk a little bit about that and then maybe some offerings that you have at GSS?

Hart Rossman:
Yeah, it's a really interesting phenomenon and to be honest, a long time coming. We started a team a number of years ago called AWS Security Assurance Services. They're a PCI QSA company, high Trust assessor. They help in a number of other ways and from day one, the mission was to use AWS's services to help drive compliance outcomes for customers. Move away from the checkbox, move away from the audit checklist, that sort of thing. Over time, we've added other services like, literally, Audit manager, which is a great service that customers can use.

Clarke Rodgers:
Great Name. Yes.

Hart Rossman:
So what we're seeing now is our customers are starting to hire compliance engineers into their audit teams, into their audit functions. That's a beautiful thing to see. It's that early acceptance, acknowledgement that we can do API driven compliance at scale. So the way AWS Security Assurance Services reacting to it is, of course, we're leaning into it.

We're providing more consultative service around compliance engineering, compliance operations. You start with engineering. Got to build things, but then naturally customers are starting to think about, "Well, how do we operate fully automated compliance workflows?", right?

Then of course, if you're operating compliance, well now you want to do it effectively and efficiently. So now we start talking about things like machine learning and Generative AI, and how do we inject that into operations to get even more value for those engineering resources.

Clarke Rodgers:
So do you see this as an evolution or just sort of complimentary to the secure by design principles that are out there? Because the way I hear it, I could be just compliance by design.

Hart Rossman:
So I think it's darn close to revolutionary. I think the evolution is compliant by design, and we're seeing more of that with AWS, our services launching with third party assurance frameworks already evaluated. So we're seeing a lot of really good that compliance by design. Now, we're talking about compliance operations and that will be revolutionary.

It's far beyond the three layers of security, audit, compliance, compliance and audit, rather. Now we start thinking about is that full life cycle and I think the way we transformed software development from that slow, erratic waterfall model to Agilent DevOps, we're going to finally see that revolution come to the compliance and security assurance space.

Clarke Rodgers:
And then I would imagine, ultimately, the outcome is the businesses that are adopting this are releasing features more quickly, they're being more innovative, they're delivering on their business promise to their customers.

Hart Rossman:
Imagine that though. Can you imagine a year or so from now when some head of risk and compliance in a bank or in a healthcare company, instead of talking about their most recent audit, they talk about the compliance features they released to their engineering community or their product development community. Like that kind of tectonic shift in the way we think about the value of compliance and we think about the way we allow things to be compliant by design, compliant in operations through deep automation, machine learning, Generative AI, right? It's going to be fantastic.

Clarke Rodgers:
How do you have those conversations with customers? What is the impetus to the discussion where you say to them, "Hey, have you thought about compliance by design or compliance engineering?"

Hart Rossman:
Honestly, from my perspective, the best time to talk about that is when we're thinking about workforce planning, professional development. I'll give you a corollary, right? Like you want to build the most modern way possible with the right tools, techniques, and team composition. If you're launching a product or service using 2014 technology and a 2014 build model, not only do you already have a decade of tech debt from day one-

Clarke Rodgers:
Right.

Hart Rossman:
... But now you've got to the hill to get to where you want to be. An example of that is for every Dev that you hire, Software Development Engineer, you hire, you should hire a Data Scientist to work alongside them. Like a modern Dev team is DevOps and data science. It's all three of those combined. So for the teams that are saying things like, "Well, I've got a bunch of great Devs. We're going to launch our product or service or capability, and then in a year or so we're going to hire a data scientist to work with these 40 devs that we have." That's a great way to do it if you were in 2020 or 2021.

But where we are today, you've got to have these deeply integrated balanced teams.  Now we circle around to compliance automation, compliance engineering. It's the same mental model. If you're thinking about automating compliance, it's little things like don't just take the engineering intern this summer and ask them to work in compliance or audit, post a job rec for a compliance engineer. Hire somebody whose vocation will be that. Now, naturally they're going to be an SDE.

Clarke Rodgers:
Right.

Hart Rossman:
And if you're super lucky, they're an SDE that has some experience with governance risk or compliance. But the more important part, again, about the workforce planning and shaping the future of your business, that you're making a conscious decision to lean in to this trend, this bow wave, and you're going to hire compliance engineers. So when those people show up, they know that's their job. There's a development path, there's a promotion path, and that the work they're going to do is going to scale to the company's needs, not as an "other duty as assigned", but as a deeply convicted set of features and capabilities that are valuable to the company.

Clarke Rodgers:
You run a large business within AWS-

Hart Rossman:
It's very kind of you to say.

Clarke Rodgers:
... With GSS or at least a complex business. How about that? What sort of mechanisms do you use to keep current on everything that's going on within your line of sight, the appropriate metrics that you track, those kinds of things to make sure that you're showing the value of your business up and out across AWS?

Hart Rossman:
I think there's really two things. I'll start, in all honesty, with the leadership principles. A lot of time we'll talk about individual leadership principles and the value of understanding them, deploying them, learning from them. That's all very true. But I think to answer your question, we really have to look at the combination of certain leadership principles that allows a leader to stay close to the details and also drive the right outcomes right at scale.

For me, what I try to do is combine earns trust with hire and develop the best to get to our leadership principle of our write a lot as a leader. This idea if that we hire and develop our people and then we teach them to earn trust amongst themselves and with other teams. The team will be right a lot. So they'll have the information they'll need, they'll have access to the people and the technology they need to make good decisions.

Then if they're right a lot right then to be honest, that gives me an opportunity to be right a lot. I can learn and grow with the team. I've got the information and the talent and access that I need so that when they do come to me and they need some of my input, if they're right a lot often, then it kind of sets me up to be a little bit right too every now and again.

Clarke Rodgers:
I love that. That's absolutely fantastic. And a nice segue into the people element of security. Within your organization, you have many different roles that you hire for. Are there some key characteristics or traits that you're looking for that next great security hire within GSS?

Hart Rossman:
We talked a little bit earlier about this idea that you can't protect what you don't understand. And so we're very much looking for people who are deeply inquisitive, who are willing to take a risk, stretch themselves, learn and adapt to the new technologies, the new best practices, and really build on top of that.

You also need people in our space in security who are empathetic, who are emotionally intelligent. It's really easy in a high intensity field like security to get a lot of really smart people who don't support and reinforce each other. You'll get a reasonably well-performing org, but you won't have a high-performing organization.

If you're operating a team that you're also looking after customers like the AWS Cert, it also means you can't truly empathize and sympathize with the situation the customer's in to make them whole, to get them to the right outcome, both technically as well as organizationally. You have to go from a bump in the night to everything and everyone is all right.

Clarke Rodgers:
And there has to be an element of psychological safety.

Hart Rossman:
Absolutely. We've partnered with this amazing guy, Rich Hua at AWS. He's studied for many years now, the value of emotional intelligence and leadership. We spent some time over the last year, working directly with him to adapt some of his material, his training, his workshop content, what-have-you, specifically to security leaders.

Clarke Rodgers:
Interesting.

Hart Rossman:
People who work in security engineering, operations, incident response. We've now run workshops for hundreds of Amazonians and probably close to hundreds of customer employees as well now and get a ton of great feedback. These are learned skills. Learning to emotionally regulate to stay calm and stressful situations. Learning to draw out other people's motivation quickly in a particular situation to understand how to align and move forward together. There's just a number of these emotional intelligent skills that are not lightweight, they're not nice to have, they're essential if you want to get the most effective security outcomes, the quickest time possible and have a team that's going to learn and grow together to do even better in the future.

Clarke Rodgers:
That's fantastic to hear. And I'd also be interested to hear more about how you present it to customers that this is important to have the emotional angle to security.

Hart Rossman:
I'll tell you, it's often a little bit of a conversation like this. Again, we've only done it a few dozen customers, a couple of hundred of their employees. A hundred percent of the time is "How soon can you come in and talk to my leadership team?"

Clarke Rodgers:
Oh, that's great.

Hart Rossman:
Yeah. They love the idea. Sometimes they're asking to pair it with tabletops, incidence response tabletops.

And that's also super cool too. You do some of the technical, the day-to-day and then we also step outside of that and talk about leadership and emotional intelligence. It's been a great pairing.

Clarke Rodgers:
Love it. Hart, thank you so much for your time today.

Hart Rossman:
Cool. Thank you, Clarke.

Listen now

Listen now

Listen now