AWS Security Profiles: Eric Brandwine, VP and Distinguished Engineer

In the weeks leading up to re:Invent, we’ll share conversations we’ve had with people at AWS who will be presenting at the event so you can learn more about them and some of the interesting work that they’re doing.

How long you have been at AWS, and what you do in your current role?

I’ve been at AWS for 11 years, and I report to Steve Schmidt, who’s our Chief Information Security Officer. I’m his Chief Engineer. I engage across the entire AWS Security Org to help keep the bar high. What we do all day, every day, is help AWS employees and customers figure out how to securely build what they need to build.

What part of your job do you enjoy the most?

It’s a trite answer, but it’s true: delivery. I enjoy impacting the customer experience. Often, the way security impacts the customer experience is by not impacting it, but we’re starting to launch a suite of really exciting security services for customers.

What’s the most challenging part of your job?

It’s a combination of two things. One is Amazon’s culture. I love the culture and would not change it, but it poses particular challenges for the Security team. The second challenge is people: I thought I had a computer job, but it turns out that like pretty much all of the Senior Engineers I know, I have a people job as much as or more than I have a computer job. Amazon has a culture of distributed ownership and empowerment. It’s what allows the company to move fast and deliver as much as it does, and it’s magnificent. I wouldn’t change it. But as the Security team, we’re often in a position where we have to say that X is no longer a best practice. Whatever the X is—there’s been a new research paper, there’s a patch that’s been published—everyone needs to stop doing X and start doing Y. But there’s no central lever we can pull to get every team and every individual to stop doing X and start doing Y. I can’t go to senior leaders and say, “Please inform your generals that Y needs to be done,” and have that message move down through the ranks in an orderly fashion. Instead, we spend a lot of our time trying to establish conditions, whether it’s by building tools, reporting on the right metrics, or offering the right incentives that drive the organization towards our desired state. It’s hacking people and groups of people at enormous scale and trying to influence the organization. It’s a tremendous amount of fun, and it can also be maddening.

Do you have any advice for people early in their careers about how to meet the challenge of influencing people?

I’ve got two lessons. The first is to take a structured approach to professional interactions such as meetings and email strings. Before you start, think through what your objective is. On what can you compromise? Where are you unwilling to compromise? What does success look like? Think through the engagement from the perspective of the other parties involved. This isn’t to say that you shouldn’t treat people like people. Much of my success is due to the amount of coffee and beer that I’ve bought. However, once you’re in the meeting or whatever, keep it on topic, drive towards that outcome, and hopefully end early so there’s time for coffee.

The other is to shift the discussion towards customers. As a security professional, it’s my job to tell you that you’ve done your job poorly. That thing that you sweated over for months? The one that you poured yourself into? Yeah, that one. It’s not good enough, it’s not ready to launch. This is always a hard discussion to have. By shifting the focus from my opinion versus yours to trying to delight customers, it becomes a much easier discussion. What should we do for customers? What is right for customers? How do we protect customers? If you take this approach, then you can have difficult conversations with peers and make tough decisions about product features and releases because the focus is always on the customer and not on social cohesion, peer pressure, or other concerns that aren’t customer-focused.

Tell us about your 2018 re:Invent topic. How did you choose it?

My talk is called 0x32 Shades of #7f7f7f: The Tension Between Absolutes and Ambiguity in Security. I chose the topic because a lot of security issues come down to judgment calls that result in very finely graduated shades of gray. I’ve seen a lot of people struggle with that ambiguity—but that ambiguity is actually central to security. And the ability to deal with the ambiguity is one of the things that enables a security team to be effective. At AWS, we don’t have a checklist that we go down when we engage with a product team. That lack of a checklist makes the teams more likely to talk to us and to bring us their problems, because they know we’re going to dig into the problem with them and come up with a reasoned recommendation based on our experience, not based on the rule book. This approach is excellent and absolutely necessary. But on the flipside, there are times when absolutes apply. There are times when you draw a bright line that none shall pass. One of the biggest things that can enable a security team to scale is knowing where to draw those bright lines and how to keep those lines immaculately clean. So my talk is about that tension: the dichotomy between absolute black and white and the pervading shades of gray.

What are some of the common misperceptions you encounter about cloud security?

I’ve got a couple. The first is that choosing a cloud provider is a long-term relationship. Make sure that your provider has a track record of security improvements and flexibility. The Internet, your applications, and your customers are not static. They’re going to change over time, sometimes quite quickly. Making it perfect now doesn’t mean that it will be perfect forever, or even for very long. At Amazon, we talk about one-way doors versus two-way doors. When going through a one-way door you have to be very sure that you’re making a good decision. We’ve not found a way to reliably and quickly make these decisions at scale, but we have found that you can often avoid the one-way door entirely. Make sure that as you’re moving your applications to the cloud, your provider gives you the flexibility to change your mind about configurations, policies, and other security mechanisms in the future.

The second is that you cannot allow the perfect to be the enemy of the good. Both within Amazon and with our customers, I’ve seen people use the migration to the cloud as an opportunity to fix all of the issues that their application has accreted over years or perhaps even decades. These projects very rarely succeed. The bar is so high, the project is so complex, that it’s basically impossible to successfully deliver. You have to be realistic about the security and availability that you have now, and you have to make sure that you get both better security when you launch in the cloud, and that you have the runway to incrementally improve over time. In 2016, Rob Joyce of the NSA gave a great talk about how the NSA thinks about zero-day vulnerabilities and gaining access to systems. It’s a good, clear articulation of a well-known security lesson, that adversaries are going to take the shortest easiest path to their objective. The news has been full of low-level side channel attacks like Spectre and Meltdown. While you absolutely have to address these, you also have to adopt MFA, minimize IAM policies, and the like. Customers should absolutely make sure that their cloud provider is someone they can trust, someone who takes security very seriously and with whom they can have a long-term relationship. But they also have to make sure that their own fundamentals are in order.

If you had to pick a different job, what would it be?

I would do something dealing with outer space. I’ve always read a lot of science fiction, so I’ve always had an interest, and it’s getting to the point where space is no longer the domain of large government agencies. There are these private companies that are doing amazing things in space, and the idea of one of my systems in orbit or further out is appealing.

The AWS Security team is hiring! Want to find out more? Check out our career page.