Overseeing AI Risk in a Rapidly Changing Landscape

Boards faced with the responsibility of overseeing a company’s use of AI are in a difficult position as the technology and its social and regulatory context change rapidly. The technology’s implications are unclear: are today’s generative AI models a powerful way to accomplish specific business goals, or are they a near-final step on the path to artificial general intelligence—AI that can mimic human intelligence? New machine learning (ML) models are constantly being released, each with different capabilities, trained on different datasets, and with different controls and guardrails built in. When those models are combined with proprietary corporate data and built into complex business workflows, the risks multiply and become evident only over time.

While society and regulations broadly agree on the principles of responsible AI—e.g., fairness, explainability (also known as interpretability), and privacy—there is little agreement on what these terms mean. There are varying (and incompatible) definitions of fairness and privacy.[1] Explainability can come in different granularities and degrees (explainable in what way, understandable to whom, and at what level of detail?). AI oversight is a crucial board responsibility, posing considerable reputational risk and ethical import. How can boards approach it?

To be more precise, I will refer to AI as the overarching field that tries to mimic human capabilities through technology. ML is a subset of AI that involves machines learning from training data and experience to construct a “model” of some aspect of the world and then use that model to perform tasks. Generative AI is a species of ML that generates content (documents, answers to questions, images, audio, etc.) in response to prompts. Generative AI is based on generic foundation models generally provided by a third party like Anthropic or OpenAI.

In a fast-changing domain like AI, three things are necessary for good oversight: (1) a way to uncover or sense risks that might not have been present or evident before, (2) an understanding of the risks’ severity and the possibilities open for mitigating them, and (3) an ability to make rapid and effective risk-based decisions that balance risk with agility and innovation despite constant change and uncertainty.

1) Identifying Risks

The first challenge is knowing what the risks are. ML models, by nature, are opaque: the model has learned a way to make decisions or produce content, and you don’t know exactly what it has learned. AI models are inconceivably complex; generative AI models may include hundreds of billions of parameters with complex relationships among them. In the many AI applications where users can prompt the model by entering text, we can’t know all the possible prompts they will type. Very complex models often exhibit “emergent” behaviors; they surprise you by becoming capable of doing things you didn’t plan for. And the behavior of AI models might trigger public sensitivities in ways you don’t expect.

To uncover emerging risks, the most important action you can take is to ensure you have a diverse group overseeing any AI initiatives, both operationally as an AI steering group and as a board. A diverse group—including technologists, HR leaders, legal counsel, and whoever in the organization is closest to diverse customer groups—is much more likely to ask the important questions and notice potential issues. Especially now, when the ethics of AI is so unsettled, it is important to have many eyes on any decisions to deploy AI applications. The goal is not just to reject AI applications that misbehave but to surface the possible issues that might arise, mitigate them when possible, and then decide whether to move forward.

2) Categorizing and Investigating Risks

To make sure the board understands the risks and possible mitigations, you should have at least one technologically savvy director. Risk levels of AI systems might be categorized on a scale of increasing risk from 1 to 4, with 1 representing a system used often but only internally and with closely constrained outputs. A 4 might be a customer-facing application with a broad scope of possible outputs. Risk levels also depend on the type of application—e.g., AI applications that act as assistants to help employees generate new ideas would be at the low end of the scale and systems that control medical devices would be at the high end. The risks might include security against tampering with the AI model, the possibility of toxic speech, the creation of a nondiverse range of images, and so on. There may or may not be many mitigations or controls available to address a specific risk. Simply having a human in the loop is often an effective mitigation. While you may want to go outside the board for independent assessments, technically savvy directors know when to seek that independent input, understand trade-offs in the company’s context, ask the right questions, and guide the conversation—especially when there is a judgment call to be made.

3) Making Risk-Based Decisions

Framing AI governance questions as questions of risk rather than purely technical matters is important for making good risk-based decisions. Put aside AI for a moment. When your company deploys any piece of software, there is risk. No matter how well it is tested, there still may be unidentified defects lurking. The purpose of testing software is to lower the risk of defects, not to eliminate it—all the more so with AI. Your company should constantly validate the AI application’s performance and explore the decisions it makes and the responses it gives while it is being used. But the risks of, say, hallucinations (untrue output that the application simply invents) or toxic speech will not be zero. As with many board oversight decisions, you must weigh the risks against the benefits.

It also helps to have a strong set of values to guide company behavior. Since the rules around AI are changing rapidly, and it is not always clear how they apply, a strong and clearly articulated set of values will help create a culture that can guide the company’s actions in the right direction. While guardrails and governance processes are important, they may not foresee all the situations that will arise, especially when decision-making is decentralized and rapid. A clear set of values, consistently reinforced, will engage all employees in ensuring that the company acts responsibly, and employees will be more likely to spot potential bad behavior by the AI and inform management.

Make sure you pay attention to information security. AI adds new attack vectors for cybercriminals to exploit. Your security people now need to worry about things like AI model poisoning and prompt injection (ask them to explain these). Bad actors can not only steal private data from your AI models but also make them misbehave, causing you reputational harm. As with other information security oversight responsibilities, you should make sure that you have empowered cybersecurity experts you can trust on the management team, the board, and among its independent advisers.

AI is tremendously powerful and will change the future of business. You can’t afford to sit back while everyone else takes advantage of it. But you also need to manage risk in an area where the risks keep changing. The best way to do that is to oversee in an agile way, responding quickly to emerging risks and evolving ways to manage them.

Mark Schwartz is an Enterprise Strategist with Amazon Web Services (AWS) and the author of Adaptive Ethics for Digital Transformation: A New Approach for Enterprise Leaders (Featuring Frankenstein vs. the Gingerbread Man).

[1] The Ethical Algorithm: The Science of Socially Aware Algorithm Design (Michael Kearns and Aaron Roth. Oxford University Press, 2019.)