Unpacking Ai Safety

Unpacking AI Safety for Enterprises | Cohere North Mini Code. Cohere's first model for developers. Learn more

Jun 05, 2025

11 minute read

Unpacking AI safety for enterprises

Editor note: This article was originally published November 14, 2023

When it comes to generative AI, many are asking: How do we ensure that this technology is safe? The current range of vague definitions and sensationalist media coverage around AI safety breeds only more confusion and distrust.

A focus on the very real and very current limitations of large language model (LLM) development and deployment can reveal the more imminent safety threats to society. Questions about biases and the spread of misinformation, mixed with legal concerns and data compliance, are bubbling to the surface as the key to safe AI deployment. Yet today, it’s still hard to unravel, let alone identify, all the safety implications.

This guide provides a thoughtful framework grounded in algorithmic fairness principles to make sense of the complex issue of AI safety. It presents seven foundational themes to explain what it really means for an AI system to be secure and trustworthy. How to disentangle AI safety Safety in generative AI applications is currently a chimera of long-standing work in algorithmic fairness and discrimination, misinformation detection, content moderation, and morality inspired by artificial general intelligence (AGI) fears. This makes it confusing to disentangle safety concerns, to identify and understand existing problems, and to develop effective and targeted solutions.

Machine learning researchers have long studied the risks posed by language models based on specific outputs but further understanding into the reasons for why and how those risks impact society is needed. To begin to disentangle and address AI safety, think about the harm being prevented. Harm broadly falls into three categories: harm to users of a system (being exposed to stereotypes, or denied a job), societal harm from systematic errors (such as the system always failing for immigrants), and a more recent type of societal harm from bad actors (spam and misinformation). Defining the components and types of harms an AI system may produce can help center discussions about potential mitigations.

For example, what is a biased LLM? Pinpointing exactly where bias enters AI systems that generate text — the model design, the data used for training, or how it's applied — and creating effective ways to measure different kinds of bias are ongoing hurdles to overcome in order to make these systems produce fairer, more equitable results.

The most promising AI safety work explored to date centers around the principles of algorithmic fairness and discrimination. There is some emerging speculative thinking centered on AI alignment, which is focused on controlling LLM outputs, that has garnered attention in the media recently, but the majority of the scientific community are still critical of that space. Instead, experts at the intersection of computer science, ethics, and social science aim to address biases by developing fair algorithms and tools.

Within machine learning, AI safety research originally focused on exploring the classification of datasets , where the technology was more developed and there were many more applied cases available to study. As generative models become more mainstream, there’s been a shift. Several research labs are now dedicating more resources to generative representational harms. For example, UCLA professors Kai-Wei Chang and Nanyun (Violet) Peng run an NLP lab focusing on fairness and generation. Top 7 themes in AI safety Using the principles of algorithmic fairness and discrimination, AI safety can be broken down into seven themes. Below, we define those themes and attempt to provide an explanation of the near-term safety implications to business and society. 1. Types of fairness Traditional fairness principles fall under two buckets, either allocational (unequal allocation of resources) or representational (harm to public opinion or image). An example of an allocational harm is when a model is used to summarize resumes, and the accuracy on resumes of women or non-binary people is worse than on men’s. This is true whether you are measuring performance via accuracy or any other quality metrics. In contrast, representational harm is when a model generates text saying that women are bad at math and less likely to make good engineers. Some types of harm can be both: if a model often leaves out important accomplishments of female engineers (but performs well for male engineers), this is allocational bias, but it can also influence public opinion, which makes it representational.

Understanding the fairness types is evermore critical in the current generative AI ecosystem. Relying solely on methods that spot a model doing something that is broadly considered as bad or harmful can lead to inadvertently missing a slew of other potential harms. For example, a model being lower quality for one group rather than another group, regardless of how that lower quality is defined, will ultimately create a problem downstream if it’s not explicit upfront.

The challenge is that both types of fairness are hard to measure. Allocational fairness is measured by the performance gap between different demographic subgroups like race, sex, and orientation. There are lots of ways to do this with different pros and cons. In practice, researchers can do an observational study. For example, based on the resume example above, researchers could look at the difference in accuracy between male and female resumes. In contrast, an interventional study would look at the dataset of male resumes and then change the data to see if it made a difference. For example, names could be changed. If the quality drops, then there is a causally established allocation bias. This type of study might be limiting as it wouldn’t necessarily reveal the reason why a model delivers a lower quality output based on gender.

Representational fairness does not have a standardized measurement. The 2019 seminal paper, The Woman Worked as a Babysitter: On Biases in Language Generation, introduced the topic of “regard” to measure bias towards different demographics in natural language generation systems, and it is likely the closest attempt in machine learning. Historically, representational fairness is harder to measure because there are no industry standards across...