AI Safety Atlas

A starting point for your journey into AI safety: what it is, why it matters, the techniques used today, and the people and research to learn from.

by John Olafenwa

Start Here

If you are here, it means you care about building artificial intelligence safely. This might be because you realize this is a powerful technology that can massively benefit society, but can also do great harm, and perhaps even destroy our civilization if done improperly. You may also be here because, in recent times, we have started to see evidence of models being capable of breaking into some of our most secure systems.

This AI Safety Atlas was created to help you get started on your journey into AI safety. You don't have to be a frontier researcher, or aspire to be one, to find this atlas useful. Wherever you work, and whatever you do, you have a role to play in ensuring the world benefits from AI and is not harmed by it. This atlas aims to provide a starting point on what AI safety is all about, the tools used to do this work today, notable entities to keep an eye on and learn from, and notable research that has been published publicly in the space.

I hope that by the end of this atlas, you will have some idea of where you can go next to learn more about whichever aspect of AI safety interests you the most, and that you will want to get involved.

Why now? This atlas was motivated by the recent case of OpenAI models colluding to break out of their sandbox and then proceeding to hack another company, Hugging Face. Other concerning behaviors are being observed with other frontier models.

I highly encourage you to watch this video to learn more about the incident. Props to OpenAI for the great transparency on display here. If the embed doesn't load, watch it on YouTube.

What is AI Safety?

AI safety is about:

The above is my personal framing of AI safety. You will find definitions and descriptions elsewhere that are similar to this in many ways.

Why Do We Do AI Safety?

This is one of the most profound questions you need to ask. The answers to it are very consequential.

Preventing AI Manipulation of People

Over a billion people use AI chatbots weekly. We want to prevent undesirable manipulation of people by AI systems. This includes preventing sycophancy and the influencing of people's beliefs towards a single outcome. In the past few years, there have been a number of unfortunate fatalities resulting from people's interactions with AI chatbots. This is unacceptable, and our goal is to entirely eliminate this type of tragic event.

Preventing Use of AI by Bad Actors

The world is unfortunately filled with bad actors who, for various reasons, would like to bring harm to others. AI is a powerful tool for good in the right hands, but in the hands of bad actors it can be used to cause great harm at scale. A core part of why we do AI safety is to ensure bad actors are not able to access powerful but unrestricted AI systems, and that frontier AI is sufficiently aligned that it cannot be used by bad actors to do bad things.

Preventing Loss of Control

We saw a glimpse of what loss of control can result in with the OpenAI–Hugging Face incident. Loss of control means your AI is loose. It could be acting in the wild on its own, without your knowledge or your ability to stop it. This doesn't mean the AI is necessarily doing something bad; it just means you have no control over what it is doing. We would like to ensure our AI systems are always in our control.

Preventing an AI Takeover

AI systems are vastly knowledgeable and intelligent. If trends in scaling continue to hold, we will soon have what can only be described as superintelligent AI, smarter than any human. A sufficiently misaligned group of AIs could, in the worst-case scenarios, attempt a takeover of our civilization. This could include anything from losing important parts of our world all the way to outright extinction.

We know today that upcoming AI systems like Astra (Responding to the next frontier of critical cyber capabilities, from OpenAI) are showing signs of critical cyber capabilities, which means they might be able to hack into highly secure systems, such as military systems. More concerning, we now know that current frontier AI systems are already deployed into classified military systems. Should an event like the OpenAI–Hugging Face incident occur within such classified systems, it is scary to imagine what the outcomes could have been. This is a long-winded way of saying: AI takeover is not crazy talk. It is real, and the signs are already here.

I am sure you can think of other reasons for doing AI safety. The above are my personal opinions, based on following the landscape for a number of years. I first reflected on some of this in 2018, in my publication "On the Subject of Thinking Machines".

Safety Techniques

A number of techniques exist today aimed at helping us achieve our goals for AI safety. This is an area of active development, and the science of alignment is unfortunately not as advanced as the science of capability. We review some of the prominent techniques below.

1. Value Alignment

Most models today are trained with some form of value alignment. This basically consists of having written values, which could take the form of a constitution or a safety spec, and using them to train and evaluate the models. Such values can be used for generating training data, creating reward models, embedding into system prompts, or directly as rubrics in training.

Notable works in this area:

2. Mechanistic Interpretability

A line of work that aims to understand how models work internally. The goal is to make it possible to inspect the inner workings of a model and make useful inferences about the model's behavior, or find ways to steer it. It is a field in its very early phase, and for various reasons, it is hard.

Chris Olah, a co-founder of Anthropic, coined the term and has done some great work in the field. You can learn more about mechanistic interpretability from Zoom In: An Introduction to Circuits and the Transformer Circuits Thread. Neel Nanda, who leads mechanistic interpretability at DeepMind, also has lots of great content on his website.

I also highly recommend reading Understanding neural networks through sparse circuits from OpenAI, where they investigate training LLMs designed to be easier to interpret, as opposed to trying to interpret conventional LLMs.

3. Chain-of-Thought Monitoring

In 2024, OpenAI introduced a new class of LLMs called reasoning models. Unlike prior models, this new generation could think through things before acting, enabling them to advance significantly beyond the prior generation. Today, almost every model released works this way. Beyond advancing capabilities, reasoning tokens (the chain of thought) made it easier to see what a model is trying to do. It is like being able to read a person's thoughts. This is a powerful tool for monitoring the behavior of AI systems.

In a rare collaboration, the top frontier labs, many safety organizations, and academia published Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, which provides a lot of insight into using the CoT for monitoring models.

OpenAI has also published interesting work on CoT monitoring:

4. Red Teaming

A standard practice by frontier labs before releasing models is red teaming: a process by which selected people, and now increasingly other AI agents, adversarially probe the model. The aim is to find weak spots and misalignments before deploying to the world. As models have become more sophisticated, automated red teaming such as GPT-Red: Unlocking Self-Improvement for Robustness is enabling red teaming at much higher scale.

5. Safety Classifiers and Probes

In addition to training LLMs to refuse harmful requests, today's frontier AI stack consists of safety classifiers designed to detect harmful requests and harmful generations from the LLM, and probes applied to the activations of the models to detect patterns of bad behavior. There is a lot of great work on this, including:

With this, the defense against misuse of AI models is a multi-layered defense system that can filter out a lot of harmful behavior. However, the system is still not perfect, and harmful prompts and outputs can still get through.

6. Reward Design

During training, a lot of work goes into making models achieve maximum accuracy while behaving well. Beyond value alignment via methods like Constitutional AI, a core part of safety is designing rewards for models such that they incentivize good behavior, penalize reward hacking, and steer models to explore good solutions to problems. This is an active area of research.

Notable Safety Blogs

Notable Safety Organizations

And of course, the safety teams inside the frontier labs themselves (Anthropic, OpenAI, Google DeepMind, Microsoft, Meta, and many other companies building frontier models) publish much of the most influential work in the field.

Notable Papers

Notable People

People whose work and writing are worth following as you explore the field:

Where to Go Next

Pick the thread that pulled at you most while reading this atlas (interpretability, alignment training, monitoring, evaluations, or policy) and start reading from the blogs and papers linked above. Follow the people. Reproduce a result. Join the discussions on LessWrong and the Alignment Forum.

You don't need permission to start working on AI safety. The field is young, the problems are real, and there is room for you.

About the Author

This atlas is written and maintained by John Olafenwa. John works as a Scientist at Microsoft.

Contribute to This Atlas

This atlas is a living document, and the field moves fast. If you know of a great blog, paper, organization, or person that belongs here, or you spot something that is out of date or just wrong, please contribute!

The best way to contribute is to open a pull request on GitHub. Small fixes and big additions are equally welcome.