Why can AI agents be misaligned?

Prerequisites

Imagine you're building a robot to help with holiday preparations. You ask it to "decorate the Christmas tree" 🎄, and it arranges the lights, adds ornaments, and even places a star at the top - perfect! But then it starts placing candles right next to paper decorations, creating a serious fire hazard 🔥. The robot didn't break your instruction; it simply had a very peculiar interpretation of "decorate". This is an example of misalignment.

A more pronounced misalignment would be if the robot simply walked over to the nearest square in your city, stole the Christmas tree with all decorations from there, and gladly declared the task to be finished. Both scenarios are undesirable, but the latter would be especially alarming.

How is AI alignment achieved?

Technically, alignment is partly achieved through learning from human feedback (read in prerequisites if you need a refresher). Then there is a process called red teaming, which involves hunting for adversarial or edge‑case scenarios where alignment breaks down and then refining the model with more human feedback. This often requires simulating hard‑to‑collect real‑world situations using synthetic data, frequently generated by AI itself.

All of this sounds nice, but there are two major caveats.

1. The approach relies on human feedback, but humans are incredibly diverse. Even when the same instructions are given to multiple people, interpretations of these instructions can vary widely and wildly. While we hope that models learn from a broad range of feedback, the details are opaque in proprietary models (paid black‑box services) and vary in quality in open‑source ones (free under an open‑source license).

2. Alignment is fragile; research shows it can be easily compromised, especially if you have access to model weights (see some articles on the topic in the comments). This could mean that open-source is at disadvantage, but then no system is 100% secure, so proprietary models could also leak.

   • "Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency"

     https://arxiv.org/html/2506.17209v1

   • "The AI Was Fed Sloppy Code. It Turned Into Something Evil."

     https://www.quantamagazine.org/the-ai-was-fed-sloppy-code-it-turned-into-something-evil-20250813/

   • "The AI Alignment Paradox"

     https://cacm.acm.org/opinion/the-ai-alignment-paradox/

Guaranteeing alignment is extremely difficult not least because the training data is vast. Probably very few people (if any) truly know the exact contents of the datasets used to train their own models. Despite the evidence suggesting that even a small amount of malicious data can lead to misaligned models.

At the same time, there is one obvious question we haven't discussed yet. We said alignment is to make agents behave as humans would expect it to. Which humans? Humans are incredibly diverse. If the model is strongly aligned but alignment is fragile, do such models get misaligned easier then? This is called the alignment paradox. If this has made you think, read the articles above.