How is AI taught to follow instructions?

In this post we'll take a high-level look at how LLMs are turned into instruction‑following assistants and how that transformation changes the nature of hallucinations.

📚 To recap the previous posts, we learned that:

1. LLMs are essentially word‑guessing machines trained to predict the next token in a text sequence (token is not the same as word).

2. LLMs can "make up" information (called hallucination or confabulation) when they're uncertain and choose an incorrect token.

3. Hallucinations often arise for two reasons:

   • a lack of training data on a particular topic or in a particular language (in which case it’s a bug);
   • the inherent ambiguity of language itself (in which case it’s a feature).

How LLMs are trained to follow instructions

Transforming an LLM into an instruction‑following assistant begins with a large, curated set of instruction-answer pairs. For example, the instruction (also called a prompt) "What are the main ingredients in a traditional Italian pizza?" might be paired with an answer "A traditional Italian pizza is made with tomato sauce, mozzarella cheese, fresh basil, olive oil, and meats such as ham or prosciutto". Training on these pairs teaches the model to generate context‑appropriate responses across a wide range of prompts.

Next, the instruction‑tuned LLM receives new prompts and generates several candidate answers. Human reviewers rank these answers, and the model is further tuned to increase its chances of producing the highest‑rated responses. This is a simplified description of Reinforcement Learning From Human Feedback (RLHF). The process repeats until the model consistently yields answers that align with human reviewers.

New kind of hallucinations 😱

This training procedure introduces its own side effects. Because the model must satisfy many reviewers simultaneously, it can produce answers that are broadly plausible but omit fine details, lack subtle distinctions, or use phrases such as "I'm not sure" or "It depends" more often than necessary. It can even lead to the invention of plausible‑sounding facts or the misrepresentation of real data. Such mistakes are much harder to spot than classic autocomplete hallucinations. These represent another kind of hallucinations that arises from the LLM's goal to please its users.

This new kind of hallucinations can be especially troublesome in systems that make decisions (almost) on their own - what people now call AI agents. Because these hallucinations are subtle and often sound like perfectly reasonable answers, they can slip past human oversight and lead to potentially severe consequences.

Training is costly

If you remember, the foundational LLM has been trained by repeatedly processing billions of texts trillions of times (not exact numbers, just to give you an idea of the scale). On top of that, LLMs are post-trained to follow human instructions.

Doing such training on a regular laptop would take ages, so there must be something much more powerful, which means consuming much more electricity.

Clearly, the training procedure has some cost to it, which we will discuss in the coming post - stay tuned!.