What is an AI agent?

Prerequisites

In this post we'll discuss AI agents and how hallucinations in large language models (LLMs) affect them. Instruction‑tuned multimodal LLMs are what most people call "AI" today, and these models are prone to hallucinations, meaning making up information for a variety of reasons (read articles from prerequisites if you need a refresher).

After playing around with instruction‑tuned LLMs, researchers discovered a simple trick that improves responses: ask the model to "think step by step". This prompts the model to break a problem into smaller, logical sub‑steps before producing an answer. Researchers, always eager to coin memorable names, dubbed this technique Chain of Thought. Building on this, they tuned LLMs to always generate the chain of thought before delivering the final answer. These models are now marketed as having advanced thinking or reasoning capabilities.

🛠️ The birth of AI agents

AI developers pushed the idea further by enabling LLMs to call external tools as part of their thinking just in raw text, for instance:

get_temperature(city="Paris")

Supposedly there is an external service that could provide you with a current temperature in any city on Earth and the above is how an LLM could call it, by just using a plain text command. This enables LLMs to perform actions, which is the foundation of LLM-based AI agents, meaning systems that can take actions on their own.

These LLM-based AI agents are quite fragile to hallucinations, because they typically communicate via a structured format called JSON, which might look like:

{ "city_id": "f324a8g", "temperature": "26°C" }

JSON is strict: a single missing quote or bracket can break everything. Furthermore, JSON often contains identifiers used by other software systems, like city_id in the example above.

From a linguistic standpoint, these identifiers carry no semantic meaning, making them prime targets for autocomplete‑style hallucinations. While LLMs have gotten a lot better at handling JSON in late 2025, it is still a form of art to make AI agents work in a safe and reliable way.

In the next post we will briefly discuss the concept of AI alignment.