What is AI?

Artificial Intelligence (AI) is a field of study where people want to create computers capable of reasoning, learning and autonomous decision making. While the field was around since 1950s with its rises and falls (see https://en.wikipedia.org/wiki/AI_winter), the renewed boom happened around 2022 when ChatGPT was released to the public.

It is quite interesting that if you talk to two people on the street and ask them what AI is to them, they would probably point directly towards ChatGPT. If you ask two researchers the same question, chances are you'll get very different answers. This is because AI has been around for quite a while, and is quite a broad term.

Somewhat simplified, most of the time AI is about creating a model of something that is usable in real-life settings. There have been broadly two approaches to tackling this problem:

• symbolic: take human knowledge, encode it in machine-readable format, and use it (keywords for searching: Symbolic AI, expert systems, ontologies);
• connectionist: create machines that learn from examples (keywords for searching: Machine Learning, Connectionist AI).

The struggle for both of these approaches was to make the AI that is useful enough in practical settings by being able to behave correctly in previously unseen situations.

For symbolic approaches it's quite tricky, since by design they can use only knowledge that was actually encoded by a human.

For connectionist approaches the tricky part is to be able to learn "the concept", meaning not simply memorize the examples but actually when given a new example being able to correctly solve the problem (for instance, if we wanted to know whether the written email is a spam or not).

ChatGPT belongs to the connectionist approach, and is powered by something called a Large Language Model (LLM). As you can guess, LLM is a model of human language, and in this series you'll learn a bit more about how it does so, as well as pros and cons.

Early LLMs were able to read text and output text, and that's it. Modern LLMs model not only text but also other kinds of information: images, audio, video in relation to text. Each kind of information is often called a modality, and thus modern LLMs are called multimodal, since they handle multiple modalities.

🎯 Why LLMs "make up" things

One of the most widely known quirks of modern LLMs is that they make up stuff, that is, generate plausible‑sounding but factually wrong statements. Some people call it hallucination, others call it confabulation. Regardless of how you call it, the effect is the same: the model produces convincing text that can even sow doubt in people who possess relevant knowledge.

The most accessible high-level explanation I've seen on this topic comes from Andrej Karpathy. Here is a quote to get you started:

In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet. You end up with these ethereal spirit entities because they're fully digital and they're mimicking humans. It's a different kind of intelligence."

-- Andrej Karpathy (discussion starts around 8:26)

I would say the fact that LLMs make stuff up is not necessarily a bug, but is in fact a feature. We just need to learn how to use it, which we will discuss in more detail in the next article.