From Language Models to Large Language Models

Language models have been around for a while, but they've only been able to grow to the scale they are today with self-supervision. This section gives a quick overview of what language model and self-supervision mean.

Language Models

A language model encodes statistical information about one or more languages. Intuitively, this information tells us how likely a word is to appear in a given context. For example, given the context My favorite color is, a language model that encodes English should predict blue more often than car.

The statistical nature of languages was discovered centuries ago.

Tokens

The basic unit of a language model is a token. A token can be a character, a word, or a part of a word (like -tion), depending on the model. For non-English languages, a single Unicode character can sometimes be represented as multiple tokens.

For example, GPT-4 — the model behind ChatGPT — breaks the phrase I can't wait to build AI applications into nine tokens. Note that the word can't is broken into two tokens, can and 't. You can see how different OpenAI models tokenize text on the OpenAI website.

Figure 1-1. An example of how GPT-4 tokenizes a phrase.

The process of breaking the original text into tokens is called tokenization. For GPT-4, an average token is approximately ¾ the length of a word. So, 100 tokens are approximately 75 words.

The set of all tokens a model can work with is the model's vocabulary. You can use a small number of tokens to construct a large number of distinct words, similar to how you can use a few letters in the alphabet to construct many words.

The tokenization method and vocabulary size are decided by model developers.

Two Main Types of Language Models

There are two main types of language models. They differ based on what information they can use to predict a token.

Figure 1-2. Autoregressive language model and masked language model.

Language Models as Completion Machines

The outputs of language models are open-ended. A language model can use its fixed, finite vocabulary to construct infinite possible outputs. A model that can generate open-ended outputs is called generative — hence the term generative AI.

You can think of a language model as a completion machine: given a text (prompt), it tries to complete that text.

Prompt (from user):       "To be or not to be"
Completion (from model):  ", that is the question."

As simple as it sounds, completion is incredibly powerful. Many tasks — translation, summarization, coding, and solving math problems — can be framed as completion tasks.

Self-Supervision

Language modeling is just one of many ML algorithms. There are also models for object detection, topic modeling, recommender systems, weather forecasting, stock price prediction, and more. What's special about language models that made them the center of the scaling approach behind the ChatGPT moment?

Supervision vs. Self-Supervision

For example, the sentence I love street food. gives six self-supervised training samples:

Table 1-1. Training samples from the sentence I love street food. for language modeling.

<BOS> and <EOS> mark the beginning and the end of a sequence. These markers are necessary for a language model to work with multiple sequences. Each marker is typically treated as one special token by the model. The end-of-sequence marker is especially important — it helps language models know when to end their responses (similar to how it's important for humans to know when to stop talking).

From Language Models to LLMs

Self-supervised learning means language models can learn from text sequences without requiring any labeling. Because text is everywhere, it's possible to construct massive training datasets — allowing language models to scale up to become LLMs.

A model's size is typically measured by its number of parameters. A parameter is a variable within an ML model that is updated through the training process. (In school, parameters were taught as a combination of weights and biases. Today, we generally use model weights to refer to all parameters.) In general — though not always — the more parameters a model has, the greater its capacity to learn desired behaviors.

From Large Language Models to Foundation Models

While language models are capable of incredible tasks, they are limited to text. As humans, we perceive the world not just via language but also through vision, hearing, touch, and more. Being able to process data beyond text is essential for AI to operate in the real world.

For this reason, language models are being extended to incorporate more data modalities. GPT-4V and Claude 3 can understand images and texts. Some models even understand videos, 3D assets, protein structures, and more.

While many people still call Gemini and GPT-4V LLMs, they're better characterized as foundation models. The word foundation signifies both the importance of these models in AI applications and the fact that they can be built upon for different needs.

A Breakthrough From the Old Structure of AI Research

For a long time, AI research was divided by data modalities. Each branch handled its own type of input, with little overlap.

A model that can work with more than one data modality is also called a multimodal model. A generative multimodal model is also called a large multimodal model (LMM). If a language model generates the next token conditioned on text-only tokens, a multimodal model generates the next token conditioned on both text and image tokens — or whichever modalities the model supports.

Figure 1-3. A multimodal model can generate the next token using information from both text and visual tokens.

Just like language models, multimodal models need data to scale up. Self-supervision works for them too. OpenAI used a variant called natural language supervision to train CLIP (OpenAI, 2021). Instead of manually generating labels for each image, they found (image, text) pairs that co-occurred on the internet — yielding a dataset of 400 million pairs, 400× larger than ImageNet, with no manual labeling cost.

CLIP isn't a generative model — it wasn't trained to generate open-ended outputs. CLIP is an embedding model, trained to produce joint embeddings of texts and images. "Introduction to Embedding" on page 134 discusses embeddings in detail; for now, you can think of embeddings as vectors that aim to capture the meanings of the original data. Multimodal embedding models like CLIP are the backbones of generative multimodal models such as Flamingo, LLaVA, and Gemini (previously Bard).

From Task-Specific to General-Purpose

Foundation models also mark the transition from task-specific models to general-purpose models. Previously, models were often developed for specific tasks, such as sentiment analysis or translation. A model trained for sentiment analysis wouldn't be able to do translation, and vice versa.

Figure 1-4. The range of tasks in the Super-Natural-Instructions benchmark (Wang et al., 2022).

Adapting a Model to Your Needs

Imagine you're working with a retailer to build an application to generate product descriptions for their website. An out-of-the-box model might generate accurate descriptions but fail to capture the brand's voice or highlight the brand's messaging. The generated descriptions might even be full of marketing speech and clichés.

There are three common AI engineering techniques to adapt a model to your needs. The rest of the book will discuss all of them in detail.

From Foundation Models to AI Engineering

AI engineering refers to the process of building applications on top of foundation models. People have been building AI applications for over a decade — a process often known as ML engineering or MLOps (short for ML operations). Why do we talk about AI engineering now?

The availability and accessibility of powerful foundation models lead to three factors that, together, create ideal conditions for the rapid growth of AI engineering as a discipline.

Three Factors Driving the Growth of AI Engineering

Figure 1-5. The number of S&P 500 companies that mention AI in their earnings calls reached a record high in 2023. Data from FactSet.

The Fastest-Growing Engineering Discipline

The world is quick to embrace this opportunity. AI engineering has rapidly emerged as one of the fastest — quite possibly the fastest — growing engineering disciplines. Tools for AI engineering are gaining traction faster than any previous software engineering tools.

Figure 1-6. Open source AI engineering tools are growing faster than any other software engineering tools, according to their GitHub star counts.

Why the Term "AI Engineering"?

Many terms are used to describe the process of building applications on top of foundation models — ML engineering, MLOps, AIOps, LLMOps, and so on. Why did I choose to go with AI engineering for this book?