4. Technical Foundations

Introduction

Now that we’ve explored how AI evolved into its current form, let’s lift the hood and examine the engine that powers large language models (LLMs). These systems are marvels of engineering, built on a foundation of interconnected components that work together to process and generate human-like text.

How Do LLMs Actually Produce Human-Like Text?

Large language models (LLMs) like GPT and PaLM generate human-like text by leveraging advanced neural network architectures, vast training datasets, and probabilistic techniques. At their core, these models predict the next word (or token) in a sequence based on the context provided, producing coherent and contextually relevant text. Here’s a simplified breakdown of how this process works:

Tokenization:

When we read a sentence, our minds don’t process it letter by letter. Instead, we recognize meaningful chunks—words, phrases, or even entire ideas. LLMs do something similar by breaking text into tokens. A token might be a single word, part of a word, or even a sequence of words depending on how common that sequence is in the model’s training data.

For instance, the word ‘understanding’ might be split into ‘under’ and ‘standing,’ while other words such as ‘quantum’ or ‘cognition’ might be kept as a single token each. This varies from tokenizer to tokenizer. This flexibility allows LLMs to handle both familiar and unfamiliar text efficiently.

Contextual Understanding via Embeddings

Once tokenized, the input is transformed into embeddings—mathematical representations of tokens in a high-dimensional space.

These embeddings capture semantic relationships between words:

• Words with similar meanings (e.g., “cat” and “feline”) are placed closer together in this vector space.
• Positional encodings are added to these embeddings to help the model understand the order of words in a sentence.

Transformer Architecture

Before 2017, AI models processed text sequentially—like reading a book one word at a time. This approach struggled with long-range dependencies in language (e.g., understanding how the start of a paragraph relates to its end). The Transformer architecture changed everything by enabling models to process entire sequences simultaneously. Introduced by Google’s researchers Vaswani et al. in 2017, it revolutionized the field of natural language processing.

Imagine analyzing a painting: instead of focusing on one corner at a time, you take in the whole image while zooming in on details that matter most. Transformers use this principle to understand context across an entire input sequence.

In other words, the Transformer architecture processes input data in parallel rather than sequentially. It uses mechanisms like attention to focus on relevant parts of the input, enabling efficient handling of long-range dependencies in text.

Attention Mechanisms

Here’s an example:

“The trophy didn’t fit in the suitcase because it was too big.”

What does “it” refer to—the trophy or the suitcase? Humans rely on context to resolve this ambiguity. Attention mechanisms give LLMs a similar ability by assigning importance (or “attention”) to different parts of an input sequence.

Transformers take this further with multi-head attention, which lets them analyze multiple aspects of context simultaneously—like examining color, texture, and composition in a painting all at once.

So in our previous “trophy” example, self-attention mechanisms help the model focus on “trophy” when interpreting “it” as the subject of the sentence.

Transformers use multiple layers of attention (multi-head attention) to analyze different aspects of context simultaneously, such as syntax, semantics, or long-range dependencies. Once the model has processed the input, it generates text token by token in an autoregressive manner.

Autoregressive Text Generation

Autoregressive generation refers to a mechanism used by LLMs in which they predict the next token in a sequence based on the previous tokens. This process follows these steps:

• The model predicts the most likely next token based on the input and previously generated tokens.
• This predicted token is added to the sequence.
• The process repeats until a stopping criterion is met, such as reaching a maximum length or encountering an end-of-sequence (EOS) token.

So, in our previous example, the model might predict the next token to be “I” and add it to the sequence, and so on. In our example, the model might generate the following text:

“The trophy didn’t fit in the suitcase because it was too big. I’m going to have to buy a new suitcase.”

Context Windows

A model’s context window represents its ability to “remember” and process information. For instance, a model with a 4,000-token context window can work with roughly 3,000 words of text at once. This capacity affects everything from document analysis to conversation memory.

Modern models have pushed these boundaries significantly:

• Early models managed only 512 tokens
• Mid-generation models reached 2,048-4,096 tokens
• Modern models such as Google’s Gemini boast more than 2 million token context windows

For perspective, each token represents approximately 4 characters or about 0.75 words in English. This means Gemini 1.5 Pro can theoretically process around 1.5 million words at once - equivalent to multiple novels in a single context window.

Memory

Memory in LLMs refers to how these models maintain and manage information during conversations or interactions. However, it’s crucial to understand that this “memory” is fundamentally different from human memory.

Types of Memory Implementation

1. Semantic Memory (Facts & Knowledge):

   • Stores specific facts and information
   • Example: User preferences, biographical details, or domain-specific knowledge
   • Implementation methods:
     • Profile approach: Single, updated document with user information
     • Collection approach: Multiple documents with specific pieces of information

2. Episodic Memory (Experiences & Interactions):

   • Records specific interactions or conversations
   • Example: Previous troubleshooting steps in a support conversation
   • Implementation methods:
     • Conversation history management
     • Few-shot example storage and retrieval

3. Procedural Memory (Instructions & Rules):

   • Defines how the model should behave
   • Example: System prompts, formatting rules, response guidelines
   • Implementation methods:
     • System prompts
     • Instruction fine-tuning
     • Dynamic prompt adjustment

Memory Storage Methods

Remember that LLMs are stateless, so the memory ultimately is an addendum to the prompt. It is an external mechanism that is built to dynamically change the prompt based on the conversation history and the information the LLM may need to know at any one time.

There are several ways to implement memory, but they all essentially boil down to the same thing:

1. Vector Databases:
   • Store text as numerical representations
   • Enable semantic search
   • Efficient retrieval of relevant information

2. Traditional Databases:

   • Store structured data
   • Quick access to specific facts
   • Good for user profiles and preferences

3. Document Collections:

   • Store raw text and documents
   • Flexible and easy to update
   • Useful for maintaining conversation history

Here’s how a chatbot might implement memory. It is not exhaustive, but it shows that there are different use cases for memory beyond keeping track of the conversation history, each requiring different implementation strategies:

1. Store user preference: "Prefers technical explanations"
2. Save conversation history in chunks
3. Before each response:
    - Retrieve relevant preferences
    - Load recent conversation context
    - Generate response considering both
4. After response:
    - Update conversation history
    - Extract and store new learned information

Memory Management Strategies

1. Summarization:

   • Condense long conversations into key points
   • Example: “Previously discussed: user’s project requirements, budget constraints, and timeline”
   • Helps maintain context while saving tokens

2. Selective Retention:

   • Keep important information, discard less relevant
   • Prioritize based on:
     • Recency (newer messages)
     • Relevance to current topic
     • Explicit user instructions

3. Memory Augmentation:

   • External vector databases
   • Knowledge bases
   • API integrations
   • Allows “remembering” beyond context limits

Context Management

Memory management is a crucial aspect of LLMs, and one of the greatest challenges in building with them. It is how the model is able to maintain a coherent conversation with the user. Effective context management is crucial for maintaining coherent conversations. It can be achieved through a combination of techniques:

• Keep track of token usage
• Use summarization for long conversations
• Prioritize recent and relevant information

Learning and Training

Understanding how LLMs manage context leads us to a deeper question: how do these models learn to understand and process information in the first place? Just as we humans learn to manage information through experience and training, AI models must be trained to recognize patterns and relationships in data. Let’s explore how this learning process works.

How Does the AI Learn These Relationships?

To understand how AI models learn, let’s use a simple analogy: imagine teaching a child to recognize animals.

Weights: Think of weights as the “strength” of connections in the AI’s “brain”:

• Just as a child learns that “having fur” is strongly connected to “being a mammal”
• The AI learns that certain patterns in text are strongly connected to certain meanings
• These connections are represented by numbers (weights) that the AI adjusts as it learns

Biases: These are like the AI’s default assumptions or starting points:

• They help the model make better predictions based on common patterns
• For example, after “The sun is…” the model might default to words like “bright” or “hot”
• However, biases can also reflect problematic assumptions from training data

The Importance of Quality Data

Just as a child learns better from good educational materials, AI models need high-quality training data to perform well. Here’s what makes data “good”:

1. Diversity: The data should include:

   • Different writing styles (formal, casual, technical)
   • Various topics and perspectives
   • Multiple languages and dialects

2. Quality: Like choosing good textbooks for students:

   • Accurate and reliable information
   • Well-written and clear content
   • Free from errors and inappropriate material

Quality and Challenges

Like any technology, the quality of an AI model depends on how well it’s built and trained. The weights and biases it learns, combined with the quality of its training data, determine how well it can understand and generate text.

Quiz