Comprehensive System Design of Large Language Models (LLMs)

LLMs 101 Deep Dive

Table of Contents

1. Introduction to Large Language Models (LLMs)
2. Architecture of LLMs
3. Tokenization and Embeddings
4. Attention Mechanism
5. Training LLMs
6. Inference and Decoding Strategies
7. Evaluation Metrics
8. Quantization
9. Retrieval-Augmented Generation (RAG)
10. Multi-modal LLMs
11. Custom Model Training
12. Conclusion

---

1. Introduction to Large Language Models (LLMs)

Large Language Models are deep neural networks trained on massive amounts of textual data to understand, generate, and manipulate human language. These models learn probabilistic patterns of language using architectures like Transformers.

Key Objectives of LLMs

• Language understanding (e.g., summarization, translation)
• Language generation (e.g., text completion, dialogue)
• Reasoning and knowledge retrieval

---

2. Architecture of LLMs

LLMs are typically built using the Transformer architecture, which consists of layers of self-attention and feed-forward networks.

Transformer Block

Input → Embedding → Multi-Head Attention → LayerNorm → Feed Forward → LayerNorm → Output

Diagram

---

3. Tokenization and Embeddings

Tokenization splits text into subword units.

Why Tokenization Matters

Tokenization directly impacts:

1. Model efficiency: Shorter sequences = faster training and inference
2. Generalization: Good tokenization reduces OOV (out-of-vocabulary) errors
3. Memory usage: Influences the length of sequences fed into LLMs
4. Language support: Determines how well the model handles multi-lingual text

Popular Tokenizers:

• Byte Pair Encoding (BPE)
• WordPiece
• SentencePiece

Comparison of Tokenization Techniques

Technique	Description	Pros	Cons
BPE	Merges frequent character pairs into subwords	Fast, widely used (GPT)	May struggle with rare tokens
WordPiece	Greedy longest-match-first subword tokenizer	Handles rare words better	Slightly slower than BPE
SentencePiece	Language-independent, treats input as raw text	No preprocessing needed	More complex training procedure

Embedding Layers

These convert tokens into dense vector representations. Positional embeddings are added to encode order. This allows the model to understand relationships like:

"king" - "man" + "woman" ≈ "queen"

Types of Embedding Layers and Their Characteristics

Embedding Type	Description	Example Use Cases	Pros	Cons
Static Embeddings	Pre-trained and fixed vectors for each word	Word2Vec, GloVe	Fast, simple, semantic relationships	No context awareness
Contextual Embeddings	Vary depending on context of the word in a sentence	ELMo, BERT	Captures word meaning in context	Computationally expensive
Learned Embeddings	Trained from scratch along with the model	GPT, custom models	Task-specific, flexible	Needs large data to generalize well
Positional Embeddings	Added to token embeddings to encode word order	Transformers	Allows sequence models without RNNs	Fixed max sequence length usually

Embedding Layer Code Example (PyTorch)

Here’s how you typically define and use an embedding layer:

import torch
import torch.nn as nn

# Example: vocab size of 10, embedding dimension of 4
embedding = nn.Embedding(num_embeddings=10, embedding_dim=4)

# Input: tensor of token indices
input_tensor = torch.tensor([1, 2, 4, 5])
output = embedding(input_tensor)

print(output.shape)  # torch.Size([4, 4])

Advanced Concepts

1. Subword Embeddings: Instead of words, we embed subword units (e.g., in BPE).

2. Learned Positional Embeddings vs Sinusoidal: Some models learn position vectors (like GPT), others use static sin functions (like original Transformer).

3. Fine-tuning: Pre-trained embeddings can be frozen or fine-tuned on downstream tasks.

4. Multimodal Embeddings: For images, audio, and text to coexist in a shared space (e.g., CLIP).

Best Practices

1. Use pre-trained embeddings when working with small datasets to avoid overfitting.

2. For sequence models (Transformers), always include positional embeddings.

3. Visualize embeddings using PCA or t-SNE to understand semantic clustering.

---

4. Attention Mechanism

The attention mechanism enables models to weigh the importance of different words dynamically.

Intuition Behind Attention

In natural language, not every word contributes equally to the meaning of a sentence. Attention allows the model to assign different weights to different tokens based on their relevance to the current word being processed.

Scaled Dot-Product Attention

Step-by-Step Formula:

Given input queries (Q), keys (K), and values (V):

\[\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{QK^\top}{\sqrt{d_k}} \right) V\]

• (Q): Query matrix (what you’re trying to match)
• (K): Key matrix (what you’re comparing against)
• (V): Value matrix (what you’re going to retrieve)
• (d_k): Dimension of key vectors (for scaling)

Explanation

• Compute similarity: (QK^\top)
• Scale it down by (\sqrt{d_k})
• Apply softmax to get probabilities (weights)
• Use those to get a weighted sum of the values (V)

Multi-Head Attention

Instead of applying attention once, it’s applied in parallel “heads”. Each head focuses on different subspaces of the input.

Input Embeddings
    ↓
Linear (Q, K, V projections)
    ↓
Split into h heads
    ↓
Apply Attention in each head
    ↓
Concatenate heads
    ↓
Final linear projection

Formula:

\[\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O\]

Where each head is:

\[\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)\]

Types of Attention and Their Roles

Type	Description	Used In	Pros	Cons
Scaled Dot-Product	Core attention formula using dot products and softmax	Transformers, GPT, BERT	Simple, efficient, effective	Can struggle with long sequences
Multi-Head Attention	Parallel attention on multiple projections	Transformers	Captures multiple relationship patterns	High memory cost
Self-Attention	Q=K=V from the same input	GPT, BERT	Enables parallel computation	Requires positional encoding
Cross-Attention	Q from one input, K/V from another	Encoder-Decoder, RAG	Good for translation and RAG	Doubles compute and memory cost

Self-Attention vs Cross-Attention

• Self-Attention: Used within a single sequence (e.g., sentence).
• Cross-Attention: Used when one input attends to a different sequence (e.g., decoder attending to encoder output in seq2seq models).

Architecture Diagram (Self-Attention Block)

Input Embedding
    ↓
Q, K, V projection (Linear Layers)
    ↓
Scaled Dot Product Attention
    ↓
Multi-Head Attention Output
    ↓
Add & LayerNorm
    ↓
Feed Forward Network
    ↓
Add & LayerNorm

Best Practices

Use masking in attention layers for autoregressive models (like GPT) to prevent future token leakage. Apply causal/self-attention for generative tasks and cross-attention for tasks like translation or RAG. Layer normalization and residual connections stabilize training and improve convergence. Sparse Attention Variants (like Longformer, BigBird) are recommended for very long sequences.

Advanced Attention Variants

Variant	Purpose	Benefit
Sparse Attention	Reduces attention complexity	Useful for long documents
Linformer	Projects keys/values to lower dimensions	Speeds up large model inference
Performer	Uses kernel approximations	Linear complexity w.r.t sequence
FlashAttention	GPU-efficient kernel	Speeds up training on large data

PyTorch Self-Attention

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = Q @ K.transpose(-2, -1) / torch.sqrt(torch.tensor(d_k))
    if mask is not None:
    scores = scores.masked_fill(mask == 0, float('-inf'))
    attn = F.softmax(scores, dim=-1)
    return attn @ V

---

5. Training LLMs

Training involves minimizing a loss function over a dataset.

Types of Training:

• Masked Language Modeling (e.g., BERT)
• Causal Language Modeling (e.g., GPT)

Pretraining Objectives

a. Masked Language Modeling (MLM)

Used in encoder-based models like BERT. Random tokens are replaced with [MASK] and the model predicts them.

Example: Input: “The [MASK] sat on the mat.” Output: “cat”

Limitations: Slower inference since BERT is not generative. Mask tokens never appear at inference time.

b. Causal Language Modeling (CLM)

Used in decoder-only models like GPT. Predicts next token given all previous ones.

Example: Input: “The cat sat on the” Output: “mat”

Strengths: Enables autoregressive generation. Easy to adapt to downstream generative tasks.

Hardware

LLMs are trained on multiple GPUs/TPUs using distributed training.

---

6. Inference and Decoding Strategies

Inference is the process of generating text from the trained model.

Decoding Techniques

• Greedy Decoding
• Beam Search
• Top-k Sampling
• Top-p (Nucleus) Sampling

Greedy Decoding

Definition

Always selects the token with the highest probability at each step.

Pros

• Fast and deterministic
• Good for constrained tasks (e.g., classification, token tagging)

Cons

• Often produces repetitive, dull, or incomplete text

Diagram

graph LR
A[Start: <s>] --> B["I"] --> C["am"] --> D["a"] --> E["robot"]

---

Beam Search

Definition

Maintains k most likely sequences at each time step, explores more options.

Pros

• Produces better quality output than greedy
• Suitable for translation and summarization

Cons

• Still tends toward safe/generic answers
• Expensive with large beam width

Best Practices

• Beam size between 3–10 for balance
• Combine with length penalty to avoid short outputs

---

Sampling

Basic Sampling

Randomly samples next token from the predicted distribution.

• Pros: Creative and diverse
• Cons: Can produce incoherent or off-topic text

Top-k Sampling

Only sample from top k highest probability tokens.

• Controls randomness and prevents rare, low-quality words

Top-p (Nucleus) Sampling

Select tokens whose cumulative probability exceeds p

• More adaptive than top-k, dynamic cutoff

Best Practices

• Top-k: Use values like 40 or 50
• Top-p: Start with 0.9 or 0.92
• Avoid combining top-k and top-p unless experimenting

---

Temperature Scaling

Definition

Adjusts the probability distribution to be more or less random

• Temperature < 1: Makes model more confident (sharper distribution)
• Temperature > 1: More exploratory (flatter distribution)

Formula

\[P'(x) = \text{softmax}\left(\frac{\text{logits}}{T}\right)\]

Best Practices

• 0.7–1.0: Balanced creativity and coherence
• < 0.5: Useful for deterministic results

---

Repetition Penalty / Frequency Penalty

Definition

Reduces probability of repeating the same tokens/phrases.

• Used in GPT-style models to avoid looped outputs
• Similar to penalizing frequency from previous outputs

Tip

• Repetition penalty ~1.1–1.3 can reduce verbosity

---

Contrastive Decoding (CoDi)

Idea

Combines two models:

• Base model: Generates fluent candidates
• Contrastive scorer: Penalizes generic or less informative options

Used in recent LLM applications to boost informativeness without loss of fluency.

---

Guidance and Constraints

• Token Constraints: For tasks like code completion or grammar correction
• Control Tokens: Used in models like CTRL to steer style/topic
• Logit Bias: Bias or suppress certain tokens (e.g., avoid offensive language)

---

Streaming and Efficient Inference

Techniques

• KV Caching: Cache key/value attention tensors to avoid recomputation
• Speculative Decoding: Draft with smaller model, verify with larger
• FlashAttention / FlashDecoding: Hardware-efficient attention computation

---

Human-in-the-Loop Inference

• Used in critical tasks like legal, medical, and enterprise decision-making
• Models suggest; humans verify and edit

Summary

Method	Use Case
Greedy	Classification, short form
Beam Search	Translation, summarization
Top-k	Poetry, dialogue
Top-p	Chat, storytelling
Contrastive	High-quality generation

---

7. Evaluation Metrics

Evaluating LLMs involves measuring how well a model performs across different tasks like text generation, classification, summarization, or question answering. Since these tasks differ greatly, choosing the right evaluation metric is essential.

Common Metrics:

• Perplexity: Measures uncertainty.
• BLEU: Measures accuracy against reference.
• ROUGE: Used for summarization.
• Human Evaluation

Intrinsic vs Extrinsic Evaluation

Intrinsic Evaluation measures performance using metrics derived from the model output alone. These are automated, fast, and reproducible.

Extrinsic Evaluation assesses the model in the context of downstream tasks or human feedback.

---

Common Metrics by Task Type

Text Generation

BLEU (Bilingual Evaluation Understudy)

• Measures n-gram overlap between generated and reference text.
• Best for machine translation.
• Limitation: Overly penalizes creative variations.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• Measures recall of overlapping n-grams (ROUGE-N) and longest common subsequences (ROUGE-L).
• Good for summarization.

METEOR

• Considers synonymy, stemming, and paraphrases.
• Better correlation with human judgment.

BERTScore

• Uses contextual embeddings from BERT to compute similarity.
• Evaluates semantic similarity.

MAUVE

• Measures distributional similarity between human and generated text.
• Uses KL divergence on embedding spaces.

Best Practice

Use a combination of metrics (e.g., ROUGE + BERTScore + human eval) to get a holistic view.

---

Classification and Regression

Accuracy

• Ratio of correct predictions to total predictions.
• Doesn’t capture class imbalance well.

Precision, Recall, F1

• Precision: How many predicted positives were correct?
• Recall: How many actual positives were captured?
• F1 Score: Harmonic mean of precision and recall.

ROC-AUC

• Area under the Receiver Operating Characteristic curve.
• Measures ability to distinguish classes.

Log Loss / Cross Entropy

• Penalizes confident wrong predictions.
• Better than accuracy for probabilistic models.

---

Retrieval Tasks (e.g., RAG)

Recall@K

• Measures how often the correct item is in top K retrieved results.

Mean Reciprocal Rank (MRR)

• Reciprocal of rank at which the first relevant document appears.

Normalized Discounted Cumulative Gain (nDCG)

• Considers both relevance and rank position.

---

Conversational AI / Chatbots

Human Evaluation

• Annotators rate coherence, helpfulness, safety.
• Expensive but reliable.

Win Rate

• % of conversations where model output is preferred over baseline.

Dialogue Evaluation Metrics

• USR (Unsupervised and Reference-Free)
• FED (Fine-grained Evaluation of Dialogue)

---

Safety and Toxicity Metrics

Perspective API / Detoxify

• Detects toxic, biased, or harmful content.

RealToxicityPrompts

• Dataset used to probe models for problematic behavior.

Bias Benchmarks

• BBQ, StereoSet, WinoBias to test gender, racial, or cultural bias.

---

Efficiency and Latency Metrics

Throughput (tokens/sec)

• Number of tokens generated or processed per second.

Latency (ms)

• Time taken for a single inference request.

Memory Footprint

• RAM or VRAM usage during inference/training.

---

Faithfulness and Hallucination Metrics

Factual Consistency Metrics

• FEVER score, FactCC for claim verification.
• GPTScore for hallucination detection using GPT-as-judge.

Best Practices

• Cross-check generated outputs with trusted data sources.
• Implement retrieval-augmented generation (RAG) to reduce hallucinations.

---

Multilingual and Multimodal Metrics

chrF / chrF++

• Character-level F-score for multilingual text.

BLEURT

• Uses pre-trained models to predict human ratings of MT outputs.

CLIPScore (for images + text)

• Measures alignment between visual and textual modalities.

---

Summary

Metric	Task Type	Pros	Cons
BLEU	Translation	Simple, Fast	Rigid matching
ROUGE	Summarization	Recall-focused	Ignores semantic meaning
BERTScore	Generation	Captures meaning	Needs BERT embeddings
Accuracy	Classification	Easy to interpret	Misleading with imbalance
F1 Score	Classification	Balanced view	Needs threshold tuning
Recall@K	Retrieval	Intuitive	Ignores rank order
GPTScore	Hallucination	High correlation w/ humans	Requires LLM as evaluator
Toxicity	Safety	Essential for deployment	Imperfect detectors

---

Recommendations

• Combine multiple metrics: no single score tells the whole story.
• Always include human evaluation for generative tasks.
• Tailor metrics to your task: ROUGE for summarization, Recall@K for retrieval, F1 for classification.
• Regularly evaluate with real-world user data to catch distribution drift or failure modes.

---

8. Quantization

Quantization compresses the model by reducing precision of weights (e.g., from FP32 to INT8).

Benefits

• Reduces model size and memory footprint
• Speeds up inference

Types

• Post-Training Quantization (PTQ)
• Quantization-Aware Training (QAT)

Post-Training Quantization (PTQ)

• Quantize a pretrained model without retraining.
• Fast and simple.
• Slight performance degradation.

Common Formats: INT8, FP16

Use case: Prototyping or deploying models quickly where slight accuracy drop is acceptable.

Quantization-Aware Training (QAT)

• Simulates quantization during training.
• Higher accuracy retention compared to PTQ.
• Requires access to training data.

Use Case: Scenarios where accuracy is critical.

Dynamic Quantization

• Activations are quantized on-the-fly during inference.
• Weights are quantized statically.

Use Case: RNNs, Transformer layers.

Static Quantization

• Both weights and activations are quantized.
• Calibration is done using a small representative dataset.

---

Precision Levels

Type	Size per Value	Example Use
FP32	32 bits	Training
FP16	16 bits	Mixed precision
INT8	8 bits	Inference (default)
INT4	4 bits	Memory-constrained

---

Challenges in Quantizing LLMs

1. Accuracy Drop: Especially with INT4/INT8 quantization
2. Outliers: Some weights/activations have very large values which dominate quantization buckets
3. Attention and Softmax Layers: Sensitive to precision reduction
4. LayerNorm & Embedding Layers: Require custom quantization strategies

---

Techniques to Overcome Challenges

1. Outlier-aware quantization
2. Mixed-Precision Quantization (e.g., keep attention heads in FP16, rest in INT8)
3. Per-Channel Quantization: Each channel has its own quantization parameters
4. Group-wise Quantization: Share quantization scale across groups of weights
5. Activation Clipping: Helps reduce dynamic range

---

Quantization Libraries and Tools

• Hugging Face Optimum
• Intel Neural Compressor
• NVIDIA TensorRT
• BitsAndBytes (for 4/8-bit quantization)
• ONNX Runtime
• PyTorch Quantization API

---

Quantization in Practice

Example: Using Hugging Face + BitsAndBytes

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b",
    load_in_4bit=True,
    device_map="auto",
    quantization_config={
    "bnb_4bit_compute_dtype": "float16"
    }
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")

---

Best Practices

• Benchmark Accuracy Post-Quantization
• Use Calibration Sets Close to Production Data
• Avoid Quantizing Embeddings Initially
• Start with INT8 before moving to INT4

• Test on hardware where deployment will occur

9. Retrieval-Augmented Generation (RAG)

RAG enhances LLMs by retrieving relevant documents during generation. Will cover this in depth in one of the upcoming articles.

Components

• Retriever: e.g., FAISS, BM25
• Generator: LLM (e.g., GPT)

Architecture

User Query → Retriever → Retrieved Contexts → Generator → Response

Benefits

• Reduces hallucinations
• Provides up-to-date information

---

10. Multi-modal LLMs

These models understand and generate data across modalities like text, image, and audio.

Examples

• GPT-4 Vision
• Flamingo
• CLIP (Contrastive Language-Image Pre-training)

Use Cases

• Visual Question Answering
• Image Captioning
• Multi-modal Search

---

11. Custom Model Training

Fine-tuning is the process of adapting a large, pre-trained LLM to a specific domain or task using a smaller, focused dataset. This can be done using full fine-tuning or parameter-efficient methods like LoRA and QLoRA.

Full Fine-tuning

Full fine-tuning involves updating all model parameters. This is effective but computationally expensive.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
model.train()

• Pros: Maximum performance
• Cons: High resource requirements

LoRA (Low-Rank Adaptation)

LoRA reduces training costs by injecting low-rank trainable matrices into transformer layers, freezing most of the model.

from peft import get_peft_model, LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
    bias="none"
)
model = get_peft_model(model, lora_config)

• Only small adapters are updated.

QLoRA

QLoRA combines 4-bit quantization with LoRA adapters for ultra-efficient training.

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)
model = AutoModelForCausalLM.from_pretrained("gpt2", quantization_config=bnb_config)

• Ideal for consumer hardware and cost-sensitive training.

Parameter-Efficient Fine-Tuning (PEFT)

Other PEFT methods include prefix tuning, prompt tuning, and adapters.

from peft import PromptTuningConfig, get_peft_model, PromptTuningInit

config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=10,
    tokenizer_name_or_path="gpt2"
)
model = get_peft_model(model, config)

Best Practices

Start with a well-aligned base model (e.g., GPT-NeoX, LLaMA). Use mixed-precision training (FP16 or BF16). Monitor training with tools like W&B or TensorBoard. Apply early stopping to prevent overfitting. Use adapters or LoRA when compute is constrained.

Case Study: LLaMA Fine-Tuning

Meta’s LLaMA models were fine-tuned using curated CommonCrawl and academic data. Training was done with FP16 precision on 2048 A100 GPUs. Efficiency techniques included flash attention and rotary positional embeddings.

Summary

Choose your fine-tuning method based on budget, dataset size, and deployment needs.

• LoRA/QLoRA: Fast, cheap, accurate for domain adaptation
• Full Fine-tuning: Maximum flexibility
• PEFT: Great for personalization at scale

---

12. Conclusion

From transformers to RAG and multi-modal systems, mastering these concepts requires a solid understanding of foundational principles as well as emerging trends in model compression, deployment, and fine-tuning.

For advanced learners, the next steps include exploring RLHF (Reinforcement Learning with Human Feedback), Mixture of Experts (MoE), and training pipelines with frameworks like HuggingFace, Ray, and Triton.