Introduction

• Recurrent Neural Networks (RNNs) are a class of artificial neural networks designed for processing sequential data
• Unlike traditional feed-forward neural networks, RNNs have connections that form directed cycles
• This allows them to maintain a memory of previous inputs in their internal state
• Particularly well-suited for tasks where the order of input data is important:
  • Time series analysis
    • e.g. Stock prices = sequence of real values
  • Natural language processing
    • e.g. a sentence = sequence of words (vector นั่นแหละ)
  • Speech recognition

Think of RNNs like reading a book: you remember what happened in previous chapters to understand the current page, unlike feed-forward networks that only look at one page at a time without context.

13.1 Architecture

Basic RNN Structure

Components:

• Input: $x$
• Hidden State: $h$
• Output: $\hat{y}$
• Delay operator: $\triangle$ (indicates feedback from previous time step)

Key Equations

Hidden state update equation at step $t$:
$\boxed{h_t = f(W_x x_t + W_h h_{t-1} + b_h)}$
Output at step $t$:
$\boxed{\hat{y}_t = W_y h_t + b_y}$

Where:

• $W_x$ = weight of the input
• $W_h$ = weight of the hidden (feedback) layer
• $W_y$ = weight of the output
• $b_h, b_y$ = biases of the hidden layer and output layer, respectively
• $x_t$ = input at step (time) $t$
• $f(\cdot)$ = activation function
• $\triangle$ = delay operator

Unrolling RNNs

• RNNs can be "unrolled" through time to visualize how they process sequences
• The network is replicated for each time step, with shared weights across all time steps

Unrolled structure:

Key assumptions:

• The hidden state at time $t$ ($h_t$) is computed from the current input $x_t$ and the previous hidden state $h_{t-1}$
• The output at time $t$ ($\hat{y}_t$) is computed from the current hidden state $h_t$
• RNNs make a Markov assumption: the current state depends on a finite fixed number of previous states

Think of unrolling like laying out a film strip: each frame (time step) shows the same camera (network) capturing different moments, but using the same lens settings (shared weights).

Multiple Recurrent Layers

• RNNs can be stacked into multiple layers for deeper representations
• Each layer processes the output of the previous layer
• Notation: $h^{[l]}_t$ represents hidden state at layer $l$ and time $t$

Example structure:

• Input → Hidden Layer 1 → Hidden Layer 2 → Hidden Layer 3 → Output
• Each layer has its own recurrent connections: $W^{[1]}_h, W^{[2]}_h, W^{[3]}_h$

13.1.1 Weights and Biases Sharing

Concept

• RNNs share weights and biases across different time steps to maintain consistency
• Regardless of how many times the RNN is unfolded, the number of parameters remains constant

Benefits of Weight and Bias Sharing

1. Parameter Efficiency
   • Reduces the number of parameters
   • Makes the model more efficient and easier to train
2. Generalization
   • Ensures the model can generalize patterns across different time steps in the sequence
3. Temporal Consistency
   • Maintains consistency in how inputs and hidden states are processed over time
   • Crucial for learning temporal dependencies

Imagine a translator who uses the same grammar rules (shared weights) for every sentence they translate, rather than learning new rules for each sentence. This makes them consistent and efficient.

Example 13.1: Stock Market Price Prediction

Given:

• Stock market prices of Company A over 5 days
• RNN with 1 hidden layer
• Parameters:
  • $w_x = 0.08$
  • $w_h = 0.02$
  • $w_y = 8$
  • $b_h = -7.9$
  • $b_y = 100$
  • $h_0 = 0$ (initial hidden state)
  • Identity activation function

Task: Predict the price of Day 6

Calculation steps:

Day 1:

• $h_1 = w_x x_1 + w_h h_0 + b_h$
  • $= (0.08)(100)+(0.02)(0)+(-7.9)=0.1$
• $\hat{y}_1 = w_y h_1 + b_y$
  • $=(8)(0.1)+100=100.8$ (Not necessary)
  • เพราะว่าเราอยากได้ $\hat{y}_5$, we need $h_5,h_4,h_3,h_2,h_1$ just that! ไม่ต้องเสียเวลาคำนวณ

Day 2:

• $h_2 = w_x x_2 + w_h h_1 + b_h$
  • $=(0.08)(105)+(0.02)(0.1)+(-7.9)=0.502$
• $\hat{y}_2 = w_y h_2 + b_y$

Day 3:

• $h_3 = w_x x_3 + w_h h_2 + b_h$
  • $=(0.08)(120)+(0.02)(0.502)+(-7.9)=1.71$
• $\hat{y}_3 = w_y h_3 + b_y$

Day 4:

• $h_4 = w_x x_4 + w_h h_3 + b_h$
  • $=(0.08)(130)+(0.02)(1.71)+(-7.9)=2.5342$
• $\hat{y}_4 = w_y h_4 + b_y$

Day 5:

• $h_5 = w_x x_5 + w_h h_4 + b_h$
  • $=(0.08)(125)+(0.02)(2.5342)+(-7.9)=2.1507$
• $\hat{y}_5 = w_y h_5 + b_y$
  • $(8)(2.1507)+100=117.2055\approx 117.21$

Day 6:

Day 6 เอามาหลอกเลยเด้อ5555

• $h_6 = w_x x_6 + w_h h_5 + b_h$
• $\hat{y}_6 = w_y h_6 + b_y$

Thus the predicted price for Day 6 is 117.21

13.1.2 How to Train RNN

Backpropagation Through Time (BPTT)

• To train an RNN, unroll it through time and use regular backpropagation
• This method is called Backpropagation Through Time (BPTT)

Training Steps

1. Forward Pass
   • Perform the forward pass through the unrolled network
   • For each time step, compute the hidden states and outputs using shared weights and biases
     
2. Loss Function
   • Use a loss (error) function $E_t$ to evaluate the output $\hat{y}_t$ against the target values $y_t$
3. Backward Pass (Gradient Calculation)
   • The gradients are propagated backwards through the unrolled network:
     • a) Output Weight ($W_y$):
       

$\boxed{\frac{\partial E_n}{\partial W_y} = \frac{\partial E_n}{\partial \hat{y}_n} \frac{\partial \hat{y}_n}{\partial W_y}}$
where $n$ is the time step of the interested output
- b) Input Weight ($W_x$):

$\boxed{\frac{\partial E_n}{\partial W_x} = \sum_{i=1}^{n} \frac{\partial E_n}{\partial \hat{y}_n} \frac{\partial \hat{y}_n}{\partial h_i} \frac{\partial h_i}{\partial W_x}}$
Important note:

• Since $h_i = f(W_x x_i + W_h h_{i-1} + b_h)$, we cannot treat $h_i$ as a constant
• The gradient from time step $t$ is calculated using the gradient from time step $t-1$
• Requires recursive calculation

c) Hidden (Recurrent) Weight ($W_h$):

$\boxed{\frac{\partial E_n}{\partial W_h} = \sum_{i=1}^{n} \frac{\partial E_n}{\partial \hat{y}_n} \frac{\partial \hat{y}_n}{\partial h_i} \frac{\partial h_i}{\partial W_h}}$

Important note:

• Similar to $W_x$, we cannot treat $h_i$ as a constant
• The gradient calculation requires recursive computation through all previous time steps

Think of BPTT like tracing back your steps: if you made a wrong turn (error) at the end of a journey, you need to walk back through all your previous steps to figure out where you first went wrong.

4. Parameter Update
   • Update the model parameters using the gradients computed during BPTT
5. Repeat
   • Continue the process until convergence

13.1.3 Vanishing and Exploding Gradients Problem in RNNs

อาจจะมีปัญหาเกิดขึ้นได้ 2 อย่าง

The Problem

• RNNs maintain a hidden state $h_t$ that gets updated at each time step $t$
• Update rule:
  $h_t = f(W_h h_{t-1} + W_x x_t + b_h)$

Gradient Flow Analysis

During BPTT, we compute gradients of the loss function $E$ with respect to network parameters:

$\frac{\partial E}{\partial W_h} \propto \sum_{t} \frac{\partial E}{\partial h_t} \frac{\partial h_t}{\partial W_h}$

Key term to understand:

$\frac{\partial h_t}{\partial h_{t-1}} = W_h \cdot \text{diag}(\sigma'(W_h h_{t-1} + W_x x_t))$

Let $\delta_t = \frac{\partial E}{\partial h_t}$. The gradient with respect to the hidden state at time step $t-1$ is:

$\delta_{t-1} = \delta_t \cdot \frac{\partial h_t}{\partial h_{t-1}} = \delta_t \cdot W_h \cdot \text{diag}(\sigma'(W_h h_{t-1} + W_x x_t))$

This recursive multiplication can cause two major problems:

1. Vanishing Gradients

Condition: If the eigenvalues of $W_h$ are less than 1 in magnitude

Mathematical expression:

$\|\delta_{t-k}\| \approx \|\delta_t\| \cdot \prod_{i=t-k+1}^{t} \|W_h \cdot \text{diag}(\sigma'(W_h h_{i-1} + W_x x_i))\|$

• For small eigenvalues of $W_h$, $\|\delta_{t-k}\|$ approaches zero as $k$ increases
• Gradients shrink exponentially as they propagate backward through time

Consequence:

• The model is unable to learn long-term dependencies
• Gradients are too small to make significant updates to the weights for earlier time steps

Like a whisper being passed through a long line of people: by the time it reaches the last person, the message has faded to almost nothing.

2. Exploding Gradients

Condition: If the eigenvalues of $W_h$ are greater than 1 in magnitude

Mathematical expression:

$\|\delta_{t-k}\| \approx \|\delta_t\| \cdot \prod_{i=t-k+1}^{t} \|W_h \cdot \text{diag}(\sigma'(W_h h_{i-1} + W_x x_i))\|$

• For large eigenvalues of $W_h$, $\|\delta_{t-k}\|$ increases exponentially as $k$ increases
• Gradients grow exponentially as they propagate backward through time

Consequence:

• The model's training becomes unstable
• Parameters update too drastically
• Leads to divergence or oscillations in the loss function

Like a snowball rolling down a mountain: it starts small but grows uncontrollably large, eventually causing an avalanche.

13.2 Long Short-Term Memory (LSTM)

Overview

• LSTM is a type of RNN architecture designed to address the vanishing gradient problem
• Introduced to enable learning of long-term dependencies in sequential data
• Uses a sophisticated gating mechanism to control information flow

13.2.1 LSTM Cell Structure

An LSTM cell consists of three main components:

1. Gates

Gates control the flow of information in the LSTM cell.

a) Forget Gate ($f_t$)

• Purpose: Decides what information to discard from the cell state
• Formula:

$\boxed{f_t = \sigma(W_{xf} x_t + W_{hf} h_{t-1} + b_f)}$

Think of the forget gate as a filter that decides which old memories to throw away, like forgetting what you had for breakfast last week.

• ถ้า Close to 0, discard, แต่ถ้า close to 1, it will change to 1
  • $\odot$ = element-wise multiplication

b) Input Gate ($i_t$)

• Purpose: Determines what new information to store in the cell state
• Formulas:
  $\boxed{i_t = \sigma(W_{xi} x_t + W_{hi} h_{t-1} + b_i)}$
  $\boxed{\tilde{c}_t = \tanh(W_{xc} x_t + W_{hc} h_{t-1} + b_c)}$

Note: $\tilde{c}_t$ is a candidate for cell state update

The input gate acts like a security checkpoint, deciding which new information is important enough to remember.

c) Output Gate ($o_t$)

• Purpose: Controls the output and the contribution of the cell state to the hidden state
• Formula:
  
  $\boxed{o_t = \sigma(W_{xo} x_t + W_{ho} h_{t-1} + b_o)}$

The output gate decides what information from your memory to actually use right now, like choosing which facts to recall for an exam.

2. Cell State ($c_t$)

• The cell state is the memory of the LSTM cell
• Carries information across different time steps
• Modified by the gates
• Update formula (where $\odot$ is element-wise multiplication):
  
  $\boxed{c_t = (c_{t-1} \odot f_t) + (\tilde{c}_t \odot i_t)}$

Interpretation:

• $(c_{t-1} \odot f_t)$: Keep parts of old memory based on forget gate
• $(\tilde{c}_t \odot i_t)$: Add new information based on input gate

The cell state is like a conveyor belt running through the entire chain, with gates adding or removing information along the way.

3. Hidden State ($h_t$)

• The hidden state is the output of the LSTM cell at each time step
• Used for predicting the next output
• Serves as input to the LSTM cell in the next time step
• Update formula:

$\boxed{h_t = \tanh(c_t) \odot o_t}$

The hidden state is your short-term working memory, focusing on what's immediately relevant from your long-term memory (cell state).

Key Distinction

• Cell State ($c_t$): Long-term memory
• Hidden State ($h_t$): Short-term memory

13.2.2 LSTM Workflow

Step-by-Step Process

1. Input
   • At each time step $t$, the LSTM cell receives:
     • Current input: $x_t$
     • Previous hidden state: $h_{t-1}$
2. Gates Computation
   • The forget, input, and output gates are computed to control information flow
3. Cell State Update
   • Cell state $c_t$ is updated based on:
     • Forget gate
     • Input gate
     • Candidate cell state
4. Hidden State Update
   • Hidden state $h_t$ is updated based on:
     • Cell state
     • Output gate
5. Output
   • The hidden state $h_t$ serves as:
     • Output of the LSTM cell
     • Input for the next time step

Visual Summary

13.2.3 Example 13.2: LSTM Calculation

Given

Input:

• $x_1 = 1, x_2 = 0.5$

Parameters:

• Weights: $w_{xf} = w_{xi} = w_{xc} = w_{xo} = 0.5$
• Weights: $w_{hf} = w_{hi} = w_{hc} = w_{ho} = -0.1$
• Biases: $b_f = 0.2, b_i = 0.3, b_c = 0.8, b_o = 0.3$
• Initial hidden state: $h_0 = 0$
• Initial cell state: $c_0 = 2$

Task: Calculate $h_1, h_2, c_1, c_2$

Time Step t = 1

Forget Gate:
$f_1 = \sigma(w_{xf} \cdot x_1 + w_{hf} \cdot h_0 + b_f)$
$= \sigma(0.5 \cdot 1 + (-0.1) \cdot 0 + 0.2)$
$= \sigma(0.7)$
$= 0.668$

Input Gate:
$i_1 = \sigma(w_{xi} \cdot x_1 + w_{hi} \cdot h_0 + b_i)$
$= \sigma(0.5 \cdot 1 + (-0.1) \cdot 0 + 0.3)$
$= \sigma(0.8)$
$=0.690$

Candidate Cell State:
$\tilde{c}_1 = \tanh(w_{xc} \cdot x_1 + w_{hc} \cdot h_0 + b_c)$
$= \tanh(0.5 \cdot 1 + (-0.1) \cdot 0 + 0.8)$
$= \tanh(1.3)$
$=0.861$

Cell State Update:
$c_1 = f_1 \cdot c_0 + i_1 \cdot \tilde{c}_1$
$=(0.668)(2)+(0.690)(0.861)=1.93$

Output Gate:
$o_1 = \sigma(w_{xo} \cdot x_1 + w_{ho} \cdot h_0 + b_o)$
$= \sigma(0.5 \cdot 1 + (-0.1) \cdot 0 + 0.3)$
$= \sigma(0.8)$
$=0.690$

Hidden State:
$h_1 = \tanh(c_1) \cdot o_1$
$=(0.9587)(0.690)=0.6615$

Time Step $t = 2$

Forget Gate:
$f_2 = \sigma(w_{xf} \cdot x_2 + w_{hf} \cdot h_1 + b_f)$
$=\sigma((0.5)(0.5)+(-0.1)(0.6615)+0.2)$

Input Gate:
$i_2 = \sigma(w_{xi} \cdot x_2 + w_{hi} \cdot h_1 + b_i)$
$=\sigma((0.5)(0.5)+(-0.1)(0.6615)+0.3)$

Candidate Cell State:
$\tilde{c}_2 = \tanh(w_{xc} \cdot x_2 + w_{hc} \cdot h_1 + b_c)$
$\tanh((0.5)(0.5)+(-0.1)(0.6615)+0.8)$

Cell State Update:
$c_2 = f_2 \cdot c_1 + i_2 \cdot \tilde{c}_2$
$=(0.595)(1.93)+(0.619)(0.755)$

Output Gate:
$o_2 = \sigma(w_{xo} \cdot x_2 + w_{ho} \cdot h_1 + b_o)$
$=\sigma((0.5)(0.5)+(-0.1)(-0.6615)+0.3)$

Hidden State:
$h_2 = \tanh(c_2) \cdot o_2$
$\tanh(1.616)\cdot0.619=(0.924)(0.619)=0.572$

From the given sequence [1,0.5], the calculated LSTM states are

• At $t=1$, $h_1=0.6615, c_1=1.93$
• At $t=2$, $h_2=0.572,c_2=1.616$
  The final output is $0.572$

13.2.3 How LSTM Solves the Vanishing Gradient Problem

Constant Error Carousel (CEC)

• The cell state enables the constant flow of error through the network during backpropagation
• Gradients can pass through many time steps without diminishing significantly
• This is the primary mechanism that addresses vanishing gradients

Why it works:

• The cell state update uses additive operations rather than multiplicative:
  $c_t = (c_{t-1} \odot f_t) + (\tilde{c}_t \odot i_t)$
• The gradient can flow through the addition operation without being repeatedly multiplied by weight matrices
• This preserves gradient magnitude over long sequences

Imagine a highway (cell state) where information can travel quickly without traffic lights (no repeated multiplications). This prevents the signal from getting weaker over long distances.

Gate Mechanism

• The gates (forget, input, output) control the amount of information flowing in and out of the cell state
• Ensures that important information is retained over long sequences
• Protects against both vanishing and exploding gradients by regulating information flow

Key advantages:

• Selective memory: Forget gate discards irrelevant information
• Selective updates: Input gate chooses what new information to add
• Selective output: Output gate controls what to use from memory

13.2.4 How LSTM Solves the Exploding Gradient Problem

While vanishing gradients are more common, LSTMs also help with exploding gradients:

1. Gradient Clipping

• Although not specific to LSTMs, gradient clipping is often used alongside LSTMs
• Technique that prevents gradients from becoming excessively large
• Sets a threshold: if gradient exceeds it, scale it down

2. Regulated Flow of Information

• The gates in LSTMs regulate the flow of information
• Prevents uncontrolled growth of gradients
• Controls input and output at each time step through sigmoid activations (bounded between 0 and 1)

Why it works:

• Gate values are bounded: $\sigma(\cdot) \in [0, 1]$
• Prevents explosive growth through multiplication
• Provides stable training dynamics

Summary

LSTMs effectively address both gradient problems through:

1. Architecture design: Cell state with additive updates
2. Gating mechanism: Controlled information flow
3. Combined techniques: Gradient clipping when needed

This makes LSTMs well-suited for various sequence modeling tasks requiring long-term dependencies.

Think of LSTM gates as a smart thermostat: they keep the temperature (gradient) in a comfortable range, preventing it from getting too cold (vanishing) or too hot (exploding).

13.3 Transformer Model

• seq2seq model e.g. machine translation

Overview

• Transformer model introduced by Vaswani et al. in "Attention is All You Need" (2017)
• Revolutionary architecture that changed NLP and sequence modeling
• Key innovation: Relies entirely on self-attention mechanism
• Does not use recurrence like RNNs and LSTMs
• Draws global dependencies between input and output

13.3.1 Self-Attention Mechanism

Overview

• Self-attention is the key innovation of the Transformer model
• Allows each position in the input sequence to attend to all other positions
• Enables parallel processing (unlike sequential RNNs)

How Self-Attention Works

Step 1: Linear Projections

Input embeddings are linearly transformed into three vectors:

1. Queries (Q): What am I looking for?
2. Keys (K): What do I contain?
3. Values (V): What information do I carry?

Formulas:
$Q = XW^Q$
$K = XW^K$
$V = XW^V$

where $X$ is the input matrix and $W^Q, W^K, W^V$ are learned weight matrices.

$Q,K,V$ ก็เอาไปผ่าน Layer Neural Network เหมือนคำนวณออกมาชั้นนึง

Step 2: Scaled Dot-Product Attention

Formula:

$z=\bar{x}=\boxed{\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V = AV}$

• บางครั้ง $z$ ($\bar{x}$) เรียกว่า Contextualized representation

where:

• $d_k$ = dimension of the key vectors
• $\frac{QK^\top}{\sqrt{d_k}}$ = scaled dot product (attention scores)
• $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ = attention weights
• Final output = weighted sum of values

Why scaling by $\sqrt{d_k}$?

• Prevents dot products from getting too large
• Keeps gradients stable during training
• Helps softmax function work in optimal range

Think of attention like a search engine: Queries are your search terms, Keys are webpage titles, and Values are the actual content. The mechanism finds which pages (Keys) match your search (Query) and returns relevant content (Values).

Step 3: Multi-Head Attention

• Uses multiple self-attention heads simultaneously
• Each head learns to focus on different aspects of the input
• Outputs are concatenated and linearly transformed

Formula:

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$

where:

$\text{head}_i = \text{Attention}(QW^Q_i, KW^K_i, VW^V_i)$

Benefits:

• Model can jointly attend to information from different representation subspaces
• Different heads can focus on different types of relationships (e.g., syntactic, semantic)
• Increases model's expressive power

Multi-head attention is like having multiple expert readers analyze a text simultaneously: one focuses on grammar, another on meaning, another on context, and their insights are combined.

Self-Attention for Long-Range Dependencies

• Self-attention allows the model to capture long-range dependencies
• Each word can directly attend to every other word in the sequence
• No need to pass information through many intermediate steps (unlike RNNs)

Example visualization:

• When processing the word "it" in "The animal didn't cross the street because it was too tired"

• Self-attention can directly connect "it" to "animal" regardless of distance

• Different attention heads might connect different word pairs

• แต่ละ Words ฝั่งซ้าย represented by $x_1,x_2,x_3,x_4,\dotso$

  • งั้นถ้าความหมายเหมือนกัน 100% คำว่า it ทั้งสองฝั่งก็ต้องเหมือนกัน เท่ากัน 100% สิ
  • เรา Calculate attention จาก it ฝั่งขวา เพื่อหา Stronger attention
    • แต่ละ Attention notation คือ $a_{1,8},a_{2,8},a_{6,8},a_{8,8},a_{12,8}$
  • สุดท้ายคำนวณ $z_8=\sum_i a_{8,i} v_i$ ซึ่งค่าของ $z_8$ ทั้งสองฝั่งก็จะไม่เท่ากันละ!

Scaled Dot-Product Attention Process (Detailed)

Given: Input sequence $X = [x_1, x_2, x_3, x_4]$

Step 1: Create Q, K, V matrices
$Q = XW^Q = [q_1, q_2, q_3, q_4]$
$K = XW^K = [k_1, k_2, k_3, k_4]$
$V = XW^V = [v_1, v_2, v_3, v_4]$

Step 2: Compute attention scores
$QK^\top = \begin{bmatrix} q_1 \cdot k_1 & q_1 \cdot k_2 & q_1 \cdot k_3 & q_1 \cdot k_4 \\ q_2 \cdot k_1 & q_2 \cdot k_2 & q_2 \cdot k_3 & q_2 \cdot k_4 \\ q_3 \cdot k_1 & q_3 \cdot k_2 & q_3 \cdot k_3 & q_3 \cdot k_4 \\ q_4 \cdot k_1 & q_4 \cdot k_2 & q_4 \cdot k_3 & q_4 \cdot k_4 \end{bmatrix}$

Step 3: Apply scaling and softmax
$A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) = \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ a_{21} & a_{22} & a_{23} & a_{24} \\ a_{31} & a_{32} & a_{33} & a_{34} \\ a_{41} & a_{42} & a_{43} & a_{44} \end{bmatrix}$

• ดังนั้น Each row รวมกันจะได้ 1 ไง

Step 4: Compute weighted sum of values
$Z = AV = \begin{bmatrix} z_1 \\ z_2 \\ z_3 \\ z_4 \end{bmatrix}$

where:
$z_1 = a_{11}v_1 + a_{12}v_2 + a_{13}v_3 + a_{14}v_4$
$z_2 = a_{21}v_1 + a_{22}v_2 + a_{23}v_3 + a_{24}v_4$
$z_3 = a_{31}v_1 + a_{32}v_2 + a_{33}v_3 + a_{34}v_4$
$z_4 = a_{41}v_1 + a_{42}v_2 + a_{43}v_3 + a_{44}v_4$

13.3.2 Key Components of the Transformer

The Transformer consists of two main components:

High-Level Architecture

┌─────────────┐  Z    ┌─────────────┐
│   Encoder   │  -->  │   Decoder   │
└─────────────┘       └─────────────┘
      ↑                      ↓
 x = "How are you?"   Y = "お元気ですか。"
   (Input)               (Output)

Encoder:

• Processes the input sequence
• Produces a matrix representation ($z$ or $\bar{x}$) of the input
• Example: English sentence "How are you?" → encoded representation

Decoder:

• Takes the encoded representation
• Generates output step by step (autoregressively)
• Example: Translates to Japanese "お元気ですか。"

Both encoder and decoder consist of a stack of identical layers.

Encoder Architecture

The encoder converts input tokens into contextualized representations.

Components

1. Input Embedding

• Converts input tokens into semantic vectors using embedding layers
• Each token becomes a dense vector representation

2. Positional Encoding

• Adds positional information to embeddings using sine and cosine functions
• Enables the model to track token order (since attention has no inherent notion of position)

Formula:
$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$
$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

where:

• $pos$ = position in sequence
• $i$ = dimension index
• $d_{model}$ = model dimension

Positional encoding is like adding timestamps to messages so the model knows which came first, even when processing them all at once.

3. Stack of Encoder Layers

The original Transformer uses N = 6 identical encoder layers. Each layer contains:

3.1 Multi-Head Self-Attention Mechanism

Purpose: Allows the model to focus on different parts of the input

Process:

1. Matrix multiplication of queries and keys to create score matrix
2. Scaling down scores for stable gradients: divide by $\sqrt{d_k}$
3. Apply softmax function to obtain attention weights
4. Multiply softmax weights by value vectors to produce output
5. Pass through linear layer

Multi-head aspect:

• Splits Q, K, V into multiple heads
• Each head processes independently
• Outputs are combined and fine-tuned
• Enriches model with diverse understanding

3.2 Normalization and Residual Connections

• Each sub-layer is followed by:
  • Layer normalization: Stabilizes training
  • Residual connection: Adds input directly to output

Formula:
$\text{Output} = \text{LayerNorm}(x + \text{Sublayer}(x))$

Benefits:

• Helps build deeper models
• Reduces vanishing gradient problem
• Enables better gradient flow

3.3 Feed-Forward Neural Network

• Point-wise feed-forward network applied to each position
• Two linear layers with ReLU activation in between

Formula:
$\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

• Applied identically to each position separately
• Followed by residual connection and normalization

4. Output of Encoder

• BOS = Beginning of sentence
  

• Final encoder layer outputs context-rich vectors representing the input

• These vectors serve as input to the decoder

• Guide decoder to focus on relevant words during generation

• By stacking N layers, model gains:

  • Deeper understanding
  • Different aspects of attention
  • Stronger predictive power

Decoder Architecture

The decoder generates output text sequences step by step.

Components

1. Output Embedding

• Similar to encoder input embedding
• Converts output tokens to vectors

2. Positional Encoding

• Same as encoder
• Tracks position in output sequence

3. Stack of Decoder Layers

The original Transformer uses N = 6 identical decoder layers. Each layer contains:

3.1 Masked Self-Attention Mechanism

• Similar to encoder self-attention but with masking
• Prevents positions from attending to subsequent positions
• Ensures predictions depend only on known outputs at previous positions
• Maintains autoregressive property (can't look into the future)

Why masking?

• During training, we have the full target sequence
• But at inference, we generate one token at a time
• Masking simulates the generation process during training

Masked attention is like taking a test where you can only see questions you've already answered, not future questions.

3.2 Encoder-Decoder Multi-Head Attention (Cross Attention)

• Queries (Q): Come from the decoder's previous layer
• Keys (K) and Values (V): Come from the encoder's output
  

Purpose:

• Aligns decoder's generation with encoded input sequence
• Allows decoder to focus on relevant parts of source input
• Creates connection between input and output sequences

Cross attention is like a translator reading the original text (encoder output) while writing the translation (decoder), constantly referring back to the source.

3.3 Feed-Forward Neural Network

• Similar to encoder's feed-forward network
• Two linear layers with ReLU activation
• Applied separately and identically to each position
• Followed by residual connection and normalization

Normalization and Residual Connections:

• Each sub-layer includes:
  • Layer normalization
  • Residual connection
• Helps with gradient flow and training stability

4. Linear Classifier and Softmax

• Final step of decoder
• Linear layer: Acts as classifier over vocabulary
• Softmax layer: Converts scores to probabilities

Formula:
$P(w) = \text{softmax}(W \cdot h_{\text{final}} + b)$

• Highest probability indicates predicted next word
• Process continues until end token is generated

Autoregressive Generation

Process:

1. Start with special start token (BOS - Beginning Of Sequence)
2. Decoder generates first output token
3. This token is added to decoder input
4. Decoder generates next token
5. Repeat until end token (EOS - End Of Sequence) is generated

Key property:

• Each step uses:
  • All previously generated tokens
  • Full encoder output (with cross attention)
  • But cannot see future tokens (due to masking)

Complete Transformer Workflow

Encoding Phase:

Input: "I want to buy a car EOS"
       ↓
  [Tokenization]
       ↓
  [Embedding + Positional Encoding]
       ↓
  [Encoder Layer 1]
       ↓
  [Encoder Layer 2]
       ↓
       ⋮
       ↓
  [Encoder Layer 6]
       ↓
  Encoded Representation (x̄₁, x̄₂, ..., x̄₇)

Decoding Phase:

Start: BOS token
       ↓
  [Decoder generates using encoded input]
       ↓
  Output: 車 (car)
       ↓
  [Feed back: BOS, 車]
       ↓
  Output: を (particle)
       ↓
  [Feed back: BOS, 車, を]
       ↓
       ⋮
       ↓
  Output: です (polite ending)
       ↓
  Output: EOS (stop)

Final output: "車を買いたいです" (I want to buy a car)

Key Advantages of Transformers

1. Parallelization

   • Can process entire sequence simultaneously
   • Much faster training than sequential RNNs
   • Better GPU utilization

2. Long-Range Dependencies

   • Direct connections between any two positions
   • No information bottleneck
   • Better at capturing long-distance relationships

3. No Vanishing Gradients

   • Direct paths for gradient flow
   • Each position has constant path length to any other position
   • More stable training

4. Interpretability

   • Attention weights can be visualized
   • Shows what the model focuses on
   • Helps understand model decisions

The Transformer is like having a team of experts (attention heads) who can all look at the entire document simultaneously and discuss it together, rather than passing notes one by one (like RNNs).

Summary Comparison

References

• von Platen, Patrick. Transformers-based Encoder-Decoder Models. Hugging Face Blog. https://huggingface.co/blog/encoder-decoder

• Vaswani et al. (2017). Attention is All You Need. NeurIPS.

Additional Notes

Key Takeaways:

1. RNNs introduced the concept of sequential processing with memory but suffer from vanishing gradients

2. LSTMs solved the vanishing gradient problem through:

   • Gating mechanisms (forget, input, output gates)
   • Cell state with additive updates
   • Ability to learn long-term dependencies

3. Transformers revolutionized sequence modeling by:

   • Eliminating recurrence entirely
   • Using self-attention to capture dependencies
   • Enabling parallel processing
   • Achieving state-of-the-art results in many tasks

4. Self-attention is the key innovation that allows:

   • Each position to attend to all other positions
   • Capturing global dependencies efficiently
   • Better handling of long-range relationships

Practical Considerations:

• Use RNNs for: Simple sequence tasks, resource-constrained environments
• Use LSTMs for: Time series, speech recognition, moderate-length sequences
• Use Transformers for: NLP tasks, machine translation, long documents, when computational resources are available

Modern deep learning has largely moved toward Transformer architectures for sequence modeling tasks, but understanding RNNs and LSTMs provides crucial foundations for understanding how we got here and when simpler models might still be appropriate.