1. Introduction to RNNs

Recurrent Neural Networks (RNNs) are specialized neural networks designed to work with sequential data like text, speech, or time-series. Unlike standard neural networks, which process inputs independently, RNNs maintain a "memory" of past inputs using hidden states. This makes them ideal for tasks where context matters, such as predicting the next word in a sentence or understanding a sentence’s sentiment.

Why RNNs?

• Temporal Dependencies: They capture relationships between elements in a sequence (e.g., "I am hungry" vs. "I am 25").
• Flexible Input/Output Lengths: Can handle variable-length sequences (e.g., translating sentences of different lengths).
• Simplicity: Shared weights across time steps reduce model complexity.

2. RNN Architecture

Imagine reading a book page by page while trying to remember the story. RNNs work similarly: they process sequences step-by-step while carrying forward a "summary" of past information.

Core Mechanism

• Hidden State: A vector representing the network’s memory at each step.
• Recurrent Cell: At each time step, the cell:
  1. Takes the current input (e.g., a word).
  2. Combines it with the previous hidden state.
  3. Updates the hidden state for the next step.

Example:

For the sentence "The cat sat on the mat", the RNN processes each word while updating its hidden state:

"the" → updates hidden state → "cat" → updates hidden state → ... → "mat"

Unrolling the RNN

Visualize the RNN as a chain of repeated cells:

Input: [x₁] → [Cell] → [h₁] → [x₂] → [Cell] → [h₂] → ...

Each cell shares the same weights, ensuring consistency across time steps.