Beyond Atomic Words: Mastering Subword Embeddings with FastText
Beyond Atomic Words: Mastering Subword Embeddings with FastText
In the early days of Word2Vec and GloVe, the NLP world treated words as atomic units. If your model encountered the word "apple," it learned a vector for it. If it then encountered "apples," it treated it as a completely new, unrelated entity.
This "atomic" approach creates two massive bottlenecks:
- The OOV (Out-of-Vocabulary) Problem: If a word wasn't in the training set, the model was blind to it.
- Morphological Blindness: The model failed to realize that "running," "runner," and "ran" share a common semantic root.
Enter FastText, an extension of the Skip-gram architecture that treats words not as atoms, but as bags of character n-grams.
🧠 The Core Intuition: Words as Composites
The fundamental thesis of FastText is simple: The meaning of a word is partially contained in its structure.
Instead of representing a word $w$ as a single vector $\mathbf{u}_w$, FastText represents it as the sum of its character n-grams. For example, if we use n-grams of length 3 to 6, the word <where> is decomposed into:
<wh,whe,her,ere,re>- And the full word
<where>itself.
By sharing vectors across words that share n-grams (e.g., "apple" and "apples" both share <appl), the model can infer the meaning of rare words and even generate vectors for words it has never seen before.
The Architecture at a Glance
📐 The Mathematical Framework
FastText optimizes a modified version of the Skip-gram objective.
1. The Scoring Function
The representation of a target word $w_t$ is the sum of its own vector and the vectors of its constituent n-grams $G_{w_t}$: $$s(w_t, w_c) = \left( \mathbf{u}{w_t} + \sum{g \in G_{w_t}} \mathbf{z}g \right)^\top \mathbf{v}{w_c}$$ Where:
- $\mathbf{u}_{w_t}$: Vector for the whole word.
- $\mathbf{z}_g$: Vector for the n-gram $g$.
- $\mathbf{v}_{w_c}$: Vector for the context word $w_c$.
2. The Objective Function
The model uses Negative Sampling to avoid calculating the softmax over the entire vocabulary. The goal is to maximize the probability of the actual context word while minimizing the probability of $N$ random noise words:
$$\mathcal{L} = \sum_{t=1}^{T} \sum_{c \in C_t} \left( -\log \sigma(s(w_t, w_c)) - \sum_{n \in N_{t,c}} \log \sigma(-s(w_t, w_n)) \right)$$
💻 Production-Ready Implementation
Below is a PyTorch implementation demonstrating the "magic" of FastText: the ability to handle OOV words.
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import random
class FastTextModel(nn.Module):
def __init__(self, vocab_size, ngram_hash_size, embedding_dim):
super(FastTextModel, self).__init__()
self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
self.ngram_embeddings = nn.Embedding(ngram_hash_size, embedding_dim)
self.context_embeddings = nn.Embedding(vocab_size, embedding_dim)
self.embedding_dim = embedding_dim
def forward(self, word_idx, ngram_indices):
# Word representation = Vector(word) + Sum(Vector(n-grams))
word_vec = self.word_embeddings(word_idx)
ngram_vecs = self.ngram_embeddings(ngram_indices)
return word_vec + torch.sum(ngram_vecs, dim=1)
def get_ngrams(word, min_n=3, max_n=6):
word = f"<{word}>"
ngrams = []
for n in range(min_n, max_n + 1):
for i in range(len(word) - n + 1):
ngrams.append(word[i:i+n])
ngrams.append(f"<{word[1:-1]}>")
return ngrams
def fnv1a_hash(string, hash_size):
"""Maps n-grams to a fixed size bucket to manage memory."""
h = 2166136261
for char in string:
h ^= ord(char)
h = (h * 16777619) & 0xFFFFFFFF
return h % hash_size
class FastTextTrainer:
def __init__(self, corpus, embedding_dim=50, ngram_hash_size=10000):
self.embedding_dim = embedding_dim
self.ngram_hash_size = ngram_hash_size
self.words = sorted(list(set(corpus)))
self.word2idx = {w: i for i, w in enumerate(self.words)}
self.vocab_size = len(self.words)
# Pre-calculate n-gram hashes
self.word_ngrams = {w: [fnv1a_hash(g, ngram_hash_size) for g in get_ngrams(w)] for w in self.words}
self.model = FastTextModel(self.vocab_size, ngram_hash_size, embedding_dim)
self.criterion = nn.BCEWithLogitsLoss()
self.optimizer = optim.Adam(self.model.parameters(), lr=0.001)
def get_word_vector(self, word):
"""The OOV Magic: Compute vectors for unseen words using subwords."""
self.model.eval()
with torch.no_grad():
# Handle OOV: Use zero vector for the word part, but still use n-grams!
w_idx = torch.tensor([self.word2idx[word]]) if word in self.word2idx else None
word_vec = self.model.word_embeddings(w_idx) if w_idx is not None else torch.zeros(1, self.embedding_dim)
ngs = get_ngrams(word)
ng_indices = torch.tensor([[fnv1a_hash(g, self.ngram_hash_size) for g in ngs]])
ngram_vecs = self.model.ngram_embeddings(ng_indices)
return (word_vec + torch.sum(ngram_vecs, dim=1)).numpy()
# Example Usage
corpus = "the apple is red the apples are red i like apple-like taste".split()
trainer = FastTextTrainer(corpus)
# Imagine 'apple-ish' was never in the training set
oov_vector = trainer.get_word_vector("apple-ish")
print(f"Vector for OOV word 'apple-ish' generated successfully!")🚀 Key Takeaways for Engineers
Why use FastText over Word2Vec?
| Feature | Word2Vec | FastText |
|---|---|---|
| Unit of Analysis | Whole Word | Character n-grams |
| OOV Words | Returns Error/Unknown | Infers vector from subwords |
| Morphology | Ignores prefixes/suffixes | Captures root meanings |
| Memory | Lower | Higher (due to n-gram table) |
Implementation Tips
- Hashing is Critical: To prevent the n-gram table from exploding in size, use a hashing function (like FNV-1a) to map n-grams into a fixed number of buckets.
- Boundary Symbols: Always wrap words in
<and>symbols. This ensures the model can distinguish between a prefix (e.g.,un-) and a sequence of characters in the middle of a word. - Language Choice: FastText is a game-changer for morphologically rich languages (like Turkish, Finnish, or German) where a single root can have dozens of variations.
Final Thoughts
FastText bridges the gap between word-level and character-level embeddings. By treating words as a collection of sub-components, it provides a robust solution to the OOV problem and allows models to generalize across related words, making it a foundational tool for any production NLP pipeline.