Word2Vec

Word2Vec is a technique introduced by researchers at Google in 2013 that converts words into numerical vectors (lists of numbers) such that words with similar meanings end up with similar vectors. It was one of the first methods to demonstrate that AI could capture the meaning of words in a mathematically useful way.

The core idea

Computers cannot natively understand words — they work with numbers. Word2Vec solves this by representing each word as a vector in a high-dimensional space (typically 100-300 dimensions). Words that appear in similar contexts during training end up close together in this space.

For example, the vectors for "king" and "queen" are close to each other, as are "Paris" and "France." More remarkably, the mathematical relationships between vectors capture semantic relationships: vector("king") - vector("man") + vector("woman") ≈ vector("queen").

How Word2Vec learns

Word2Vec uses one of two training approaches:

• CBOW (Continuous Bag of Words): Predicts a word from its surrounding context. Given "the cat sat on the ___", predict "mat."
• Skip-gram: Predicts surrounding context from a word. Given "mat," predict that "the," "cat," "sat," "on," and "the" are likely to appear nearby.

Both approaches learn by processing billions of words and adjusting the vectors so that words appearing in similar contexts get similar representations.

Why Word2Vec was revolutionary

Before Word2Vec, most natural language processing treated words as isolated symbols with no inherent relationship to each other. "Cat" and "kitten" were as different as "cat" and "quantum." Word2Vec demonstrated that unsupervised learning on raw text could automatically discover meaningful semantic relationships, without any human-provided definitions.

From Word2Vec to modern embeddings

Word2Vec had a significant limitation: each word got one vector regardless of context. The word "bank" had the same vector whether it referred to a financial institution or a riverbank. This limitation was addressed by contextual embeddings like ELMo, BERT, and eventually the embeddings used in modern large language models, where the same word gets different representations depending on its context.

Practical applications

Word2Vec and its successors are used in search engines, recommendation systems, document classification, and anywhere that measuring the similarity between pieces of text is useful. The concept of representing meaning as vectors — embedding — has become one of the most fundamental techniques in all of AI.