What is Word2Vec

How It Works

Word2Vec trains a shallow two-layer neural network on large text corpora using one of two architectures: Continuous Bag of Words (CBOW), which predicts a target word from surrounding context words, or Skip-gram, which predicts context words given a target word. The learned weight matrix becomes the word embedding table, where each word maps to a dense vector that encodes semantic meaning.

Technical Details

Word2Vec typically produces 100-300 dimensional vectors trained on sliding windows of 5-10 words. It uses negative sampling or hierarchical softmax to make training efficient on large vocabularies. The resulting vectors exhibit linear algebraic properties, such as king - man + woman = queen, demonstrating that the model captures relational semantics in vector space.

Best Practices

• Use Skip-gram for rare words and smaller datasets, CBOW for frequent words and larger corpora
• Train on domain-specific text for specialized applications rather than relying solely on pretrained vectors
• Set vector dimensionality between 100-300 based on vocabulary size and task complexity
• Preprocess text carefully with consistent tokenization and lowercasing before training

Common Pitfalls

• Assuming Word2Vec captures sentence-level meaning when it only encodes word-level semantics
• Using generic pretrained vectors for domain-specific tasks without fine-tuning
• Ignoring out-of-vocabulary words that have no learned embedding
• Training on insufficient data which produces low-quality embeddings

Advanced Tips

• Combine Word2Vec with subword models like FastText to handle morphological variations
• Use Word2Vec embeddings as initialization for downstream neural network layers
• Leverage negative sampling with 5-20 negatives for optimal training speed and quality
• Evaluate embedding quality with analogy tasks and downstream task performance, not just similarity