Word Embedding - OpenML's AI Blogs

Introduction#

Those who ever used Siri, Google Assistant, Alexa, Google Translate, or even smartphone keyboard with next-word prediction has already been benefited from this idea that has become central to Natural Language Processing models. There has been quite a development over the last couple of decades in using embeddings for neural models (Recent developments include contextualized word embeddings leading to cutting-edge models like BERT and GPT2).

Here is trained word-vector examples (also called word embeddings):

1
[
2
    0.50451, 0.68607, -0.59517, -0.022801, 0.60046, -0.13498, -0.08813, 0.47377, -0.61798,
3
    -0.31012, -0.076666, 1.493, -0.034189, -0.98173, 0.68229, 0.81722, -0.51874, -0.31503,
4
    -0.55809, 0.66421, 0.1961, -0.13495, -0.11476, -0.30344, 0.41177, -2.223, -1.0756,
5
    -1.0783, -0.34354, 0.33505, 1.9927, -0.04234, -0.64319, 0.71125, 0.49159, 0.16754,
6
    0.34344, -0.25663, -0.8523, 0.1661, 0.40102, 1.1685, -1.0137, -0.21585, -0.15155,
7
    0.78321, -0.91241, -1.6106, -0.64426, -0.51042
8
]

It’s a list of 50 numbers. We can’t tell much by looking at the values. But let’s visualize it a bit so that we could compare it with other word vectors. First let’s put all these numbers in one row:

Next let’s color code the cells based on their values (red if they’re close to 2, white if they’re close to 0, blue if they’re close to -2):

We proceed by ignoring the numbers and only looking at the colors to indicate the values of the cells. Let’s now contrast “King” against other words:

See how “Man” and “Woman” are much more similar to each other than either of them is to “king”? This tells us something. These vector representations capture quite a bit of the information/meaning/associations of these words.

Here’s another list of examples (compare by vertically scanning the columns looking for columns with similar colors):

A few things to point out:

There’s a straight red column through all of these different words. They’re similar along that dimension (and we don’t know what each dimensions codes for)
We can see how “woman” and “girl” are similar to each other in a lot of places. The same with “man” and “boy”
“boy” and “girl” also have places where they are similar to each other, but different from “woman” or “man”. Could these be coding for a vague conception of youth? possible.
All but the last word are words representing people. I added an object (water) to show the differences between categories. We can, for example, see that blue column going all the way down and stopping before the embedding for “water”.
There are clear places where “king” and “queen” are similar to each other and distinct from all the others. Could these be coding for a vague concept of royalty?

Analogies
The famous examples that show an incredible property of embeddings is the concept of analogies. We can add and subtract word embeddings and arrive at interesting results. The most famous example is the formula: “king” - “man” + “woman”. Using the Gensim library in python, we can add and subtract word vectors, and it would find the most similar words to the resulting vector. The image shows a list of the most similar words, each with its cosine similarity.

As we add and subtract word vectors, we would find the most similar words to the resulting vector. The image shows a list of the most similar words, each with its cosine similarity.

We can visualize this analogy as we did previously:

The resulting vector from “king-man+woman” doesn’t exactly equal “queen”, but “queen” would be the closest word to it from this example of 400,000 word embeddings.

Now that we’ve looked at trained word embeddings, let’s learn more about the training process. But before we get to word2vec, we need to look at a conceptual parent of word embeddings: the neural language model.

Language Modeling#

If one wanted to give an example of an NLP application, one of the best examples would be the next-word prediction feature of a smartphone keyboard. It’s a feature that billions of people use hundreds of times every day.

Next-word prediction is a task that can be addressed by a language model. A language model can take a list of words (let’s say two words), and attempt to predict the word that follows them.

In the screenshot above, we can think of the model as one that took in these two words (“thou” and “shalt”) and returned a list of suggestions (“not” being the one with the highest probability):

We can think of the model as looking like this black box:

In practice, however, the model doesn’t output only one word. It actually outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words). The keyboard application then has to find the words with the highest scores, and present those to the user.

The output of the neural language model is a probability score for all the words the model knows. We are referring to the probability as a percentage here, but 40% would actually be represented as 0.4 in the output vector.

After being trained, early neural language models (Bengio 2003) would calculate a prediction in 3 steps:

The first step is the most relevant for us as we discuss embeddings. One of the results of the training process was this matrix that contains an embedding for each word in our vocabulary. During prediction time, we just look up the embeddings of the input word, and use them to calculate the prediction:

Let’s now turn to the training process to learn more about how this embedding matrix was developed.

Language Model Training#

Language models have a huge advantage over most other machine learning models. That advantage is that we are able to train them on running text – which we have an abundance of. Think of all the books, articles, Wikipedia content, and other forms of text data we have lying around. Contrast this with a lot of other machine learning models which need hand-crafted features and specially-collected data.

We get embeddings of words by looking at which other words they tend to appear next to. The mechanics of that is that

We get a lot of text data (say, all Wikipedia articles, for example). then
We have a window (say, of three words) that we slide against all of that text.
The sliding window generates training samples for our model

As this window slides against the text, we (virtually) generate a dataset that we use to train a model. To look exactly at how that’s done, let’s see how the sliding window processes this phrase:

When we start, the window is on the first three words of the sentence:

We take the first two words to be features, and the third word to be a label:

We now have generated the first sample in the dataset we can later use to train a language model.

We then slide our window to the next position and create a second sample:

The second example is now generated.

Pretty soon we have a larger dataset of which words tend to appear after different pairs of words:

The example above is trying to predict the target word by looking at two words before it, we can also look at two words after it. Another architecture that also tended to show great results does things a little differently and is the one we will be using as part of our following discussion: instead of guessing a word based on its context (the words before or maybe even after it), this architecture tries to guess neighboring words within certain radius using the current word. It is called skipgram, which has a window sliding across the texts like this:

The word in the green slot would be the input(or current) word, each pink box would be a possible output within its radius. In this case, the radius is 2 (words)

A single snapshot of the sliding window creates four separate samples in our training dataset:

We then iteratively slide our window to the next positions… A couple of positions later, we have a lot more examples:

Now that we have our skipgram training dataset (shown in the image above) that we extracted from existing running text, let’s glance at how we use it to train a basic neural language model that predicts the neighboring word.

We start with the first sample in our dataset. We grab the feature and feed to the untrained model asking it to predict an appropriate neighboring word.

The model conducts the three steps and outputs a prediction vector (with a probability assigned to each word in its vocabulary). Since the model is untrained, it’s prediction is sure to be a wild guess at this stage. But that’s okay. We know what word it should have guessed – the label/output cell in the row we’re currently using to train the model:

How far off was the model? We could choose to subtract the two vectors resulting in an error vector:

This error vector can now be used to update the model so the next time, it’s a little more likely to guess thou when it gets not as input.

And that concludes the first step of the training. We proceed to do the same process with the next sample in our dataset, and then the next, until we’ve covered all the samples in the dataset. That concludes one epoch of training. We do it over again for a number of epochs, and then we’d have our trained model and we can extract the embedding matrix from it and use it for any other application.

TIP
One training step processes one sample of dataset while one epoch iterates through the entire dataset once

While this extends our understanding of the process, it’s still not how word2vec is actually trained. We’re missing a couple of key ideas:

Cosine similarity
Negative samples

Cosine Similarity#

Recall the 3 steps of how this neural language model calculates its prediction:

The 3rd step (Project to output vocabulary) is very expensive from a computational point of view - especially knowing that we will do it once for every training sample in our dataset (easily tens of millions of times). We need to do something to improve performance, which is missing from the basic training strategy introduced above.

One solution for boosting the performance is to split our target into 2 steps:

Generate high-quality word embeddings (Don’t worry about next-word prediction).
Use these high-quality embeddings to train a language model (to do next-word prediction).

We will be focusing on step 1 as we’re focusing on embeddings. To generate high-quality embeddings using a high-performance model, we can switch the model’s task from predicting a neighboring wordto taking the input and output word, and outputing a score indicating if they’re neighbors or not (0 for “not neighbors”, 1 for “neighbors”), i.e.:

This simple switch changes the model we need from a neural network, to a logistic regression model - thus it becomes much simpler and much faster to calculate.

This switch requires we switch the structure of our dataset – the label is now a new column with values 0 or 1. They will be all 1 since all the words we added are neighbors.

This can now be computed at blazing speed – processing millions of examples in minutes. But there’s one loophole we need to close. If all of our examples are positive (target: 1), we open ourselves to the possibility of a smartass model that always returns 1 - achieving 100% accuracy, but learning nothing and generating garbage embeddings.

Negative Samples#

To address this, we need to introduce negative samples to our dataset - samples of words that are not neighbors. Our model needs to return 0 for those samples. Now that’s a challenge that the model has to work hard to solve - but still at blazing fast speed.

But what do we fill in as output words? We randomly sample words from our vocabulary

This idea is inspired by Noise-contrastive estimation. We are contrasting the actual signal (positive examples of neighboring words) with noise (randomly selected words that are not neighbors). This leads to a great tradeoff of computational and statistical efficiency.

We have now covered two of the central ideas in word2vec: as a pair, they’re called skipgram with negative sampling:

Word2vec Training Process#

Now that we’ve established the two central ideas of skipgram and negative sampling, we can proceed to look closer at the actual word2vec training process.

Before the training process starts, we pre-process the text we’re training the model against. In this step, we determine the size of our vocabulary (we’ll call this vocab_size, think of it as, say, 10,000) and which words belong to it.

At the start of the training phase, we create two matrices – an Embedding matrix and a Context matrix. These two matrices have an embedding for each word in our vocabulary (So vocab_size is one of their dimensions). The second dimension is how long we want each embedding to be (embedding_size – 300 is a common value, but we’ve looked at an example of 50 earlier in our discussion here).

At the start of the training process, we initialize these matrices with random values. Then we start the training process. In each training step, we take one positive example and its associated negative examples. Let’s take our first-step data (highlighted in light blue rows):

Now we have 4 words: the input word not and output/context words: thou (the actual neighbor), aaron, and taco (the negative examples). We proceed to look up their embeddings - for the input word, we look in the Embedding matrix. For the context words, we look in the Context matrix (even though both matrices have an embedding for every word in our vocabulary).

Then, we take the dot product of the input embedding with each of the context embeddings. In each case, that would result in a number, that number indicates the similarity of the input and context embeddings

Now we need a way to turn these scores into something that looks like probabilities - we need them to all be positive and have values between zero and one. This is a great task for sigmoid, the logistic operation.

And we can now treat the output of the sigmoid operations as the model’s output for these examples. We can see that taco has the highest score and aaron still has the lowest score both before and after the sigmoid operations.

Now that the untrained model has made a prediction, and seeing as though we have an actual target label to compare against, let’s calculate how much error is in the model’s prediction. To do that, we just subtract the sigmoid scores from the target labels (error = target - sigmoid_scores).

Here comes the “learning” part of “machine learning”. We can now use this error score to adjust the embeddings of not, thou, aaron, and taco so that the next time we make this calculation, the result would be closer to the target scores.

This concludes the training step. We emerge from it with slightly better embeddings for the words involved in this step (not, thou, aaron, and taco). We now proceed to our next step (the next positive sample and its associated negative samples) and do the same process again.

The embeddings continue to be improved while we cycle through our entire dataset for a number of times. We can then stop the training process, discard the Context matrix, and use the Embeddings matrix as our pre-trained embeddings for the next task.

Window Size and Number of Negative Samples
Two key hyperparameters in the word2vec training process are the window size and the number of negative samples.
Different tasks are served better by different window sizes. One heuristic is that smaller window sizes (2-15) lead to embeddings where high similarity scores between two embeddings indicates that the words are interchangeable (notice that antonyms are often interchangable if we’re only looking at their surrounding words – e.g. good and bad often appear in similar contexts). Larger window sizes (15-50, or even more) lead to embeddings where similarity is more indicative of relatedness of the words.
The number of negative samples is another factor of the training process. The original paper prescribes 5-20 as being a good number of negative samples. It also states that 2-5 seems to be enough when you have a large enough dataset.