End-to-end Neural Networks for NLP¶

Word Embeddings¶

If everything is going well, you have used PyTorch to make a classifier based on logistic regression. Also, you are probably mildly annoyed by the fact that the accuracy was essentially the same as what we had with our previous version using scikit-learn, with the added difficulty that we had to write a loop all by ourselves.

Now it is time to do some math and review our models.

Some underlying math of the bag-of-words model¶

In our current model, we represent each word by a line-vector $x_n$ with $v$ dimensions, where $v$ is the length of our vocabulary. Therefore, a document with $n$ words can be represented by a matrix of words within a document $X^{(d)}$ as:

$$ X^{(d)} = \begin{bmatrix} x^{(d)} _{1,1} & x^{(d)}_{1,2} & \cdots & x^{(d)}_{1,d} \\ x^{(d)}_{2,1} & x^{(d)}_{2,2} & \cdots & x^{(d)}_{2,d} \\ \vdots & \vdots & \ddots & \vdots \\ x^{(d)}_{N,1} & x^{(d)}_{N,2} & \cdots & x^{(d)}_{N,d} \\ \end{bmatrix} $$

Then, we essentially sum all elements of the matrix to get a count. This maps the sequence of words to a single vector $x$ that represents our document:

$$ x^\prime_j = \sum _{i=1}^n x^{(d)}_{i,j} $$

This is the same as pre-multiplying $W$ by a line-matrix of ones:

$$ x^\prime_j = [1, 1, \cdots, 1] \begin{bmatrix} x^{(d)}_1 \\ x^{(d)}_2 \\ \cdots \\ x^{(d)}_n \end{bmatrix} $$

Last, if we want to binarize this vector, we simply apply an element-wise non-linearity such as:

$$ x = f(x^\prime), $$ where $f(x)=1$ if $x>0$ and $f(x)=0$ otherwise.

Essentially, what we must do is:

1. Get a representation for each word
2. Combine the word representations for a document into a single representation
3. Use some non-linearity to modulate the representation as needed

Our solutions, so far, are:

1. Use a one-hot encoding
2. Use a simple summation
3. Use a step-function with a suitable threshold

The reason we have these solutions is that they somehow make sense. But now let's raise a question: are they the best solutions?

Problems with the sparse representation¶

The one-hot encoding for words is easy to understand, but, at the same time, has some problems.

The first, and more obvious, is that the dimensionality of the representation grows when the vocabulary grows. This is a problem because the amount of required data to train a system is typically a function of its dimension - hence, the more words we have, the more data we need. The vocabulary typically grows when we add more texts to the dataset, so this actually means: the more data we have, the more data we need.

The second is that our data matrix starts getting too big. Maybe it stops fitting memory - and this is because we have so many columns! Maybe we could do something about this?

The third, and less obvious, is that the distance between words is the same, regardless of their meaning. Hence, when we learn about dungeons, our system learns nothing about dragons. In fact, classifiers based on one-hot bag-of-words have no internal representation indicating that dungeons and dragons go more or less together.

Dense representations¶

So, let's do something else. Let's say that each word is now going to be represented as a vector in some $\mathbb{R}^N$ space. We are free to choose $N$. This representation is called embedding.

Embeddings are dense representations, meaning that all dimensions are important. The name dense opposes to the sparse idea of the one-hot encoding in which only one dimension is important per word, and only a few dimensions are important per document.

What we want here is to have all words being represented in locations of $\mathbb{R}^N$ that reflect their meaning. But, what is meaning?

This is a complicated question, which we will approach during this course.

For now, let's go on and code a little.

Coding with embeddings¶

To map words to their embeddings, we use a data structure that is very similar to a dictionary. The structure is wrapped in an embedding layer in Pytorch.

It receives a sequence of integers as input. Each of these integer corresponds to a token in the vocabulary. For now, we will assume that each token corresponds to a word. Then, it yields a sequence of vectors for each document:

Note that, in the code above, we yielded a batch of three texts to the embedding layer. The batch has a length of four. Then, the embedding layer returns a tensor with tridimensional shape. The shape $3 \times 4 \times 2$ means:

• We have 3 documents
• Each document has 4 tokens
• Each word is encoded in 2 dimensions

Strategies for tokenization¶

Just some minutes ago, we discussed that a token corresponds to a word. In fact, a token is a piece of the text that conveys some information - and this could be a word. Also, we have used n-grams as tokens when we wanted to consider the sequence of words.

Now, let's remember. If we had around, maybe, $10^4$ words in our vocabulary, then we would have potentially $10^8$ bi-grams, and $10^{12}$ potential tri-grams. This is because we are aggreting more and more possible symbols.

However, we could go on the opposite direction here.

If we had the words "working", "works", "walking", "walks", "calling", and "calls", we could easily tokenize them into 6 tokens, each corresponding to full words.

However, we could separate the suffixes and the stems. In this case, we could have tokens for: "work", "walk", "call", "s", and "ing". See: now we have 5 tokens instead of 6.

This is generally called a "subword" tokenization strategy. One of the most used algorithms for such is called Sentence Piece Tokenizer. It works by building a vocabulary like this:

1. Assume each character is a separate token
2. Create a new token that merges the most common token sequence into a separate token
3. Repeat the merging until we reach a reasonable vocabulary size

A nice advantage of using this method is that it is less likely to find unknown tokens - afterall, we start from a limited vocabulary.

Also, we are not going to build a tokenizer from scratch: instead, we are going to use a ready-made one. If you want to read all about it, refer to:

Kudo, T., and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018)

Example code for training a Sentence Piece tokenizer:¶

Example code to test the trained tokenizer:¶

Zero-padding and truncation: keeping sentences the same length¶

If you have been paying attention, you probably realized that, in any language, sentences have different lengths. However, PyTorch works with tensors, hence the inputs for an embedding layer should have the same length.

If we find long sentences, we could just truncate them so a desired length.

However, if we find a short sentence, the procedure is different. Usually, we inser a special token called "padding" so that the sentence artificially becomes our desired length:

How far have we gone?¶

So far, we have tokenized a batch of $b$ texts and have cropped/padded them to length $l$. After that, we calculated embeddings of dimension $d$ for our words. If everything went well, we now have embeddings $E \in \mathbb{R}^{b \times l \times d}$:

However, if we are going to classify our text using Logistic Regression, we need a single vector to represent the whole sequence, that is, we need to convert:

$$ E \in \mathbb{R}^{b \times l \times d} $$

to

$$ X \in \mathbb{R}^{b \times d} $$

In other words, we need to summarize the word-level embeddings into a single document-level embedding.

The easiest way to do so is to calculate the mean of the embeddings, then we can proceed to classification:

Overall, we get 4-step architecture (tokenization, embedding, summarization, classification) that is, in fact, very similar to the one we had when we were using the vectorizer-classifier pipelines with the bag-of-words approach:

Making a pipeline with PyTorch¶

Remember that in Scikit-Learn we had pipelines? In PyTorch, the equivalent procedure is to make a class. This class should inherit from nn.Module and we have to define, at least, the methods:

• __init__, which will initialize the class, instantiate all blocks that have parameters, and so on;
• forward, which is called when the object is called (it bothers me that the more pythonic __call__ was not chosen for this...). This method implements the actual workings of the pipeline. See this example:

Evaluating the model¶

We can convert our

Some steps on optimization¶

You may have noticed that we are using the MSE loss in the code above. This is definitely not optimal, because reductions in MSE+ does not necessarily correspond to increases in accuracy. Instead, we could use the cross-entropy loss.

Cross-entropy loss¶

The idea of the cross-entropy loss begins with the negative log likelihood (NLL). Is is usually written as:

$$ \text{NLL} = - \sum_i \log{P(C_i = c_i|X_i)}, $$

where $C_i=c_i$ is the event that the predicted class $C_i$ for the $i$-th input $X_i$ is the correct class $c_i$.

Note that NLL measures the likelihoood that the correct class will be predicted for each item $i$ in the dataset. Therefore, maximizing the negative log likelihood means minimizing the probability that a wrong class will be predicted. Importantly, this is very different from minimizing the MSE!

We know that $P(C_i = c_i|X_i)$, in the logistic regressor, is the result of applying the sigmoid (logistic) function to the logits $z_i$. I will call $y_i = P(C_i = c_i|X_i)$ to make reading easier:

$$ y_i = \sigma(z_i) = \frac{1}{1+e^{-z_i}} $$

We can substitute $y_i$ in the NLL equation to get:

$$ \text{NLL} = - \sum_i \log{y_i} = -\sum_i \log{\frac{1}{1+e^{-z_i}}}, $$

Remember that the log of a division is the subtraction of logs. Hence, we get:

$$ \text{NLL} = -\sum_i \log{1} - \log{1+e^{-z_i}} = -\sum_i 0 - \log{1+e^{-z_i}} = \sum_i \log{(1+e^{-z_i})}. $$

Also, we are using SGD as the optimizer. The SGD optimizer uses a fixed-step optimization over the parameter space. A more modern approach is the Adam (Adaptive Momentum) optimizer, which adapts the step size so that smoother regions of the parameter space are swept more quickly, and rougher regions automatically lead to smaller steps.

Then, the code becomes this:

Visualizing model¶

Accuracy is an important feature with regards to checking if your model works. However, it is not all. We like the idea of checking what is happening to our model. In special: where are our documents being projected in the embedding space? Also, are we overfitting?

Let's change our model and the training loop a bit so we can track everything:

Practice¶

Part 1 of 2: Recall concepts¶

Before proceeding, make sure you understand the role of:

Part 2 of 2: Improve performance of our classifier¶

You may have noticed that we used many complicated techniques and ultimately got subpar performance. Let's work with that.

Make yourself a clean notebook in which you can clearly control:

1. The number of tokens in the vocabulary
2. The dataset size
3. The embedding dimension

Make experiments showing how each of these parameters impact the final result. Use accuracy, but also learning curves over time and embedding plots to support your observations.

Make a slide deck showing your experiments. For such, use:

1. The initial slide has figures summarizing the conclusions and results
2. The following slides summarize each of the performed experiments

Use at most 5 slides.

Extension: sequence models¶

If you want some other challenges, upgrade your model: use sequence models instead of a simple mean to summarize your document embeddings. In Pytorch, you might want to try SimpleRNNs, GRUs and LSTMs. Study how each one of them work and why should they improve results.