Multi-layer perceptrons¶

So far, we have been using Logistic Regression for all our classification needs. Logistic regression is very similar to linear regression, except for that $\sigma(z)$ in the end - it is essentially a linear projection and a choice between the "positive" and the "negative" sides of the projection surface.

Also, we have seen that we can choose to project our data $X$ into an intermediate representation $z$ so that $z$ has more than one dimension. We can use that for multi-class classification.

Now, we are going to view the effects of mapping the intermediate projection $z$ to another intermediate projection (let's call it $z_2$). As we will see, increasing the dimensionality of each representation $z_i$ and the number of intermediate projections has the effect of creating intermediate regions in which we can apply linear transformations separately. If you want a theoretical reference for such, refer to: Thiago Serra, Christian Tjandraatmadja, Srikumar Ramalingam Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4558-4566, 2018..

In the examples shown here, we will start with the use of Linear Regression to show some limitations of this approach. These animated examples work like this:

1. We define a random dataset $X$
2. We define a target $y$ by applying some function over $X$
3. We initialize a prediction model with an identity function (that is, our weights are such that )
4. We train the prediction model to predict $\hat{y} = f(X)$ using gradient descent, and store $\hat{y}_t$ for each iteration $t$
5. We make an animation of all $\hat{y}_t$ so we can see what happens to our predictions.

A simple dataset: a rotation + translation¶

A linear regression is capable to find the correct rotation and translation of a dataset. This is because rotations and translations can be immediately expressed by the linear prediction equation $y = xw^t + b$ - in this case, the weight matrix $w$ can be constructed from a rotation, and $b$ corresponnds to the translation:

¶

A more complicated dataset: linear by parts¶

Now, let's get a more complicated dataset. Now, $X$ (our input) will have three different clusters, and we will apply a different linear transform in each cluster.

When we try to approximate this using a linear layer, we obviously can't. This is due to our data being more complicated than the model - or, in other words, our model is not expressive enough to model our data. In the animation, we clearly see that the linear layer can only apply the same transform to all points in the input vector space, hence they all "bend" in the same way.

Our model is unable, for example, to model the different cluster variances generated by the different multiplications applied when we generated each part of $y$.

Exercise

Multi Layer Perceptron (MLP) models¶

A possible upgrade to the linear model is the MLP model. The MLP model is:

$$ \hat{y} = f(xw_1^T+b_1)w_2^t+b_2, $$ where $f$ is the Rectifying Linear Unit (ReLU) function given by $f(z)=0, z<0, f(z)=z, z>0$.

We can interpret this equation as two layers of linear projections, separated by a non-linear operation, that is:

$$ \hat{y} = (xw_1^T+b_1) \circ f(x) \circ (xw_2^t+b_2), $$ where $\circ$ denotes a function aggregation.

We can draw it like this:

Why MLP? A small example¶

Let's suppose we have two 1-dimensional inputs: $x_1$ and $-x_2$, where $x_1$ and $x_2$ are real and positive. Our network has 1-d outputs as well.

For simplicity, let's assume $b_1=b_2=0$

$w_1$ will be equal to $\begin{bmatrix} 1 \\ -1 \end{bmatrix}$. Thus, $xw_1^T=\begin{bmatrix} x_1 & -x_2 \\ -x_1 & x_2 \end{bmatrix}$.

Now, after applying $f(.)$ to $xw_1^T$, we have:

$$ f(xw_1^T)=\begin{bmatrix} x_1 & 0 \\ 0 & x_2 \end{bmatrix} $$

This is important: each one of our rows are independent of each input!

Now, note that $w_2$ must be 2x1 (let's say it is equal to $[c, d]$), and:

$$ y = \begin{bmatrix} x_1 & 0 \\ 0 & x_2 \end{bmatrix} \begin{bmatrix} c \\ d \end{bmatrix} = \begin{bmatrix} c x_1 \\ d x_2 \end{bmatrix} $$

Now, importantly: in our model, the first input received a scale of $c$, while the second input received a scale of $d$.

Thus, the model operates in two layers. In the first layer, it divides the inputs into groups; in the second layer, it applies a different linear transform for each group.

Exercise

Ok, now do it yourself.

assume:

$$ x = \begin{bmatrix} 1 & 1 \\ 2 & 2 \\ -1 & -1 \\ -2 & -2 \end{bmatrix} $$

Assume your first weight matrix is:

$$ w_1 = \begin{bmatrix} 1 & 1 \\ -1 & -1 \end{bmatrix} $$

First, calculate $z_1 = x w_1^T$.

Then, calculate $y_1 = \text{ReLU}(z_1)$

Now, assume that:

$$ w_2 = \begin{bmatrix} 1 & -1 \end{bmatrix} $$

Calculate $z_2 = y_1 w_2^T$

What happened to the first and second data point? What happened to the third and fourth data points? Can we do that using a simple linear transformation?

Henceforth, what is the role of using a non-linearity to isolate layers in our network?

The Vanishing Gradient problem¶

ReLU is a powerful non-linearity, but is has an inherent problem.

Exercise

Assume:

$$ x = \begin{bmatrix}1 & 1\end{bmatrix}, $$

$b=-5$

and

$$ w = \begin{bmatrix}1 & 1\end{bmatrix} $$

In this case, what is the value of $z = xw^T+b$?

What is the value of $\frac{dz}{dx_1}$ and $\frac{dz}{dx_2}$?

Why is this harmful to gradient descent?

Visualizing the vanishing gradient¶

The values for the weight and bias matrices must be trained using gradient descent. However, $f(.)$ has zero gradient for all negative inputs. For this reason, it is common to see output values organizing in straight lines - located exactly where the point of inflection is.

Residual blocks¶

The problem of zero-gradient has been tackled by many approaches. One of the most successfull was to create an alternate route for gradients to propagate. This route is called "residual", and involves adding the input to the output of the network, that is:

$$ \hat{y} = x + (f(xw_1^T+b_2)w_2^T+b_2), $$

or, using function aggregation:

$$ \hat{y} = x + ((xw_1^T+b_1) \circ f(x) \circ (xw_2^T+b_2)), $$ where $\circ$ denotes a function aggregation.

Normalization¶

The non-linearities allow applying different transforms to each region of the input space. The residual connections avoids the vanishing gradient problem. Now, we add an extra layer of stability by normalizing data in each layer. Normalization helps maintaining all representations within reasonable values, which helps numerical stability and has ultimately been linked to faster convergence in neural networks.

Normalization works as an extra block, which is usually inserted after adding the residual connection:

Using normalization after each layer, we observe a faster convergence.

Conclusion¶

Althoug our reference states that there is a theoretical upper bound for the number of regions created by subsequent ReLU-separated projections, it is still challenging to find what are the optimal regions and corresponding projections for a particular dataset.

Our toolset for such is:

1. We can use simple linear regressions or logistic regressions to find a baseline for our system.
2. We can use the MLP topology to create potential regions in our dataset. More layers, and more neurons per layer, increase the expressivity of the network, that is, the number of linear regions it can model.
3. Adding a residual connection helps propagating gradients to the earlier layers of the MLP, which favors using the whole potential of the network.
4. Normalization layers help leading to a more numerically stable fit.

Practice¶

Make a neural network that maps $X$ to $y$ using the data below. Plot an animation of the convergence process, like the ones shown above. Try the different model variations - what happens in each case?