Intuition¶

In this example we are going to illustrate the idea of what PCA means, here we present the next data in 2D and we want to find a projection in 1D such that the maximum quantity of variability is preserved.

Let's remember that the projection of a vector $\vec{x}$ over another vector $\vec{v}$ is given by take a look here

where $c$ is the size of the projection of $\vec{x}$ over $\vec{v}$

Now let's inspect some projections¶

Now the objective is to find the directions with the biggest and the lowest variance by inspection¶

So we take the one that maximize

Principal component Analysis (PCA)¶

The principal component analysis is a technique used to described a set of data in terms of new no correlated variables (components). The components are ordered from higher to lower variance that they explain. this technique is useful to reduce significantly the dimension of the set of data.

In other words, PCA try to find the projection over which the data is best represented in terms of minimum squares. This means that it transforms the set of observations (probably correlated) in a new set of variables without linear correlation. this is very well known when building predictive models.

Limitations:¶

• Linearity: it is assumed that the observed data is a linear combination of a certain base.
• The criteria that it uses is build in the assumption that the data is Gaussian distributed therefore, the eigen-values

Recipe to Compute the PCA¶

1. once you have your data represented in a matrix form $$ X=\begin{pmatrix} x_{11}& x_{12} &\ldots & x_{1p}\\ x_{21}& x_{22} &\ldots & x_{2p} \\ \vdots& \vdots & \vdots &\vdots \\ \vdots& \vdots & \vdots &\vdots \\ x_{N1}& x_{N2} &\ldots& x_{Np} \end{pmatrix} $$ we compute the vector of means $$ \bar{X}=\frac{1}{N}\left[\sum_{i=1}^{N}x_{i1},\sum_{i=1}^{N}x_{i2},\ldots,\sum_{i=1}^{N}x_{pi}\right] $$ and we compute $X_{norm}=X-\bar{X}$, here $X_{norm}$ means that we normalized the data by its mean.
2. We compute the covariance matrix given by $C=\frac{1}{N-1}(X_{norm}^{T}X_{norm})$.
3. compute the eigenvectors and eigenvalues of the covariance matrix and then project the data into those vectors.
4. choose the quantity of variance to keep and compute the Matrix projection and project your data into that space. Voilà you have reduced your dimension.

Exercise¶

Do the same with the data found in here, it is useful to start from now on using the library Pandas. if you choose to do it by yourself, well done, download the data and read it from scratch, if you don't want to complicate the things. you can use the next command

import pandas as pd
data=pd.read_csv("https://gist.githubusercontent.com/tijptjik/9408623/raw/b237fa5848349a14a14e5d4107dc7897c21951f5/wine.csv")

see what is in data, you will see something similar to a sheet of Excel, something like

first you will need to delete from the data the column of "Wine", since this is just to know what kind of wine has those properties. Then in order to work with data in the same range, we can standardize the data, to each columns we are going to take each column and subtract the mean of each column, just as we did before, and divide each column to its respective standard deviation. ( to call a specific column in pandas we use data["name of column"]), the result should look something like this

Pandas allow us to make a plot of all variables and its combinations, it can be done by using the module of plotting in pandas( which under the hood is using matplotlib), so if we want to plot this we can use:

d = pd.plotting.scatter_matrix(data_normalized, c=data["Wine"],figsize=(15, 15))

where we have used the parameter "c" in the function to plot using the values of the wine column of the data set

Now your task is to implement a PCA as before but this time for the Wine data set.

• Compute the covariance matrix.
• Find the eigen-vectors ans eigen-values of that matrix.
• Plot the cumulative explained variance( it should look like this)

• Now do take only the first two components of the data and plot component 1 vs component 2 of the PCA hint: to visualize better the result color the point with the column "Wine" of the data set. use the following function to plot x,y (the data transformed X and the labels of the column wine).

  def plot_data(X, y):
    y_unique = np.unique(y)
    colors = plt.cm.rainbow(np.linspace(0.0, 1.0, y_unique.size))
    for this_y, color in zip(y_unique, colors):
        this_X = X[y == this_y]
        plt.scatter(this_X[:, 0], this_X[:, 1],  c=color,
                    alpha=0.5, edgecolor='k',
                    label="Class %s" % this_y)
    plt.legend(loc="best")
    plt.title("Data")
  

  The result looks like this ( ignore the error that appears):

What can you conclude from the last plot?