Let's start by understanding what is unsupervised learning at a high level, starting with a dataset and an algorithm.

We have a dataset *without* labels. Our goal is to learn something interesting about the structure of the data:

- Clusters hidden in the dataset.
- Outliers: particularly unusual and/or interesting datapoints.
- Useful signal hidden in noise, e.g. human speech over a noisy phone.

At a high level, an unsupervised machine learning problem has the following structure:

$$ \text{Dataset} + \text{Algorithm} \to \text{Unsupervised Model} $$The unsupervised model describes interesting structure in the data. For instance, it can identify interesting hidden clusters.

As a first example of an unsupervised learning dataset, we will use our Iris flower example, but we will discard the labels.

We start by loading this dataset.

In [6]:

```
# import standard machine learning libraries
import numpy as np
import pandas as pd
from sklearn import datasets
# Load the Iris dataset
iris = datasets.load_iris()
print(iris.DESCR)
```

We can visualize this dataset in 2D. Note that we are no longer using label information.

In [10]:

```
from matplotlib import pyplot as plt
plt.rcParams['figure.figsize'] = [12, 4]
# Visualize the Iris flower dataset
plt.scatter(iris.data[:,0], iris.data[:,1], alpha=0.5)
plt.ylabel("Sepal width (cm)")
plt.xlabel("Sepal length (cm)")
plt.title("Dataset of Iris flowers")
```

Out[10]:

Text(0.5, 1.0, 'Dataset of Iris flowers')

We can use this dataset as input to a popular unsupervised learning algorithm, **$K$-means**.

- The algorithm seeks to find $K$ hidden clusters in the data.
- Each cluster is characterized by its centroid (its mean).
- The clusters reveal interesting structure in the data.

In [7]:

```
# fit K-Means with K=3
from sklearn import cluster
model = cluster.KMeans(n_clusters=3)
model.fit(iris.data[:,[0,1]])
```

Out[7]:

KMeans(n_clusters=3)

Running $K$-means on this dataset identifies three clusters.

In [16]:

```
# display the clusters in 2D
plt.scatter(iris.data[:,0], iris.data[:,1], alpha=0.5)
plt.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], marker='D', c='r', s=100)
plt.ylabel("Sepal width (cm)")
plt.xlabel("Sepal length (cm)")
plt.title("Dataset of Iris flowers")
plt.legend(['Datapoints', 'Probability peaks'])
```

Out[16]:

<matplotlib.legend.Legend at 0x1231df4a8>

These clusters correspond to the three types of flowers found in the dataset, which we obtain from the labels.

In [17]:

```
p1 = plt.scatter(iris.data[:,0], iris.data[:,1], alpha=1, c=iris.target, cmap='Paired')
plt.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], marker='D', c='r', s=100)
plt.ylabel("Sepal width (cm)")
plt.xlabel("Sepal length (cm)")
plt.title("Dataset of Iris flowers")
plt.legend(handles=p1.legend_elements()[0], labels=['Iris Setosa', 'Iris Versicolour', 'Iris Virginica'])
```

Out[17]:

<matplotlib.legend.Legend at 0x123247fd0>

Unsupervised learning has numerous applications:

- Visualization: identifying and making accessible useful hidden structure in the data.
- Anomaly detection: identifying factory components that are likely to break soon.
- Signal denoising: extracting human speech from a noisy recording.

Unsupervised learning can discover structure in digits without any labels.

Dimensionality reduction applied to DNA reveal the geography of European countries:

Modern unsupervised algorithms based on deep learning uncover structure in human face datasets.

We will explore several types of unsupervised learning problems.

- Density estimation
- Anomaly and novelty detection
- Clustering
- Dimensionality Reduction

Next, we will start by setting up some notation.

Next, let's look at how to define an unsupervised learning problem more formally.

At a high level, an unsupervised machine learning problem has the following structure:

$$ \underbrace{\text{Dataset}}_\text{Attributes} + \underbrace{\text{Learning Algorithm}}_\text{Model Class + Objective + Optimizer } \to \text{Unsupervised Model} $$The unsupervised model describes interesting structure in the data. For instance, it can identify interesting hidden clusters.

We define of size $n$ a dataset for unsupervised learning as $$\mathcal{D} = \{x^{(i)} \mid i = 1,2,...,n\}$$

Each $x^{(i)} \in \mathbb{R}^d$ denotes an input, a vector of $d$ attributes or features.

We will assume that the dataset is sampled from a probability distribution $\mathbb{P}$, which we will call the *data distribution*. We will denote this as
$$x \sim \mathbb{P}.$$

The dataset $\mathcal{D} = \{x^{(i)} \mid i = 1,2,...,n\}$ consists of *independent and identicaly distributed* (IID) samples from $\mathbb{P}$.

The key assumption in that the training examples are *independent and identicaly distributed* (IID).

- Each training example is from the same distribution.
- This distribution doesn't depend on previous training examples.

**Example**: Flipping a coin. Each flip has same probability of heads & tails and doesn't depend on previous flips.

**Counter-Example**: Yearly census data. The population in each year will be close to that of the previous year.

We can think of a unsupervised learning algorithm as consisting of three components:

- A
**model class**: the set of possible unsupervised models we consider. - An
**objective**function, which defines how good a model is. - An
**optimizer**, which finds the best predictive model in the model class according to the objective function

We'll say that a model is a function $$ f : \mathcal{X} \to \mathcal{S} $$ that maps inputs $x \in \mathcal{X}$ to some notion of structure $s \in \mathcal{S}$.

Structure can have many definitions (clusters, low-dimensional representations, etc.), and we will see many examples.

Often, models have *parameters* $\theta \in \Theta$ living in a set $\Theta$. We will then write the model as
$$ f_\theta : \mathcal{X} \to \mathcal{S} $$
to denote that it's parametrized by $\theta$.

Formally, the model class is a set $$\mathcal{M} \subseteq \{f \mid f : \mathcal{X} \to \mathcal{S} \}$$ of possible models that map input features to structural elements.

When the models $f_\theta$ are paremetrized by *parameters* $\theta \in \Theta$ living in some set $\Theta$. Thus we can also write
$$\mathcal{M} = \{f_\theta \mid f : \mathcal{X} \to \mathcal{S}; \; \theta \in \Theta \}.$$

To capture this intuition, we define an *objective function* (also called a *loss function*)
$$J(f) : \mathcal{M} \to [0, \infty), $$
which describes the extent to which $f$ "fits" the data $\mathcal{D} = \{x^{(i)} \mid i = 1,2,...,n\}$.

When $f$ is parametrized by $\theta \in \Theta$, the objective becomes a function $J(\theta) : \Theta \to [0, \infty).$

An optimizer finds a model $f \in \mathcal{M}$ with the smallest value of the objective $J$. \begin{align*} \min_{f \in \mathcal{M}} J(f) \end{align*}

Intuitively, this is the function that bests "fits" the data on the training dataset.

When $f$ is parametrized by $\theta \in \Theta$, the optimizer minimizes a function $J(\theta)$ over all $\theta \in \Theta$.

As an example, let's use the $K$-Means algorithm that we saw earlier.

Recall that:

- The algorithm seeks to find $K$ hidden clusters in the data.
- Each cluster is characterized by its centroid (its mean).
- The clusters reveal interesting structure in the data.

We can think of the model returned by $K$-Means as a function $$f_\theta : \mathcal{X} \to \mathcal{S}$$ that assigns each input $x$ to a cluster $s \in \mathcal{S} = \{1,2,\ldots,K\}$.

The parameters $\theta$ of the model are $K$ *centroids* $c_1, c_2, \ldots c_K \in \mathcal{X}$. The class of $x$ is $k$ if $c_k$ is the closest centroid to $x$.

How do we determine whether $f_\theta$ is a good clustering of the dataset $\mathcal{D}$?

We seek centroids $c_k$ such that the distance between the points and their closest centroid is minimized: $$J(\theta) = \sum_{i=1}^n || x^{(i)} - \text{centroid}(f_\theta(x^{(i)})) ||,$$ where $\text{centroid}(k) = c_k$ denotes the centroid for cluster $k$.

We can optimize this in a two stop process, starting with an initial random cluster assignment $f(x)$.

Repeat until convergence:

- Set each $c_k$ to be the center of the its cluster $\{x^{(i)} \mid f(x^{(i)}) = k\}$.
- Update clustering $f(x)$ such that $x^{(i)}$ is in the cluster of its closest centroid.

This is best illustrated visually (from Wikipedia):

**Type**: Unsupervised learning (clustering)**Model family**: $k$ centroids**Objective function**: Sum of distances to closest centroid**Optimizer**: Iterative optimization procedure.

We will now look at some practical considerations to keep in mind when applying supervised learning.

We will assume that the dataset is sampled from a probability distribution $\mathbb{P}$, which we will call the *data distribution*. We will denote this as
$$x \sim \mathbb{P}.$$

The dataset $\mathcal{D} = \{x^{(i)} \mid i = 1,2,...,n\}$ consists of *independent and identicaly distributed* (IID) samples from $\mathbb{P}$.

In machine learning, **generalization** is the property of predictive models to achieve good performance on new, heldout data that is distinct from the training set.

How does generalization apply to unsupervised learning?

We can think of the data distribution as being the sum of two distinct components $\mathbb{P} = F + E$

- A signal component $F$ (hidden clusters, speech, low-dimensional data space, etc.)
- A random noise component $E$

A machine learning model generalizes if it fits the true signal $F$; it overfits if it learns the noise $E$.

Consider the following dataset, consisting of a mixture of Gaussians.

In [2]:

```
import numpy as np
from sklearn import datasets
from matplotlib import pyplot as plt
plt.rcParams['figure.figsize'] = [12, 4]
np.random.seed(0)
X, y = datasets.make_blobs(centers=4)
plt.scatter(X[:,0], X[:,1])
```

Out[2]:

<matplotlib.collections.PathCollection at 0x120073b38>

We know the true labels of these clusers, and we can visualize them.

In [30]:

```
plt.scatter(X[:,0], X[:,1], c=y)
```

Out[30]:

<matplotlib.collections.PathCollection at 0x122d99b00>

Underfitting happens when we are not able to fully learn the signal hidden in the data.

In the context of $K$-Means, this means not capturing all the clusters in the data.

Let's run $K$-Means on our toy dataset.

In [4]:

```
# fit a K-Means
from sklearn import cluster
model = cluster.KMeans(n_clusters=2)
model.fit(X)
```

Out[4]:

KMeans(n_clusters=2)

The centroids find two distinct components in the data, but they fail to capture the true structure.

In [52]:

```
plt.scatter(X[:,0], X[:,1], c=y)
plt.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], marker='D', c='r', s=100)
print('K-Means Objective: %.2f' % -model.score(X))
```

K-Means Objective: 462.03

Consider now what happens if we further increase the number of clusters.

In [44]:

```
Ks = [4, 10, 20]
f, axes = plt.subplots(1,3)
for k, ax in zip(Ks, axes):
model = cluster.KMeans(n_clusters=k)
model.fit(X)
ax.scatter(X[:,0], X[:,1], c=y)
ax.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], marker='D', c='r', s=100)
ax.set_title('K-Means Objective: %.2f' % -model.score(X))
```

Overfitting happens when we fit the noise, but not the signal.

In our example, this means fitting small, local noise clusters rather than the true global clusters.

In [53]:

```
model = cluster.KMeans(n_clusters=50)
model.fit(X)
plt.scatter(X[:,0], X[:,1], c=y)
plt.scatter(model.cluster_centers_[:,0], model.cluster_centers_[:,1], marker='D', c='r', s=100)
print('K-Means Objective: %.2f' % -model.score(X))
```

K-Means Objective: 4.94

The Elbow method is a way of tuning hyper-parameters in unsupervised learning.

- We plot the objective function as a function of the hyper-parameter $K$.
- The "elbow" of the curve happens when its rate of decrease substantially slows down.
- The "elbow' is a good guess for the hyperparameter.

In our example, the decrease in objective values slows down after $K=4$, and after that the curve becomes just a line.

In [77]:

```
Ks, objs = range(1,11), []
for k in Ks:
model = cluster.KMeans(n_clusters=k)
model.fit(X)
objs.append(-model.score(X))
plt.plot(Ks, objs, '.-', markersize=15)
plt.scatter([4], [objs[3]], s=200, c='r')
plt.xlabel("Number of clusters K")
plt.ylabel("Objective Function Value")
```

Out[77]:

Text(0, 0.5, 'Objective Function Value')

In unsupervised learning, overfitting and underfitting are more difficult to quantify than in supervised learning.

- Performance may depend on our intuition and require human evaluation
- If we know the true labels, we can measure the accuracy of the clustering

If our model is probabilistic, we can detect overfitting without labels by comparing the log-likelihood between the training set and a holdout set (next lecture!).

There are multiple ways to control for overfitting:

- Reduce model complexity (e.g., reduce $K$ in $K$-Means)
- Penalize complexity in objective (e.g., penalize large $K$)
- Use a probabilistic model and regularize it.

The concept of generalization applies to both supervised and unsupervised learning.

- In supervised learning, it is easier to quantify via the accuracy.
- In unsupervised learning, we may not be able to easily detect overfitting, but it still happens.

In [ ]:

```
```