Lecture 22: Tools for Diagnosing Model Performance

Applied Machine Learning

Volodymyr Kuleshov
Cornell Tech

Practical Considerations When Applying Machine Learning

Suppose you trained an image classifier with 80% accuracy. What's next?

We will next learn how to prioritize these decisions when applying ML.

Part 1: Learning Curves

Learning curves are a common and useful tool for performing bias/variance analysis in a deeper way.

This section is mostly based on materials from an e-book by Andrew Ng.

Review: Overfitting (Variance)

Overfitting is one of the most common failure modes of machine learning.

Models that overfit are said to be high variance.

Review: Underfitting (Bias)

Underfitting is another common problem in machine learning.

Because the model cannot fit the data, we say it's high bias.

Learning Curves

Learning curves show performance as a function of training set size.

Learning curves are defined for fixed hyperparameters. Observe that dev set error decreases as we give the model more data.

Visualizing Ideal Performance

It is often very useful to have a target upper bound on performance (e.g., human accuracy); it can also be visualized on the learning curve.

Extrapolating the red curve suggests how much additional data we need.

In the example below, the dev error has plateaued and we know that adding more data will not be useful.

Learning Curves for the Training Set

We can further augment this plot with training set performance.