Improving Predictions and Overfitting

STAT 20: Introduction to Probability and Statistics

Warmup
10:00

Which of the following models looks most impressive to you?

(notice that \(Y\) is measured in meters, m)

I’d say B is definitely the best but what’s after that?

Concept questions

Eyeball the sample RMSE and then MSE of the models.

Case RMSE MSE Case RMSE MSE Case RMSE MSE
A \(\approx 2 \mbox{ m}\) \(\approx 4 \mbox{ m}^2\) B \(\approx 1 \mbox{ m}\) \(\approx 1 \mbox{ m}^2\) C \(\approx 2 \mbox{ m}\) \(\approx 4 \mbox{ m}^2\)
D \(\approx 5 \mbox{ m}\) \(\approx 25 \mbox{ m}^2\) E \(\approx 3 \mbox{ m}\) \(\approx 9 \mbox{ m}^2\) F \(\approx 6 \mbox{ m}\) \(\approx 36 \mbox{ m}^2\)

If you must be more accurate…

Eyeball the \(Y\)-variances of the samples (i.e. the MSE of the baseline models).

Case \(\mbox{MSE}_\mbox{baseline}\) Case \(\mbox{MSE}_\mbox{baseline}\) Case \(\mbox{MSE}_\mbox{baseline}\)
A \(\approx 2^2=4 \mbox{ m}^2\) B \(\approx 3^2=9 \mbox{ m}^2\) C \(\approx 2^2 = 4 \mbox{ m}^2\)
D \(\approx 5^2=25 \mbox{ m}^2\) E \(\approx 5^2=25 \mbox{ m}^2\) F \(\approx 6^2=36 \mbox{ m}^2\)

Or, more accurately…

And finally the \(R^2\) are?

Case \(R^2\) Case \(R^2\) Case \(R^2\)
A \(\approx 1- \frac{4\mbox{ m}^2}{4\mbox{ m}^2} = 0\) B \(\approx 1-\frac{1 \mbox{ m}^2}{9 \mbox{ m}^2} \approx 0.89\) C \(\approx 1- \frac{4\mbox{ m}^2}{4\mbox{ m}^2} = 0\)
D \(\approx 1- \frac{25\mbox{ m}^2}{25\mbox{ m}^2} = 0\) E \(\approx 1- \frac{9 \mbox{ m}^2}{25 \mbox{ m}^2} = 0.64\) F \(\approx 1- \frac{49\mbox{ m}^2}{49\mbox{ m}^2} = 0\)

The models in A, C, D, F just don’t really do much better than baseline at all…

A and D because the function is bad. Choose a nonlinear function next time! C and F just seem to have no interesting structure at all in them, it’s just useless data…

Did you notice?

\(\mbox{RMSE}_C \approx 2 \mbox{ m} < 3 \mbox{ m} \approx \mbox{RMSE}_E\) but… \(R^2_C \approx 0 \ll 0.64 \approx R^2_E\)

Did you notice?

\(\mbox{RMSE}_C \approx 2 \mbox{ m} < 3 \mbox{ m} \approx \mbox{RMSE}_E\) but… \(R^2_C \approx 0 \ll 0.64 \approx R^2_E\)

  • The model in C makes smaller errors than E in absolute terms.
  • But the model in E improves much more over the baseline!
  • The model in C is almost useless.

The Moral of the Story

  • MSE/RMSE measure how good your predictions are in absolute terms. They have units of measurement.
    • MSE behaves better as a mathematical object and is important theoretically.
    • RMSE behaves worse because of the square root, but is more intuitive to a human. (It has the same units as the \(Y\) variable)
  • \(R^2\) measures how good your predictions are relative to the baseline (or the variance of the \(Y\) variable). How impressive your model is. It will forgive you a little if your data has a lot of variation. It’s a proportion, so it has no units.

Concept question

  • Using time_being_pet, which lm formula is most appropriate to predict hunger, based on the below visualization?

formula = hunger ~ I(time_being_pet^2). Careful! poly(time_being_pet, degree = 2, raw = TRUE) wouldn’t be equivalent. Why?

Activity: Learning Latin!

Activity: Learning Latin!

In Latin

‘Dog’ is canis ([ˈka.nɪs])

(write this down)

Activity: Learning Latin!

In Latin

‘House’ is domus ([ˈdɔ.mʊs])

(write this down)

Activity: Learning Latin!

In Latin

‘Girl’ is puella ([puˈɛl.la])

(write this down)

Activity: Learning Latin!

Do you know Latin now? Let’s test you

Activity: Learning Latin!

Do you know Latin now? Let’s test you

What’s Latin for ‘dog’?

What’s Latin for ‘house’?

What’s Latin for ‘girl’?

Activity: Learning Latin!

Do you know Latin now? Let’s test you

Wow! 3/3??

Perfect! You clearly know Latin now!

Activity: Learning Latin!

What’s Latin for ‘trumpet’?

Activity: Learning Latin!

You overfitted!

You just learned the training set by heart because you have a complex brain ( = complex model) and you had a 100% score on the training set.

But when I tested you on new data you got 0% on the test set.

Break

05:00

Worksheet: Evaluating, Improving predictions, and Overfitting

40:00

Lab

45:00