Intro to Prediction

Modeling, Prediction, and ‘Least Squares’ Regression

  • What is a statistical “model”?
  • How linear models make predictions
    • “target variable” (also called the “outcome variable” or “dependent variable”)
    • “features” (also called “predictors” or “explanatory variables” or “independent variables”)
    • “coefficients”
    • the purpose of the “intercept” (essentially, a feature that is always 1)
  • What makes a “good” model?
    • Predictions close to the truth (small errors, also known as residuals)
    • Specifically, small squared errors / mean squared error (MSE)
    • Baseline models
    • Adding features
    • Why do we need an intercept?
  • (At the end) Understanding the mathematical notation for all of this:
    • \(\hat{y} = \beta_0 + \beta_1 x_1\)
    • \(y = \beta_0 + \beta_1 x_1 + \epsilon\)
    • \(y\) = target variable, \(\hat{y}\) = model prediction for that variable
    • \(\beta_0\) = intercept, \(\beta_1\) = slope on \(x_1\), \(\epsilon\) = error term

What’s a model anyway?

In normal conversation, we might describe an “architectural model” or a “model citizen” or a “fashion model.” All of these have something in common with the “models” we build in statistics.

Architectural model: a simplified version of a real thing

“The center has to be at least… three times bigger than this!” (Zoolander)

An architectural model is a miniature of something the builders are thinking of making. Having the miniature helps visualize what the final thing will look like. It gives you an idea of important things before you sink millions into building it: if the layout of the plaza seems to be working, where the light will land, things like that.

A model building is, obviously, NOT a building. It’s a simplified version of the real thing, which helps us understand how the real thing might work.

Model citizen / Fashion Model

Fabio on the cover of People magazine, 1993

Fabio modeling a Gap shirt

“Fabio” was a famous Italian fashion model back in the 90s. (He also had a cameo in the Zoolander movie referenced above.)

When we look at an ad for clothes, usually it’s someone much better looking than us, showing off how good the clothes can look at their best. That is, it’s an idealized version of the real thing. As you’ll see, all statistical models are idealized versions of phenomena in the real world. Because the real world is far too complicated to fully bake into any math equation, we have to make idealized assumptions to have any shot at learning anything about the world.

Statistical models

Question: Does money buy happiness?

This is a question philosophers and psychologists have studied for ages, and the answer is complicated. With enough money, you don’t have to stress about rent, you can pay for daycare for your kids, you can go to college without taking on debt, you can travel to beautiful and interesting places – money helps. But it doesn’t fix your relationship with your mother, it can’t cure cancer, it doesn’t buy you close friends, and it can’t download statistics into your brain.

So how much does money really matter?

One way to approach this question is to build a model of the relationship between money and happiness. Because it’s complicated, we have to simplify reality by making some idealized assumptions.

We’ll gather some relevant data first. This one comes from the (wonderful) website Our World in Data. Have a look at the plot below of all the world’s countries. Each country is a point, its wealth is on the x axis (measured by GDP per capita), and its happiness on the y axis (average scores people gave on a survey, rating their life from 0 to 10).

“GDP per capita” you can (very) roughly think of as “how much money a typical person makes in a year”

We’ll measure GDP per capita in thousands of dollars, so a country with a GDP per capita of $50,000 has a gdp of 50 in our dataset.

That’s a funny shape, but it looks like wealthier countries tend to be happier? Roughly?

Let’s make a statistical model of happiness.

A statistical model is a mathematical model, an equation. It tries to describe where a number, like happiness, comes from.

We’ll start with a very dumb model: “every country has the same happiness, no matter how much money they have.” That’s not going to be very helpful to our question about money and happiness, but you’ll see why we start here in a moment.

“Every country has the same happiness” looks like this, mathematically:

happiness = ?

We get to pick the value of ?. We could say “every country has a happiness of 5” or “every country has a happiness of 6.1.” All of these are valid models! None are very good, but some are definitely worse than others. “happiness = 0” is obviously worse than “happiness = 5.”

Whatever value we pick, we can plot our model equation on the scatter plot. Since these are all of the form happiness = (some constant), we’ll get a horizontal line.

Drag the line in the chart below to find the one you think is “best.”

Make a note of what value you picked. We’ll check it out below.

How good is this model?

A good model should make predictions that are close to the truth. The difference between a prediction and the truth is called the error (or “residual” – what’s left over).

Below, we use the model happiness = 5. Since most countries don’t have a happiness of exactly 5, most of our model’s predictions are a little wrong (they have error). Hover over the points in the plot below to see the difference between a country’s true happiness and our model’s prediction.

Obviously a better model has smaller errors. But in most of statistical modeling, we want to minimize squared errors. There are good theoretical reasons for using the square, which are beyond our scope here, but you can think of it as a way of avoiding really bad predictions.

Below is the same plot, but hovering shows you the squared errors (visualized as actual squares). These are what we are actually trying to minimize.

Notice how the squares grow really fast as you get further from our line. By trying to minimize squares, we try to avoid really bad predictions (that generate huge squares).

Mean Squared Error (MSE)

If you were to average the sizes of all the squares in the plot above, you’d get the mean squared error (MSE) of the model. For our purposes, MSE is a single number that tells us how good a model is. If your model fits the data well, errors are small, squared errors are small, and so mean squared error (MSE) is small. The ideal model would have ZERO error (every prediction is spot-on), meaning zero squared error and zero MSE.

Smaller MSE = better model

The plot above used happiness = 5. But remember, we can pick any value for ? in our model, happiness = ?.

Now drag the horizontal line in the chart below to find the intercept (? value) that minimizes MSE.

What intercept did you pick? As it turns out, for an intercept-only model like this, the best you can do is to just predict the average. The countries’ average happiness in this dataset is 5.68, and a horizontal line there gives an MSE of about 1.14. Was this close to what you settled on?

This best intercept-only (horizontal line) model we will call our baseline model. It’s the best we can do if we just make the same prediction for everyone. If we want to make a fancier model, we at least need to do better than that.

Baseline model: happiness = 5.68 (MSE = 1.14)

Doing better: different predictions for different levels of wealth

The whole point of this was to explore the relationship between money and happiness. So let’s make a model that lets wealthier countries have higher predicted happiness.

You might be tempted to write something like this:

happiness = ? * gdp

This model says, “A country’s happiness is some multiple of its gdp.” Sounds reasonable enough.

Drag the handle on the line below to change the value of ? and see what kinds of models you can make.

These lines are… terrible. What’s the best you could do? An MSE of around 12 is about as good as it gets. That’s way worse than our previous model (MSE = 1.14) which just said “every country has a happiness of 5.68”! Why?

The problem is that this model, given the formula happiness = ? * gdp, HAS to predict a happiness of zero for a country with a gdp of zero. If you plug in 0 for gdp, no matter what ? is, ?*0 will be zero. In other words, this line HAS to go through the origin, (0,0). That’s a huge limitation, and it leads to only horrible models.

The fix: add an intercept

We let the line float up and down by adding a new adjustable part to the general model form:

happiness = ? + ? * gdp

You’ll recognize this as a variant of the classic formula for a line, y = mx + b (here we’ve just written it as y = b + mx).

This extra ? lets the line slide up and down and intersect the y-axis anywhere. Now play with the figure below to set both ? values. Drag up and down on the left end to change the intercept (first ?), and drag the handle on the right to rotate the line, changing the slope on gdp (the second ?). Try to find the best line!

How did you do? What values for the intercept and slope did you find gave you the smallest MSE?

It turns out the very best ? values for this model formula are: an intercept of 4.90 and a slope of 0.028:

happiness = 4.90 + 0.028 * gdp (MSE = 0.59)

Try it out. Recreate this “line of best fit” in the plot above. You should get an MSE of about 0.59. That’s a lot better than the MSE of the baseline model, which was 1.14.

When you truly minimize MSE, given the model constraints, you have found the “least squares” solution. Hence, what we are doing here is called least squares regression.

Interpretation

The model we just made says, “A country’s happiness works like this: happiness is 4.90, plus 0.028 for every thousand dollars of GDP per capita.”

Based on this model, it looks like money does matter… some. “0.028 for every thousand dollars” is not nothing, but it clearly doesn’t explain everything either (we didn’t predict everything exactly right). But it’s useful – it suggests that money does seem to matter, and by about what amount. Just like making a model of a building can suggest if the plaza layout is going to work out when you build it – it’s not a perfect approach, but it can be useful.

Importantly, we can’t conclude that more money causes more happiness from this model alone. All it says is that we can better predict a country’s happiness by knowing how wealthy it is. The evidence is merely suggestive of how much money might matter. We have a full lesson later on the intricacies of causation, which is at the core of science.

Finally, notice that even the best straight line we made doesn’t seem to fit our data all that great. There’s this whole curving thing going on – it looks like as countries get a little wealthier, happiness goes up a lot, but then further wealth doesn’t help as much. Keep this in mind for the next lesson, as we explore ways to capture this relationship more accurately.

Mathematical notation

We’ve been casual in our notation so far, which was important to get the intuition. But now let’s be a little more formal.

The thing we are trying to predict (happiness here) is called the target variable (also called the “outcome variable” or “dependent variable”) and usually written as \(y\).

The attributes of a country that inform our predictions (gdp here) are called features (also called “predictors” or “explanatory variables” or “independent variables”). These are usually written as \(x_1, x_2, x_3, ...\) depending on how many features we’re using. With our happiness = 4.90 + 0.028 * gdp model, we just have \(x_1\) (gdp).

The numbers we multiply the features by (the ? values we played with) are called coefficients. These are usually written as \(\beta_1, \beta_2, \beta_3, ...\). Here, we just have \(\beta_1\) (the slope on gdp, 0.028).

The intercept is actually a coefficient! Usually referred to as \(\beta_0\), it is one of the ?s we played with, and it is multiplied by an artificial feature \(x_0\) that is just 1 for every country. So we have TWO coefficients and TWO features – one feature is just always 1 for every country. Watch how that reduces our formula:

\(y = \beta_0 x_0 + \beta_1 x_1 = \beta_0 * 1 + \beta_1 x_1\)

So:

\(y = \beta_0 + \beta_1 x_1\)

Where \(y\) is a country’s happiness, \(x_1\) is a country’s gdp, \(\beta_0\) is the intercept (4.90), and \(\beta_1\) is the slope on gdp (0.028).

If a new country were to be formed tomorrow, and we know its gdp was (say) $40,000, we could predict its happiness by plugging in \(x_1 = 40\) into our model:

\(y = \beta_0 + \beta_1 x_1 = 4.90 + 0.028 * 40 = 6.02\)

But hold on a minute…

This is an equation. It has an equals sign. But we know our predictions are usually a little wrong! Where is the error?

To make this a true equality, we need to add an error term, usually written as \(\epsilon\), to the right-hand side:

\(y = \beta_0 + \beta_1 x_1 + \epsilon\)

Let’s consider Egypt. Its gdp per capita is $17,265, so \(x_1 = 17.27\). If we plug this into our model, we get a predicted happiness of:

\(y_{Egypt} = \beta_0 + \beta_1 x_{1 (Egypt)} = 4.90 + 0.028 * 17.27 = 5.38\)

…but Egypt’s actual happiness is 3.86, not 5.38! This equation is… not equal!

What we really have is an equation for the predicted happiness of Egypt. To indicate a prediction, we draw a “hat” on top of the \(y\), writing it \(\hat{y}\).

\(\hat{y}_{Egypt} = \beta_0 + \beta_1 x_{1 (Egypt)} = 4.90 + 0.028 * 17.27 = 5.38\)

This is fine and correct. Our prediction is 5.38. It’s not equal to the truth, but we never claimed it was. The true happiness of Egypt is 3.86 – meaning we have an error/residual of -1.52.

We can keep the original \(y\) (no hat) equation if we include the error term explicitly.

\(y_{Egypt} = \beta_0 + \beta_1 x_{1 (Egypt)} + \epsilon_{Egypt}\)

This gives us a true equality:

\(y_{Egypt} = 4.90 + 0.028 * 17.27 + (-1.52) = 3.86\)

Which is exactly right.