set.seed(20)
shuffled_email <- slice_sample(email, replace = FALSE, n = nrow(email))
train_rows <- seq(from = 1, to = nrow(shuffled_email)* 0.65, by = 1)
email_train <- slice(shuffled_email, train_rows)
email_test <- slice(shuffled_email, -train_rows)Logistic Regression
A model for binary classification
- train/test split
- fitting a logistic regression with
glm - interpreting coefficients
- computing \(\hat{p}(X)\) and predicted \(\hat{Y}(X)\)
- computing confusion matrix, MCR, Precision, Recall
The Ideas in Code
Splitting test and training sets
This is identical to what we did last time.
Quiz: what training/test split ratio did we chose?
65% training, 35% test.
Why do you think I chose such a large percentage for testing?
Fitting Logistic Regression with glm()
The computational machinery for fitting logistic regression looks almost identical to what we used for linear least squares regression. The primary function we’ll use is glm(). We will also use a different broom function called tidy() to display the coefficients \(b_0\) and \(b_1\). tidy() can also be used in a linear regression context as well.
m1 <- glm(spam ~ log(num_char), data = email_train, family = "binomial")
m1 |>
tidy() |>
select(term,estimate,std.error)# A tibble: 2 × 3
term estimate std.error
<chr> <dbl> <dbl>
1 (Intercept) -1.70 0.0740
2 log(num_char) -0.580 0.0459
What’s the deal with family = "binomial"?
What’s “binomial” about the Logistic regression?
Well, this is exactly where the assumption of the \(Y\) being a Bernoulli (and therefore a special case of a Binomial!) variable and the consequent form of the Cross Entropy in the notes come in.
We are telling the code that we are making these assumptions, so that it will find the best values of the coefficients by optimizing the correct Cross Entropy function.
These coefficients are a bit more challenging to interpret since they’re no longer linearly related to the response. The sign of the coefficient for log(num_char), however, is informative: it tells us that messages with more characters will be predicted to have a lower probability of being spam.
Evaluating model performance
Calculating \(\text{MCR}\), Precision, and Recall.
Let’s take a look at the result of training this model. When using the predict() function on logistic regression models, there are several different types of estimations or predictions that it can return, so be sure to use the additional argument type and supply it with "response". This will return \(\hat{p}(\mathbf{X})\) (probability of 1) for each observation.
We can then move from values of \(\hat{p}(\mathbf{X})\) to values of \(\hat{Y}(\mathbf{X})\). In general, our rule will be to check and see whether each value of \(\hat{p}(\mathbf{X})\) is greater or equal than .5. If so, we will assign the value of 1; otherwise, we will assign it 0. This sounds similar to creating a logical variable, but assigning something different than TRUE and FALSE to each observation in the dataset. We will use a function called ifelse(), therefore, which will allow us to assign we want (1 and 0).
The following block of code completes both of these two steps and saves the results back into the email_test data frame.
email_test <- email_test |>
mutate(p_hat_m1 = predict(object = m1,
newdata = email_test,
type = "response"),
y_hat_m1 = ifelse(test = p_hat_m1 >= .5,
yes = 1,
no = 0))
head(email_test,5) |>
select(num_char, spam, p_hat_m1, y_hat_m1)# A tibble: 5 × 4
num_char spam p_hat_m1 y_hat_m1
<dbl> <fct> <dbl> <dbl>
1 2.47 0 0.0973 0
2 19.1 0 0.0319 0
3 20.4 0 0.0307 0
4 0.831 0 0.168 0
5 28.7 0 0.0253 0
We are now ready to calculate our evaluation measures. First, we can find all of the confusion matrix elements. Then, we can calulate the cirrect proportions, using the definitions in the notes.
email_test |>
summarize(
true_poss = sum(spam == 1 & y_hat_m1 == 1),
false_poss = sum(spam == 0 & y_hat_m1 == 1),
true_negs = sum(spam == 0 & y_hat_m1 == 0),
false_negs = sum(spam == 1 & y_hat_m1 == 0),
n_total = n()
) |>
mutate(
MCR = (false_poss + false_negs) / n_total,
Precision = true_poss / (true_poss + false_poss),
Recall = true_poss / (true_poss + false_negs)
) |>
select(MCR, Precision, Recall, true_poss, false_poss, true_negs, false_negs)# A tibble: 1 × 7
MCR Precision Recall true_poss false_poss true_negs false_negs
<dbl> <dbl> <dbl> <int> <int> <int> <int>
1 0.0947 0.125 0.00806 1 7 1242 123
Overall, we are misclassifying around 9% of all e-mails (both spam and genuine), which doesn’t look like a bad start, but we might want to take a little deeper of a dive to see how the model is misclassifying observations.
We actually see that the model is doing great at predicting correctly when the e-mail is genuine (few false positives, many many many true negatives), but doing horribly at detecting spam (many, many false negatives): only \(1\) out of the \(124\) spam e-mails are being classified correctly (Recall = 1/124 \(\approx\) 0.8%). Essentially all of our mistakes are in the form of false negatives! Even when we predict spam we’re usually wrong about it, only 1 out of 8 positive predictions was right (Precision = 1/8 = 12.5%) We also can see here that there are way more genuine e-mails than spam e-mails in the dataset, so our misclassification rate is being deflated as a result. Clearly, more than just the length of an e-mail is necessary to help us detect spam.
Which estimate do you trust more? Precision or Recall?
Effectively, if you think about it, the sample size for Precision is 8 (that’s the denominator), for Recall it’s 124.
They’re both not great, but the size for Precision is really really tiny. I wouldn’t trust that number at all as an estimate of the population Precision (what do you think this is?). It would carry a gigantic confidence interval…
Can you calculate it?
Do you see now why I chose a larger percentage for the test set than usual?