December 2022

The foundation of predictive R-squared is cross-validation. We will examine the LOOCV method in this post. First, the data set that we used in the past exercises.

Then,

• The first row is removed from the list, and the regression model is developed with the other 11 data (2:12)
• The model is used to predict observation 1 (y). By plugging in the x value (e.g. 2) in the formula (cubic form)
• The predicted y is subtracted from the actual y for observation 1 and squared (called the squared residual)
• Observation 1 is returned to the list, and observation 2 is removed (1, 3:12)
• The process is continued until the last observation and squared residual are collected
• Sum all the squared residual to get what is known as PRESS (predicted residual error sum of squares)
• Predicted R² = 1 – (PRESS/TSS)

res_sq <- 0
for (i in 1:12) {
  new_presi <- Presi_Data[-i,]
  model1 <- lm(new_presi$Historians.rank ~ new_presi$Approval.High +  I(new_presi$Approval.High^2) + I(new_presi$Approval.High^3))
  
  res <- Presi_Data[i,1] - (model1$coefficients[1] + model1$coefficients[2]*Presi_Data[i,2]+model1$coefficients[3]*Presi_Data[i,2]^2 +model1$coefficients[4]*Presi_Data[i,2]^3)
  
    res_sq <- res_sq + res^2
}
res_sq

The res_sq is PRESS.

TSS (or SST) is the total sum of squares = sum of (response (y) – mean of response)²

sum((Presi_Data$Historians.rank-mean(Presi_Data$Historians.rank))^2)

predict_r_sq <- 1 - (res_sq/sum((Presi_Data$Historians.rank-mean(Presi_Data$Historians.rank))^2))

Last time we fitted a set of data using the cubic function.

Let’s analyse the quality of the fit using the predicted R-squared method. It is also called Leave-Out-One Cross Validation (LOOCV). What it does is that it systematically leaves out one data at a time and estimates how the model performs under those circumstances. For a good fit, the predicted R-squared should be high (as close to 1).

We will estimate the predicted R-squared of the dataset using the library, “olsrr“.

library(olsrr)
model <- lm(Presi_Data$Historians.rank ~ Presi_Data$Approval.High +  I(Presi_Data$Approval.High^2) + I(Presi_Data$Approval.High^3))

ols_pred_rsq(model)

The answer is -0.2, which suggests that the cubic function is overfitting the observation.

A usual tendency in regression is to chase better R-squared by increasing the dependent variables and ending up overfitting. Overfitting is a term used when your model is too complex that it sharts fitting noise. An example is this dataset taken from Regression Analysis Book by Jim Frost. It describes the highest approval ratings of the US presidents and their rank by historians.

First, we try out linear regression and check the goodness of fit.

lm(Presi_Data$Historians.rank ~ Presi_Data$Approval.High)

The R-squared value is close to zero (0.006766). Now, we try cubic fit.

model <- lm(Presi_Data$Historians.rank ~ Presi_Data$Approval.High +  I(Presi_Data$Approval.High^2) + I(Presi_Data$Approval.High^3))

But with an impressive R-squared (0.6639)

Now, try thinking how one can explain this complex relationship, i.e. Historians Rank = -9811 + 388.9 x Approval High -5.098 x Approval High^2 + 0.02213 x Approval High^3!