Perform leave-one-out cross validation

Source: R/loocv.R, R/loocv_glm.R

Perform leave-one-out cross validation with options for computationally efficient approximations for big data.

# S3 method for ssn_lm
loocv(object, cv_predict = FALSE, se.fit = FALSE, ...)

# S3 method for ssn_glm
loocv(object, cv_predict = FALSE, se.fit = FALSE, ...)

Arguments

object

A fitted model object from ssn_lm() or ssn_glm().

cv_predict

A logical indicating whether the leave-one-out fitted values should be returned. Defaults to FALSE.

se.fit

A logical indicating whether the leave-one-out prediction standard errors should be returned. Defaults to FALSE.

...

Other arguments. Not used (needed for generic consistency).

Value

If cv_predict = FALSE and se.fit = FALSE, a tibble indicating several leave-one-out cross validation error metrics. If cv_predict = TRUE or se.fit = TRUE, a list with elements: stats, a tibble indicating several leave-one-out cross validation metrics; cv_predict, a numeric vector with leave-one-out predictions for each observation (if cv_predict = TRUE); and se.fit, a numeric vector with leave-one-out prediction standard errors for each observation (if se.fit = TRUE).

If an ssn_lm object, the cross validation error metrics are:

  • bias: The average difference between the predicted value and true value

  • std.bias: The average standardized difference between the predicted value and true value

  • MSPE: The average squared difference between the predicted value and true value

  • RMSPE: The root average squared difference between the predicted value and true value

  • std.MSPE: The average standardized squared difference between the predicted value and true value

  • RAV: The root of the average estimated variance of the predicted value

  • cor2: The squared correlation between the predicted and true values

  • cover.80: Coverage rates of 80% prediction intervals built for the true values

  • cover.90: Coverage rates of 90% prediction intervals built for the true values

  • cover.95: Coverage rates of 95% prediction intervals built for the true values

If an ssn_glm object, the cross validation error metrics are:

  • bias: The average difference between the predicted value and true value

  • MSPE: The average squared difference between the predicted value and true value

  • RMSPE: The root average squared difference between the predicted value and true value

  • RAV: The root of the average estimated variance of the predicted value (on the link scale)

Details

Each observation is held-out from the data set and the remaining data are used to make a prediction for the held-out observation. This is compared to the true value of the observation and several model-fit statistics are computed across all observations.

Examples

# Copy the mf04p .ssn data to a local directory and read it into R
# When modeling with your .ssn object, you will load it using the relevant
# path to the .ssn data on your machine
copy_lsn_to_temp()
temp_path <- paste0(tempdir(), "/MiddleFork04.ssn")
mf04p <- ssn_import(temp_path, overwrite = TRUE)

ssn_mod <- ssn_lm(
  formula = Summer_mn ~ ELEV_DEM,
  ssn.object = mf04p,
  tailup_type = "exponential",
  additive = "afvArea"
)
loocv(ssn_mod)
#> # A tibble: 1 × 10
#>     bias std.bias  MSPE RMSPE std.MSPE   RAV  cor2 cover.80 cover.90 cover.95
#>    <dbl>    <dbl> <dbl> <dbl>    <dbl> <dbl> <dbl>    <dbl>    <dbl>    <dbl>
#> 1 0.0449   0.0456 0.273 0.522     1.23 0.418 0.930      0.8    0.889    0.911