• Learn about the assumptions of linear regression
• Learn how to use diagnostic plots
• Learn about the limitations of statistical tests for assumption checking

• Total error composed of two parts: bias and random error

• Bias - systemic error, i.e. systemic under- or overestimation of the expected values
• Random error - random deviations from the true value (due to the sampling and measurement error)

• Violations of assumptions can increase either or both

• Precision

  • reciprocal to the random error (high precision = low random error)

• Efficiency

  • how many observations we need to achieve a specific level of performance
  • e.g. model based on data that meet its assumptions require less observations to achieve 80% statistical power, i.e. is more efficient

All models are wrong, but some are useful.
- George Box

• No model is perfect picture of reality and that’s ok.
• How good the picture is?

• Small violations lead to only small error, big ones…

• Don’t think about assumptions in binary term (“ok” vs “bad”)
• Instead, think how much we deviate from them

• Not all assumption are also important for all tasks

• Helpfully sorted by Gelman et al. (2020) in order of (general) importance:

  1. Validity

  2. Representativeness

  3. Linearity and additivity

  4. Independence of errors

  5. Homoscedasticity of errors

  6. Normality of errors

1. Are the data we use are sufficient to answer the research question at hand?
2. Are all traits measured with sufficient quality?
3. Are data gathered to so that inference is possible?
4. Is the model correctly specified?

• Violation of this assumption will lead to bias

• If our goal is either prediction or inference, our data needs to be a representative sample of the population of interest

• More specifically, we assume that the distribution of the dependent variable is representative of the population, given the the predictors

• Do the data need to be representative in all aspects?

• Consider model predicting income based on age. What happens if our data are not representative?

  • More specifically, what if only people with above average age/income participate in our survey?

• Estimates even when sample is not representative in age (independent variable).

• Two ways people talk about linearity:
  • Linearity of relationship between variables
  • Linearity of regression terms

• What is the difference?

• Simple, the relationship between variables is a straight line.

• Linear models are linear, because they assume linear form:

\[ y = \beta_0 + \beta_1*x_1 + \beta_2*x_2\:+\:...\:\beta_p*x_p \]

• No all relationship are neccesary linear though:

\[ y = \beta_0 + (\beta_1*x_1) * (\beta_2*x_2) \]

• Model is linear, if the regression coefficients are only either add or subtracted from each other

• Some nonlinear models can be translated into linear form using an appropriate transformation:

\[ \beta_0 + log[(\beta_1*x_1) * (\beta_2*x_2)] = \beta_0 + log(\beta_1*x_1) + log(\beta_2*x_2) \]

• Linear models can only capture relationships that fulfill this assumption
• Violation of this assumption leads to bias

• We assume that the effects of independent variables can be simply added together.
• Sometimes, this may not be true, e.g. a interaction is needed.

• Violation of this assumption leads to bias.

• Linear regression assumes that errors are independent of each other.

• Often violated due to sampling design, time series and spatial analysis.

• Violation of this assumption leads to incorrect estimation of standard error, and therefore presents a risk to inference

• Homoscedasticity = variance of residuals is the same for all levels of \(Y\)
• If the variance of the errors is not constant, the the error are heteroscedastic.

• The violation of this assumption has two consequences.

• Firstly, the estimation of the regression coefficients become inefficient.
• Secondly, standard errors of the coefficients would be biased

• Linear regression assumes that the errors are normally distributed.

• Especially important for the prediction of individual observations.

• Also important for inference on small samples.

• Violation of this assumptions lead to incorrect prediction for individual observations and for incorrect p values and confidence intervals in small samples.

• The main, and arguably best, tool for checking assumptions are diagnostic plots.

• The assumption of linearity can be checked by plotting predicted values against residuals (usually both are Z transformed)

• Ideally, there should be no pattern in the data, nonlinear patterns suggests nonlinear relationships that haven’t been accounted for

• Can be checked using the same plots and linearity, or the scale location plot.

• Can be checked either by histogram or Quantile-Quantile plot

• Not strictly an assumption, but still important.
• Influential observations can distort the relationship between variables

• Basic measure of influence is leverage
• leverage = how big a role an observation plays when fitting a regression line
• More precisely, how far the observation lies from the centre of the data

• The further the observation from the centre, the higher its leverage

• Leverage formally:

\[ leverage_i = \frac{\partial \hat{y_i}}{\partial y_i} = \frac{partial\:change\:in\:expected\:y_i}{partial\:change\:in\:observed\:y_i} \]

• How much the bigger effect on the expected value would changing the observed value have, the bigger that observation’s leverage

• However, just because an observation has high leverage does not necessarily mean it will distort our model
• Only observations with high leverage and high residuals distorts the model

• The information about leverage and residuals can be summarized using Cook’s distance:

\[ Cook's\:distance_i=\frac{residual_i^2}{number\:of\:parameters*MSE}*\left[\frac{leverage_i}{(1-leverage_i)^2}\right] \] - An observation will generally have high Cook’s distance if both their residual and their leverage is very high

• Classic diagnostic plots allow us to check the model as a whole
• But if something looks wrong, how can we tell which variable is the source of the problem?
• Consider model predicting infant mortality rate (infantMortality) by total fertility rate (tfr) and GDP per capita (GDPperCapita):

mod_res = lm(infantMortality ~ tfr + GDPperCapita, data = un)

• From the residual plot, there seems to be a problem with heteroskedasticity and linearity, but which of the predictors is problematic? tfr? GDPperCapita? Both?

• We can produce partial residual plot (also known as component residual plot) for each predictor (e.g. function crPlots from the car package)

• Some researchers prefere testing assumptions formally, using statistical tests

  • Shapiro-Wilk test for normality, Levene’s test for homoscedasticity, etc.

• This is universally a bad idea and should be avoided

• Two reasons:

  • In real data, most (all?) assumptions literaly cannot be absolutely true
  • It is not neccesary to fulfill the assumptions to the letter

• Consider normal distribution
• In real life, normal distribution doesn’t exist -> we know that our residuals don’t follow normal distributio before we even run a test
• Consequently, all observed nonsignificant results are false negatives

• Some say:
  

“We should use normality tests for small samples, in large samples they are too sensitive!”

• This misunderstands the problem
• The problem is the hypothesis, not the test

• Tests of equal variance are somewhat more plausible, but still highly unrealistic

• Luckily for us, we don’t need all the assumptions to be fullfiled perfectly
• Small deviations from the assumptions only lead to a small errors -> for practical purposes, our model will still work fine