Residuals are your best diagnostic tool for assessing your regression model. Not only can they tell you if you’ve violated assumptions, but they can point to specific cases that contribute to the violations. This may help you to:
Notice patterns, which may lead you to change your theory
Assumption 3: Correctly specified model - This is especially important for multiple regression. - Two problems:
3) Model
Correctly specified model
Cohen and Cohen (1983) discuss several problems associated with the inclusion of covariates/multiple independent predictors in a model. Those problems were:
Computational accuracy
not a problem now, because computers
Sampling stability (tolerance)
Interpretation
3) Model
Correctly specified model: Tolerance
Recall that, computationally, including multiple predictors in your model requires adjusting for the overlap in these predictors.
\[\large se_{b} = \frac{s_{Y}}{s_{X}}\sqrt{\frac {1-R_{Y\hat{Y}}^2}{n-p-1}}\sqrt{\frac{1}{1-R_{12}^2}}\] (in a two-predictor regression)
Tolerance is:
\[\large 1-R_{12}^2\]
3) Model
Correctly specified model: Tolerance
If your two independent predictors are highly correlated, tolerance gets smaller.
As tolerance gets smaller, the standard error gets larger. This is referred to as variance inflation. The variance inflation factor is an index to assess this problem.
As the standard error becomes larger, the confidence intervals around coefficients becomes larger. When confidence intervals around estimates are large, then we say the partial coefficient estimates are unstable.
3) Model
Tolerance
VIF is not bounded, but generally large numbers (greater than 5 or 10, based on who’s giving you the heuristic) indicates a problem.
library(car)vif(model.2)
Support Stress
1.391713 1.391713
The lesson from tolerance is that, as you add predictors to your model, it is best to select predictors that are not correlated with each other.
What about (3) Interpretation?
3) Model
Lynam et al (2006)
Main takeaways:
Partialling changes the meaning and interpretation of a variable.
Partialling only takes variance away from the reliable \((r_{XX})\) part of a measurement.
Nothing is a good substitute for good theory and reliable measurement.
Always present zero-order correlations.
3) Model
Endogeneity
“Under control” and your coefficient is no longer interpretable
Endogeneity is when your your error term is associated with an IV.
Regression models are considered robust meaning that even when you violate assumptions, you can still use the same models with some safety.
E.g., t-tests are robust to the assumption of normality, because we can fall back on the central limit theorem.
Regression is robust to violations of some assumptions, primarily
Homoscedasticity
Normality of errors
Steps for screening
Calculate univariate and bivariate descriptive stats
Check the class (factor or numeric) of the variable
Check the min and max
make sure data were entered correctly
look for ceiling or floor effects
Check for skew
Compare correlation matrices with pairwise and listwise deletion for bias in missingness.
Calculate reliability for your scales.
Steps for screening
Plot univariate and bivariate distributions
Look for skew and outliers
Check correlation heat maps for expected and unexpected patterns in items
Steps for screening
Test the assumptions of your regression model(s)
Calculate the VIF of each term (if you have two or more) to check for correctly specified models.
Graph residuals by predictors to check for Endogeneity.
Graph residuals by fitted values to check for homoscedasticity.
Graph residuals by ID number (or date, or another variable not in your model) to check for independence.
Graph the distribution of residuals or the Q-Q plot to check for normality.
Steps for screening
Look for univariate or multivariate outliers…
Outliers
Broadly defined as atypical or highly influential data point(s)
Due to contamination (e.g. recording error) or accurate observation of a rare case
Univariate vs. Multivariate
How do we typically describe or identify outliers?
Outliers can be described in terms of three different metrics. Each metric conveys a sense of the magnitude of outliery-ness the case exhibits. However, some metrics also describe the degree to which your inferences will change:
Leverage: How unusual is this case from the rest of the cases in terms of predictors?
Distance: How distant is the observed case from the predicted value?
Influence: How much the does regression coefficient change if case were removed?
Leverage
How far observed values for a case are from mean values on the set of IVs (centroid).
Not dependent on Y values
High leverage cases have greater potential to influence regression results
Leverage
One common metric for describing leverage is Mahalanobis Distance, which is the multidimensional extension of Euclidean distance where vectors are non-orthogonal. Given a set of variables, \(\mathbf{X}\) with means \(\mathbf{\mu}\) and covariance \(\Sigma\):
Distance is the distance from prediction, or how far a case’s observed value is from its predicted value
i.e., residual
In units of Y.
What might be problematic at looking at residuals in order to identify outliers?
Distance
Raw residuals come from a model that is influenced by the outliers, making it harder to detect the outliers in the first place. To avoid this issue, it is advisable to examine the deleted residuals.
This value represents the distance between the observed value from a predicted value that is calculated from a regression model based on all data except the case at hand
The leave-one-out procedure is often referred to as a “jack-knife” procedure.
Distance
Standardized residuals: takes raw residuals and puts them in a standardized unit – this can be easier for determining cut-offs.
\[\large z_e = \frac{e_i}{\sqrt{MSE}}\]
Distance
Other residuals are available:
Studentized residuals: The MSE is only an estimate of error and, especially with small samples, may be off from the population value of error by a lot. Just like we use a t-distribution to adjust our estimate of the standard error (of the mean) when we have a small sample, we can adjust out precision of the standard error (of the regression equation).
\[\large r_i = \frac{e_i}{\sqrt{MSE}(1-h_i)}\]
where \(h_i\) is the ith element on the diagonal of the hat matrix, \(\mathbf{H} = \mathbf{X}(\mathbf{X'X})^{-1}\mathbf{X}'\). As N gets larger, the difference between studentized and standardized residuals will get smaller.
Distance
Warning: Some textbooks (and R packages) will use terms like “standardized” and “studentized” to refer to deleted residuals that have been put in standardization units; other books and packages will not. Sometimes they switch the terms and definitions around. The text should tell you what it does.
?MASS::studres
library(olsrr)ols_plot_resid_stud_fit(model.2)
Influence
Influence refers to how much a regression equation would change if the extreme case (outlier) is removed.
Influence = Leverage X Distance
Like distance, there are several metrics by which you might assess any case’s leverage. The most common are:
Cook’s Distance (change in model fit)
DFFITS (change in fitted value, standardized)
DFBETAS (change in coefficient estimate)
Influence
DFFITS indexes how much the predicted value for a case changes if you remove the case from the equation.
DFBETAs index how much the estimate for a coefficient changes if you remove a case from the equation.
Leverage – How unusual is this case from the rest of the cases in terms of predictors?
Distance – How distant is the observed case from the predicted value?
Influence – How much the does regression coefficient change if case were removed?
Consider each of these metrics in the context of research degrees of freedom. Do some of these metrics change the independence of successive inferential tests? How might this be problematic for Type I error?
What can you do to ensure your analysis is robust?
Recommendations
Analyze data with/without outliers and see how results change
If you throw out cases you must believe it is not representative of population of interest or have appropriate explanation
Don’t throw out data just to be “safe”. Data are hard to collect and outliers are expected!
Multicollinearity
Multicollinearity occurs when predictor variables are highly related to each other. - This can be a simple relationship, such as when X1 is strongly correlated with X2. This is easy to recognize, interpret, and correct for. - Sometimes multicollinearity is difficult to detect, such as when X1 is not strongly correlated with X2, X3, or X4, but the combination of the latter three is a strong predictor of X1.
Multicollinearity
Multicollinearity increases the standard errors of your slope coefficients.
\[\large se_{b} = \frac{s_{Y}}{s_{X}}\sqrt{\frac {1-R_{Y\hat{Y}}^2}{n-p-1}}\sqrt{\frac{1}{1-R_{12}^2}}\] - Perfect collinearity never happens. Like everything in statistics (except rejecting the null), it’s never a binary situation; there are degrees of multicollinearity. More multicollinearity = more problematic model.
Diagnosis
Multicollinearity can be diagnosed with tolerance.
Tolerance: \(1-R_{12}^2\)
Also look for models in which the coefficient of determination is large and the model is significant but the slope coefficients are small and non-significant.
Look for unstable slope coefficients (i.e., large standard errors)
Look for sign changes
Multicollinearity
Ways to address
Increase sample size
Remember, with small samples, our estimates can be wildly off. Even if the true relationship between X1 and X2 is small, the sample correlation might be high because of random error.
Remove a variable from your model.
Composite or factor scores
If variables are highly correlated because they index the same underlying construct, why not just use them to create a more precise measure of that construct?
