09. Correlation, Linear Regression

R101

Outline

01. Correlation (Pearson’s r)

02. Linear Regression

Data and packages

Hate Crimes

• Packages:

ipak <- function(pkg){
  new.pkg <- pkg[!(pkg %in% installed.packages()[, "Package"])]
  if (length(new.pkg)) 
    install.packages(new.pkg, dependencies = TRUE)
  sapply(pkg, require, character.only = TRUE)
}

packages <- c("ggplot2", "dplyr")
ipak(packages)

• Create a subset of the dataframe:

Hate_Crimes_selected <- read.csv("hate_crimes.csv") %>% 
  select(c(share_voters_voted_trump, share_white_poverty, median_household_income))

Correlation

Pearson product-moment correlation coefficient:

Předpoklady použití :

• Alespoň intervalová úroveň měření proměnných
• Normálně rozložená data
• Homoskedascita

• Normalita rozložení:

hist(Hate_Crimes_selected$share_voters_voted_trump, 
     main="Histogram podílu voličů Donalda Trumpa", 
     xlab="Podíl (v %)", 
     border="red", 
     col="darkblue", 
     xlim=c(0,1), 
     breaks=20)

• Homoskedascita:

plot(Hate_Crimes_selected$share_voters_voted_trump,
Hate_Crimes_selected$median_household_income, 
main="Scatterplot podílu voličů Donalda Trumpa \n a mediánu ročního příjmu domácnosti", 
xlab="Podíl voličů Donalda Trumpa", 
ylab="Medián ročního
příjmu domácnosti (v $)", 
pch=19)

• Normal probability plot:

qqnorm(Hate_Crimes_selected$share_voters_voted_trump)

base

• Compute the correlations among share_white_poverty, share_voters_voted_trump and median_household_income:

cor(Hate_Crimes_selected)

                         share_voters_voted_trump share_white_poverty
share_voters_voted_trump                1.0000000           0.5528492
share_white_poverty                     0.5528492           1.0000000
median_household_income                -0.5975281          -0.8180451
                         median_household_income
share_voters_voted_trump              -0.5975281
share_white_poverty                   -0.8180451
median_household_income                1.0000000

• The cor() function does not calculate p-values to test for significance, but the cor.test() function does:

cor.test(Hate_Crimes_selected$share_voters_voted_trump, Hate_Crimes_selected$share_white_poverty, use = pairwise)

    Pearson's product-moment correlation

data:  Hate_Crimes_selected$share_voters_voted_trump and Hate_Crimes_selected$share_white_poverty
t = 4.6442, df = 49, p-value = 2.589e-05
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.3271014 0.7189034
sample estimates:
      cor 
0.5528492 

cor.test(Hate_Crimes_selected$share_white_poverty, Hate_Crimes_selected$median_household_income, use = pairwise)

    Pearson's product-moment correlation

data:  Hate_Crimes_selected$share_white_poverty and Hate_Crimes_selected$median_household_income
t = -9.9561, df = 49, p-value = 2.329e-13
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.8924377 -0.7003477
sample estimates:
       cor 
-0.8180451 

cor.test(Hate_Crimes_selected$share_voters_voted_trump, Hate_Crimes_selected$median_household_income, use = pairwise)

    Pearson's product-moment correlation

data:  Hate_Crimes_selected$share_voters_voted_trump and Hate_Crimes_selected$median_household_income
t = -5.2163, df = 49, p-value = 3.669e-06
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.7496651 -0.3854093
sample estimates:
       cor 
-0.5975281 

Rcmdr

• The rcorr.adjust() function of the Rcmdr package computes the correlations with the pairwise p-values among the correlations:

library("Rcmdr")

• Two types of p-values are computed: the ordinary p-values and the adjusted p-values:

RcmdrMisc::rcorr.adjust(Hate_Crimes_selected)

 Pearson correlations:
                         share_voters_voted_trump share_white_poverty
share_voters_voted_trump                   1.0000              0.5528
share_white_poverty                        0.5528              1.0000
median_household_income                   -0.5975             -0.8180
                         median_household_income
share_voters_voted_trump                 -0.5975
share_white_poverty                      -0.8180
median_household_income                   1.0000

 Number of observations: 51 

 Pairwise two-sided p-values:
                         share_voters_voted_trump share_white_poverty
share_voters_voted_trump                          <.0001             
share_white_poverty      <.0001                                      
median_household_income  <.0001                   <.0001             
                         median_household_income
share_voters_voted_trump <.0001                 
share_white_poverty      <.0001                 
median_household_income                         

 Adjusted p-values (Holm's method)
                         share_voters_voted_trump share_white_poverty
share_voters_voted_trump                          <.0001             
share_white_poverty      <.0001                                      
median_household_income  <.0001                   <.0001             
                         median_household_income
share_voters_voted_trump <.0001                 
share_white_poverty      <.0001                 
median_household_income                         

Linear Regression

• Jak to, že děti vysokých rodičů samy bývají vysoké, ale ne tak jako jejich rodiče?
• Jak to, že děti útlých rodičů samy bývají útlé, ale ne tak útlé jako jejich rodiče?
• Jak to, že nejlepší atlet minulé sezóny letos podává o něco horší výkon než loni?

Regression towards mediocrity

“It appeared from these experiments that the offspring did not tend to resemble their parent seeds in size, but to be always more mediocre than they-to be smaller than the parents, if the parents were large; to be larger than the parents, if the parents were very small.”

“The point of convergence was considerably below the average size of the seeds contained in the large bagful I bought at a nursery garden, out of which I selected those that were sown, and I had some reason to believe that the size of the seed towards which the produce converged was similar to that of an average seed taken out of beds of self-planted specimens.”

Galton, 1886, s. 246

• Lineární regrese
  • Nakolik lze z IQ skóru usuzovat o výkonu v matematice?
    • Predikce

• Vícenásobná lineární regrese
  • Přispívá k výši platu kromě úrovně vzdělání také pohlaví?
    • Predikce
    • Inkrementální validita
    • Statistická kontrola

Notace

\(Y = Y' + e\)

• Lineární regrese:

\(Y' = a + bX\)

\(Y' = b_{0} + b_{1}X_{1}\)

• Vícenásobná lineární regrese:

\(Y' = b_{0} + b_{n}X_{n}\)

\(Y' = b_{0} + b_{1}X_{1} + b_{2}X_{2} + ... + b_{n}X_{n}+ e\)

• \(Y\) = Outcome (= dependent) variable
• \(Y'\) = Our model
• \(e\) = Error
• \(a\) or \(b_{0}\) = Intercept
• \(b\) or \(b_{1...n}\) = Slope
• \(X_{1...n}\) = Predictor (= independent variable)

Přímka (model) je proložena daty tak, aby jim co nejlépe odpovídala.

• Metoda odhadu nejmenších čtverců (Least Squares Estimation)
  • Suma (druhých mocnin) vzdáleností modelu od dat je nejmenší možná

\(S_{T}\)\(^{2}\) = \(S_{M}\)\(^{2}\) + \(S_{R}\)\(^{2}\)

• Neboli \(SS_{T} = SS_{res} + SS_{reg}\)

\(R^{2}\) = \(SS_{M}\)\(^{2}\) + \(SS_{T}\)\(^{2}\)

\(SS_{M}\)

• Rozdíl mezi nulovým modelem (průměr \(Y\)) a námi stanoveným modeleme (přímkou).

\(SS_{R}\)

• Rozdíl mezi daty a námi stanoveným modelem (přímkou).

\(SS_{T}\)

• Rozdíl mezi daty a nulovým modelem (průměr \(Y\)).

\(R^{2}\)

• Podíl rozptylu závislé (outcome) proměnné vysvětlené modelem (= koeficient determinance).

Metoda nejmenších čtverců graficky

Model

ModelHateCrime <- lm(formula = share_voters_voted_trump ~ share_white_poverty, data = Hate_Crimes_selected)

• Compute the summary statistics for model:

# Generic functions (summary) change their behaviour based on an object's class.

summary(ModelHateCrime)

Call:
lm(formula = share_voters_voted_trump ~ share_white_poverty, 
    data = Hate_Crimes_selected)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.312542 -0.052037  0.007795  0.061240  0.214686 

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)           0.2463     0.0543   4.536 3.72e-05 ***
share_white_poverty   2.6554     0.5718   4.644 2.59e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.09992 on 49 degrees of freedom
Multiple R-squared:  0.3056,    Adjusted R-squared:  0.2915 
F-statistic: 21.57 on 1 and 49 DF,  p-value: 2.589e-05

• Perform an analysis of variance on model:

anova(ModelHateCrime)

Analysis of Variance Table

Response: share_voters_voted_trump
                    Df  Sum Sq  Mean Sq F value    Pr(>F)    
share_white_poverty  1 0.21536 0.215355  21.569 2.589e-05 ***
Residuals           49 0.48924 0.009985                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

• Predict based on the fitted function:

predict(ModelHateCrime)

        1         2         3         4         5         6         7         8         9 
0.5649769 0.4056510 0.4853139 0.5649769 0.4853139 0.4322053 0.4056510 0.4587596 0.3525424 
       10        11        12        13        14        15        16        17        18 
0.5384226 0.4853139 0.4322053 0.5384226 0.4322053 0.5649769 0.4853139 0.5384226 0.6977485 
       19        20        21        22        23        24        25        26        27 
0.5649769 0.5649769 0.4056510 0.4587596 0.4853139 0.3790967 0.6180855 0.4322053 0.5118683 
       28        29        30        31        32        33        34        35        36 
0.4322053 0.4587596 0.4056510 0.4322053 0.5118683 0.5118683 0.5118683 0.4853139 0.5118683 
       37        38        39        40        41        42        43        44        45 
0.5118683 0.5118683 0.4853139 0.4587596 0.4853139 0.4587596 0.5915312 0.4587596 0.4587596 
       46        47        48        49        50        51 
0.5118683 0.4322053 0.4853139 0.6180855 0.4853139 0.4853139 

Coefficients

\(b_{i}\)

• Vyjadřuje nárůst \(Y’\) při nárůstu \(X_{i}\) o jednu jednotku v jednotkách Y, při kontrole všech ostatních prediktorů (tj. semiparciální korelace); jedinečný přínos
  • K porovnání síly prediktoru v různých skupinách, modelech, vzorcích

\(β_{i}\)

• Beta vyjadřuje nárůst \(Y’\) při nárůstu \(X_{i}\) o 1; pokud jsou \(X_{i}\) a \(Y\) standardizovány, při kontrole všech ostatních prediktorů (tj. semiparciální korelace), jedinečný přínos
  • K porovnání prediktorů mezi sebou v rámci jednoho modelu
  • K porovnání různě operacionalizovaného prediktoru v různých modelech
  • Ukazatel velikosti účinku

\(b_{0}\)

• Po vycentrování (odečtení průměru od všech hodnot \(X_{i}\)) odpovídá průměru \(Y\).

QuantPsyc::lm.beta(ModelHateCrime)

share_white_poverty 
          0.5528492 

Resources

American Psychological Association. (2001). Publication manual of the American Psychological Association (6th ed.). Washington, DC: APA.

Field, A. (2009). Discovering statistics using SPSS, 3th Ed. Los Angeles: Sage.

Fox, J. (2016). Applied Regression Analysis and Generalized Linear Models, 3th Ed. Los Angeles: Sage.

Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute, 15, pp. 246-63. Dostupné online z “http://galton.org/essays/1880-1889/galton-1886-jaigiregression-stature.pdf”

Robotková, A., & Ježek, S. (2012). Vícenásobná lineární regrese. Prezentace ke kurzu PSY252.

 

---

A work by Vit Gabrhel

vit.gabrhel@gmail.com