Introduction to econometrics
III. Simple linear regression model
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 1 / 54
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 2 / 54
Simple regression model
No intercept.
No need of matrix algebra.
Yi = βXi + i .
Koop (2008) - chapter 3.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 3 / 54
A review of basic concepts in probability
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 4 / 54
A review of basic concepts in probability
Randomness of dependent variable
Uncertainty expressed by probability density function.
House prices example – N(61.153, 683.812).
Koop (2008) - figure 3.1.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 5 / 54
A review of basic concepts in probability
Normal p.d.f. of house price (lot size = 5000)
−50 0 50 100 150 200
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
Cena domu (tis. kanadských dolarù)
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 6 / 54
A review of basic concepts in probability
Normal p.d.f. of house price – density interval
−40 −20 0 20 40 60 80 100 120 140 160 180 200
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
Cena domu (tis. kanadskych dolaru)
Pr(60<Y<100)
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 7 / 54
A review of basic concepts in probability
Z-score
Y ∼ N(µ, σ2) and new random variable:
Z =
Y − µ
σ
.
Expected value and variance of Z:
E(Z) = E
Y − µ
σ
=
E(Y − µ)
σ
=
E(Y ) − µ
σ
=
µ − µ
σ
= 0;
var(Z) = var
Y − µ
σ
=
var(Y − µ)
σ2
=
var(Y )
σ2
=
σ2
σ2
= 1.
Z-score: Z ∼ N(0, 1).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 8 / 54
A review of basic concepts in probability
Z-score – example
Figure: Y ∼ N(61.153, 683.812).
Shaded area: Pr(60 ≤ Y ≤ 100).
Pr(60 ≤ Y ≤ 100) = Pr
60 − µ
σ
≤
Y − µ
σ
≤
100 − µ
σ
= Pr
60 − 61.153
√
683.812
≤
Y − 61.153
√
683.812
≤
100 − 61.153
√
683.812
= Pr (−0.04 ≤ Z ≤ 1.49) .
Using statistical tables: F(1.49) − F(−0.04) = 0.4479.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 9 / 54
Classical assumptions for LRM
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 10 / 54
Classical assumptions for LRM
Classical assumption using dependent variable
1 E(Yi ) = βXi .
2 var(Yi ) = σ2.
3 cov(Yi , Yj) = 0 pro i = j.
4 Yi is normally distributed.
5 Xi is fixed (not a random variable).
Compact notation:
Yi ∼ N(βXi , σ2
),
for i = 1, . . . , N, where Yi a Yj are uncorrelated with one another if
i = j.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 11 / 54
Classical assumptions for LRM
Expected value of dependent variable
E(Yi ) = βXi : the linearity assumption
Multiple regression model: E(Yi ) = α + β1X1i + β2X2i + . . . + βkXki .
We expect (in average) Yi to lie on the regression line.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 12 / 54
Classical assumptions for LRM
Alll observation have the same variance
Violated in practice (house price example – prices of small and big
houses; consumption example – households with different incomes).
Homoskedasticity × heteroskedasticity.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 13 / 54
Classical assumptions for LRM
Different observations are uncorrelated
Remember:
corr(Yi , Yj) =
cov(Yi , Yj)
var(Yi )var(Yj)
.
cov(Yi , Yj) = 0 ⇔ corr(Yi , Yj) = 0.
Not a problem for cross-sectional data.
Time series – correlation over tima (e.g. unemployment).
Autocorrelation.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 14 / 54
Classical assumptions for LRM
Dependent variable normally distributed
Hard to motivate.
In many empirical applications – a reasonable assumption.
Asymptotic theory to relax this assumption.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 15 / 54
Classical assumptions for LRM
Fixed explanatory variables
Experimental sciences – reasonable assumption.
Economic research – usually not an experimental science.
Asymptotic theory to relax this assumption (explanatory variables
random but uncorrelated with the error term.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 16 / 54
Classical assumptions for LRM
Classical assumptions using error term
1 E( i ) = 0.
2 var( i ) = E( 2
i ) = σ2. Constant variability (homoskedasticity).
3 cov( i , j) = 0 for i = j. Errors are uncorrelated.
4
i is normally distributed.
5 Xi is fixed (not random variable).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 17 / 54
Classical assumptions for LRM
Violated classicalassumption.
Problem → OLS results not correct.
Testing using residuals (usually).
Solution – asymptotic theory (large samples) or other estimation
methods.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 18 / 54
Properties of OLS estimator
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 19 / 54
Properties of OLS estimator
Estimaor and estimate
Estimator – a function of possible observations.
Estimate – evaluating the estimator using data.
Classical econometrics – unknown parameter is a scalar number × its
estimate is a random variable.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 20 / 54
Properties of OLS estimator
OLS estimator
No intercept:
Yi = βXi + i .
Minimizing sum of squared resiuals:
SSR =
N
i=1
2
i → min .
Straightforward calculus problem⇒
β =
N
i=1 Xi Yi
N
i=1 X2
i
.
β – normally distibuted (linear function in Yi = normally distributed
random variables).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 21 / 54
Properties of OLS estimator
Alternative expression for OLS estimator
Replacing Yi → true parameter + termi involving the explanatory
variables and errors.
Not useful for evaluating OLS estimator in practice.
β =
Xi Yi
X2
i
=
Xi (Xi β + i )
X2
i
= β +
Xi i
X2
i
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 22 / 54
Properties of OLS estimator
Expected value of the OLS estimator
Unbiased estimator – „in average“ equals its true value.
E β = β.
E β = E β +
Xi i
X2
i
= β + E
Xi i
X2
i
= β +
1
X2
i
E Xi i = β +
1
X2
i
Xi E( i )
= β.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 23 / 54
Properties of OLS estimator
Variance of the OLS estimator
To quantify the variablitity of the random variable (to evaluate the
effectivness of the estimator).
var β =
σ2
X2
i
.
var β = var β +
Xi i
X2
i
= var
Xi i
X2
i
=
1
X2
i
2
var Xi i =
1
X2
i
2
X2
i var( i )
=
1
X2
i
2
σ2
X2
i =
σ2
X2
i
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 24 / 54
Properties of OLS estimator
Efficiency of the estimator
Many unbiased estimator → choose the most efficient (the best),
with the smallest variance.
LInear regression model without intercept:
˜β =
N
i=1 Yi
N
i=1 Xi
.
Unbiased wstimator, has a larger variance than OLS estimator.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 25 / 54
Properties of OLS estimator
Distribution of the OLS estimator
Under classical assumptions:
β ∼ N β,
σ2
X2
i
.
To derive confidence intervals and for hypothesis testing.
Příklad: β ∼ N(2, 1) a ˜β ∼ N(2, 4).
Using statistical tables: e.g. Pr(1.0 ≤ β ≤ 3.0) = 0.68 or
Pr(0 ≤ β ≤ 1) = 0.14.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 26 / 54
Properties of OLS estimator
The p.d.f. of the OLS estimator
−2 −1 0 1 2 3 4 5 6 7
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Hodnota OLS estimatoru
β=2
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 27 / 54
Properties of OLS estimator
OLS estimator and a less efficient estimator
−6 −4 −2 0 2 4 6 8 10 12
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Hodnota estimatoru
Funkce hustoty
OLS estimatoru
Funkce hustoty
jineho estimatoru
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 28 / 54
Properties of OLS estimator
Gauss-Markov theorem
If the classical assumptions hold (not necessary normality): OLS
estimator is BLUE (Best Linear Unbiased Estimator).
Carl Friedrich Gauss (1777–1855) Andrej Markov (1856–1922)
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 29 / 54
Properties of OLS estimator
Gauss-Markov theorem – unbiasness
Simple LRM, no intercept, all assumptions hold.
Linear estimator: β∗ = c1Y1 + . . . + cNYN for constants c1, . . . , cN.
Unbiasness: E (β∗) = β.
E (β∗
) = E (c1Y1 + . . . + cNYN)
= c1E (Y1) + . . . + cNE (YN)
= c1βX1 + . . . + cNβXN
= β
N
i=1
ci Xi .
Unbiased estimator, β∗ – it holds N
i=1 ci Xi = 1.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 30 / 54
Properties of OLS estimator
Gauss-Markov theroem – effectivness
var (β∗
) = var (c1Y1 + . . . + cNYN)
= c2
1 var (Y1) + . . . + c2
Nvar (YN)
= c2
1 σ2
+ . . . + c2
Nσ2
= σ2
N
i=1
c2
i .
Looking for c1, . . . , cN to minimize σ2 N
i=1 c2
i subject to constraint
N
i=1 ci Xi = 1.
Standard calculus problem.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 31 / 54
Properties of OLS estimator
Gauss-Markov theorem – solution
Solution: cj =
Xj
N
i=1 X2
i
, pro j = 1, . . . , N.
Plugging this into expression for β∗:
β∗
= c1Y1 + . . . + cNYN
=
X1Y1
X2
i
+ . . . +
XNYN
X2
i
=
Xi Yi
X2
i
= β.
Conclusion: linear unbiased estimator with minimum variance is the
OLS estimator.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 32 / 54
Maximal likelihood method
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 33 / 54
Maximal likelihood method
Motivation
Maximal likelihood (ML).
Probability density function for Y should be in accordance with
observed data.
Example – house prices (for different parameter values β).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 34 / 54
Maximal likelihood method
Motivation – figure
−150 −100 −50 0 50 100 150 200 250 300 350
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
Cena domu (tis. kanadskych dolaru)
β=0.040β=−0.010 β=0.012
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 35 / 54
Maximal likelihood method
Likelihood function funkce
Likelihood function: joint probability density function for all
observation.
Under classical assumption: Y1, . . . , YN are normally distributed,
uncorrelated random variables (mean βXi and variance σ2).
P.D.F. for each random variable: p(Yi |Xi , β) (conditional densities).
Joint p.d.f for all observations:
p (Y1, . . . , YN) =
N
i=1
p (Yi |Xi , β) .
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 36 / 54
Maximal likelihood method
ML estimate
We know Xi and Yi ⇒ likelihood function depends on parameters –
L(β).
ML estimate = selecting β that maximizes L(β) (in case of LRM –
without numerical optimalization).
p (Yi |Xi β) =
1
√
2πσ2
exp −
1
2σ2
(Yi − βXi )2
.
L(β) =
N
i=1
1
√
2πσ2
exp −
1
2σ2
(Yi − βXi )2
=
1
(2πσ2)
N
2
exp −
1
2σ2
N
i=1
(Yi − βXi )2
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 37 / 54
Maximal likelihood method
Maximizing likelihood function.
Working with loga-likelihood, l(β) (maximum is not changed).
l(β) = ln [L(β)]
= ln
1
(2πσ2)
N
2
exp −
1
2σ2
N
i=1
(Yi − βXi )2
= ln
1
(2πσ2)
N
2
−
1
2σ2
N
i=1
(Yi − βXi )2
.
The value of β that maximizes l(β) using calculus → one nedd to
minimize N
i=1 (Yi − βXi )2
→ OLS estimator.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 38 / 54
Confidence intervals and hypothesis testing
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 39 / 54
Confidence intervals and hypothesis testing
Confidence interval for the parameter
β is normally distributed with the mean β and a given variance
(explained later).
To construct Z-skóru:
Z =
β − E(β)
var(β)
=
β − β
σ2
x2
i
.
Z ∼ N(0, 1) ⇒ using statistical tables for standardized normal
distribution.
For example:
Pr [−1.96 ≤ Z ≤ 1.96] = 0.95.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 40 / 54
Confidence intervals and hypothesis testing
Deriving confidence interval
95% confidence interval → to put β in the middle:
Pr



−1.96 ≤
β − β
σ2
x2
i
≤ 1.96




= Pr −1.96
σ2
x2
i
≤ β − β ≤ 1.96
σ2
x2
i
= Pr β − 1.96
σ2
x2
i
≤ β ≤ β + 1.96
σ2
x2
i
= 0.95.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 41 / 54
Confidence intervals and hypothesis testing
Alternative expressions
95% confidence interval:
β − 1.96
σ2
x2
i
≤ β ≤ β + 1.96
σ2
x2
i
Or:
β ± 1.96
σ2
x2
i
β − 1.96
σ2
x2
i
, β + 1.96
σ2
x2
i
.
„95 %“: confidence level.
Other confidence levels can be handled beginning with a different
probability (e.g. 90 %) ⇒ other „critical values“ (e.g. 1.64).
Practical problem: unknown error variance, σ2.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 42 / 54
Confidence intervals and hypothesis testing
General steps on hypothesis testing
1 Specify a null hypothesos, H0 and an alternative hypothesis H1.
2 Specify test statistic.
3 Figure out the distribution of the test statistic, assuming H0 is true.
4 Choose a level of significance.
5 Use steps 3 and 4 to obtain critical value.
6 Calculate your test statistics from step 2 and compare with the
critical value from step 5:
Reject H0 if the absolut value of the test statistic is greater than the
critical value (else do not reject H0.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 43 / 54
Confidence intervals and hypothesis testing
Hypothesis test about a parameter
1 H0 : β = β0, where β0 is known (usually β0 = 0). Alternative
hypothesis is usually H1 : β = β0 (one-sided hypothesis possible =
other critical value).
2 Z =
β − β
σ2
X2
i
.
3 Z =
β − β0
σ2
X2
i
∼ N(0, 1).
4 Common choice 5 % (i.e. 0.05).
5 Since Z ∼ N(0, 1) and Pr [−1.96 ≤ Z ≤ 1.96] = 0.95, the critical
value is 1.96.
Two-sided test (like two-sided confidence interval) → critical value =
97.5% (0.975) quantile of standardized normal distribution
(1 − 0.05/2).
6 In our case: reject H0 if |Z| > 1.96.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 44 / 54
Confidence intervals and hypothesis testing
Error variance estimator
Commonly used estimator for σ2 is reffered as s2 (sample variance).
OLS residuals: i = Yi − βXi ; OLS estimator σ2: s2 =
2
i
N−1 .
May be shown that E(s2) = σ2.
Intuition: E 2
i = σ2 ⇒ 2
i might be a good estimator for σ2.
Use all errors → sample average of squared errors to estimate σ2:
2
i
N
.
Substitute unobserved i :
2
i
N (biased estimator for σ2 – ML
estimator).
Unbiased OLS estimator: N replaced by N − 1 (degrees of freedom)
Multiple regression with k explanatory variables and with the
intercept term:
s2
=
2
i
N − k − 1
=
SSR
N − k − 1
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 45 / 54
Confidence intervals and hypothesis testing
Modifications when error variance is unknown
Construct Z-score:
Z =
β − β
s2
X2
i
∼ tN−1,
Hypothesis testing and test statistics:
t =
β − β0
s2
X2
i
∼ tN−1.
tN−1 is Student t-distribution with N − 1 degrees of freedom.
Statistical tables for t-distribution or p-value for the test (reject H0 if
p-value is less than significance level).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 46 / 54
Using asymptotic theory
Content
1 A review of basic concepts in probability
2 Classical assumptions for LRM
3 Properties of OLS estimator
4 Maximal likelihood method
5 Confidence intervals and hypothesis testing
6 Using asymptotic theory
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 47 / 54
Using asymptotic theory
Motivation
What happens in large (infinite) samples?
No assumpotion about what the p.d.f. of the errors is.
Xi is independent and identically distributed – i.i.d., independent of
error terms ⇒ E(Xi ) = µX and var(Xi ) = σ2
X .
Use OLS estimator:
β = β +
Xi i
X2
i
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 48 / 54
Using asymptotic theory
Consistency of OLS estimator – introduction
plim(β) = β.
plim β = plim β +
Xi i
X2
i
= β + plim
Xi i
X2
i
by Slutsky’s theorem
= β + plim
1
N Xi i
1
N X2
i
= β +
plim 1
N Xi i
plim 1
N X2
i
by Slutsky’s theorem.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 49 / 54
Using asymptotic theory
Consistency of OLS estimator (continued)
Law of large numbers to figure out the plim 1
N Xi i .
„Average converge to expected values“.
Errors and explanatory variables are assumed to be independent of
one another:
plim
1
N
Xi i = E(Xi i ) = 0
Law of large numbers → plim 1
N X2
i = E(X2
i ).
From definition of the variance, var(Xi ) = E(X2
i ) − [E(Xi )]2
, we can
write E(X2
i ) = var(Xi ) + [E(Xi )]2
= σ2
X + µ2
X ⇒
plim β = β +
0
σ2
X + µ2
X
= β.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 50 / 54
Using asymptotic theory
Asymptotic normality – introduction
As N → ∞ we have:
√
N(β − β) ∼ N 0,
σ2
σ2
X + µ2
X
.
Equation can be written:
√
N(β − β) =
√
N
Xi i
X2
i
=
√
N
1
N Xi i
1
N X2
i
.
Central limit theorem as N → ∞:
√
N
1
N
Xi i ∼ N (0, var(Xi i )) .
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 51 / 54
Using asymptotic theory
Asymptotic normality (continued)
var(Xi i ) = E(X2
i
2
i ) − [E(Xi i )]2
= E(X2
i )E( 2
i ) − [E(Xi )E( i )]2
= (σ2
X + µ2
X )σ2
− [µX 0]2
= (σ2
X + µ2
X )σ2
.
We showed:
plim
1
N
X2
i = (σ2
X + µ2
X ).
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 52 / 54
Using asymptotic theory
Asymptotic normality (end)
Using Cramer’s theorem we can combine the result form the central
limit theorem with the forms for var (Xi i ) and plim
1
N
X2
i :
√
N(β − β)
N
→ N 0,
(σ2
X + µ2
X )σ2
(σ2
X + µ2
X )2
.
Cancelling out the common factor in the variance:
√
N(β − β)
N
→ N 0,
σ2
(σ2
X + µ2
X )
.
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 53 / 54
Using asymptotic theory
Using the asymptotic results in practice
As N → ∞,
√
N(β − β) converges to a normal distribution.
Using the properties of the expected value and variance operators:
β ∼ N β,
σ2
N(σ2
X + µ2
X )
.
Problem: (σ2
X + µ2
X ) unknown.
It holds plim 1
N X2
i = σ2
X + µ2
X ⇒ 1
N X2
i is a consistent
estimator of σ2
X + µ2
X .
Main result:
β ∼ N β,
σ2
X2
i
.
All the derivations and results for finite samples hold (approximately)!
Introduction to econometrics (INEC) III. Simple regression model Autumn 2011 54 / 54