Bootstrap

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals Bootstrap for Regression

• Sampling distribution of θ^ˆis hard to derive.

⁾Or, we just don’t feel like taking the time to.

• Our sample size is not tiny but is too small for “asymptopia”.

Nonparametric bootstrap: Let Y= [Y, . . . , Y] ^iidF

1. ) (some unknown CDF).

1 ^n ∼Y

1. • F^ˆ(y)

   n i=1 Let F^ˆ_n(y) = P^{^}r(Y≤ y) = ^1ΣnI(Yi≤ y) (fraction of the sample ≤ y)

⇒ = The distribution of the sample approximates the distribution of the population as nincreases.

⇒ = Sampling from a sufficiently large sample approximates sampling from the population.

^{Parametricbootstrap:► Ifanestimatorθˆ}n^{isconsistentandthemodeliscorrect,}

^p(·|θˆn^{)approximatesp(·|θ).}

^{=⇒Samplingfromp(·|θˆ}n^{)approximatessamplingfromthepopulation.}

Notation: Y ^s = [Y ^s , . . . , Y ^s ] the bth “resample” with replacement from observed

^b b1 bn _ˆ

_ˆ ^{vector Y or p(·|θ}n⁾

≡ θ(FY) θ the true parameter value or some population quantity function of

FY

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals Bootstrap for Regression

Let Y₁, . . . , Y

iid

f^n ∼Y

(y; θ) = θe^−θy, for y> 0 and θ > 0.

Σn

l(θ) = (log θ − θYi) = nlog θ − θnY^¯

i =1

Σ_► But,

n i=1

Yi∼ Ga(a= n, b= θ)

θ^ˆ= (Y^¯)⁻¹

• Y^¯∼ Ga(a= n, b= nθ)
• n 1 E(θ^ˆ) = E(1/Y^¯) = EInvGa(a= n, b= nθ) = ⁿθ > θ

Also, by Jensen’s Inequality, −

• E(Y^¯) = 1/θ (a convex function)

∴ E(θ^ˆ) = E(1/Y^¯) > 1/ E(Y^¯) = θ.

Recall,

FY population from which Y are drawn

θ^ˆ(FY) population quantity

θ^ˆ_n(Y) some estimator of θ^ˆ(FY)

biasF_Y[θ^ˆ_n(Y)] = EF_Y[θ^ˆ_n(Y)] − θ^ˆ(FY)

• An estimator θ^ˆ_n(Y) is biasedif biasF_Y[θ^ˆ_n(Y)] ƒ= 0
• Note that Wakefield (2013, Sec. 2.7.1) implicitly defines it the other way, as

θ^ˆ(FY) − EF_Y[θ^ˆ_n(Y)].

• Jensen’s inequality: If it happens that E(Y) = g(θ) for some g, and θ^ˆ= g⁻¹(Y^¯), and g(y) is concave or convex, θ^ˆwill be biased for θ.

0.5

n <- 20 # Samplesize

theta.true <- 1 #Exponentialdistributiontruerateparameter

S <- 1000 # Number of replications for the simulation

B <- 999 #Bootstrapresamplesize

#Afunctiontogenerateadataset

mk.data <- function(n, theta=theta.true) rexp(n,theta)

# MLE

theta.mle <- function(y) 1/mean(y)

# MLE sampling distribution

theta.mle.dist <- replicate(S, y.obs <- mk.data(n) theta.mle(y.obs)

)

mean(theta.mle.dist) # Mean of the sampling distribution

## [1] 1.047768

mean(theta.mle.dist)-theta.true #Bias

## [1] 0.04776839

1.0 1.5 2.0 * = mean

b

b

• Suppose we have a biased estimator:

biasF_Y[θ^ˆ_n(Y)] = E[θ^ˆ_n(Y)] − θ^ˆ(FY) =ƒ 0.

⁾ We don’t know either expectation.

• ^{Idea:treatFˆ}n^asF.
• ^Ys∼Fˆn^{,soθˆ(Ys)willbebiasedrelativetoθˆ(Y)≡θˆ(Fˆ}n^{)byapproximatelythe}

b b

same amount:

bias_ˆn[θ^ˆ_n(Y^s)] = E_ˆn[θ^ˆ_n(Y^s)] − θ^ˆ_n(Y) ≈ EF[θ^ˆ_n(Y)] − θ^ˆ(FY) = biasF[θ^ˆ_n(Y)]

F ^b F ^b Y

B

B

b

b

cancel some of the bias:

B

b=1

b

b=1
^≈1Σθˆn^(Ys)−θˆn^(Y)

^{=⇒Subtractingoffθˆ}n^(Ys)−θˆn^(Y)=1ΣBθˆn^(Ys)−θˆn^(Y)fromθˆn^(Y)can

^θˆn,u^(Y)=θˆn^(Y)−.^θˆn^(Ys)−θˆn^(Y)Σ ^=2θˆn^(Y)−θˆn^(Ys)

# Nonparamametric bootstrap sampling

myboot.np <- function(B, y.obs, theta.f)

replicate(B,

y.b <- sample(y.obs,replace=TRUE) theta.f(y.b)

)

* = mean

 

# Sampling distribution of the mean of the bootstrap samples

2.0 theta.boot.dist <- replicate(S, y.obs <- mk.data(n)

mean(myboot.np(B, y.obs, theta.mle))

)

1.5mean(theta.boot.dist)-theta.true #Biasedbyabouttwiceasmuch

## [1] 0.1177986

# Debiased estimate and simulation

1.0 theta.bc <- function(theta.obs, theta.boot) theta.obs – (mean(theta.boot) – theta.obs)

0.5

theta.bc.dist <- replicate(S, y.obs <- mk.data(n)

theta.obs <- theta.mle(y.obs)

theta.boot <- myboot.np(B, y.obs, theta.mle)

theta.bc(theta.obs,theta.boot)

)

mean(theta.bc.dist)-theta.true #Muchlessbiased.

## [1] -0.001975362

MLE

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals Bootstrap for Regression

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals

Basic approaches

Refined approaches Demonstration

Bootstrap for Regression

Wakefield (2013, Sec. 2.7.1); Carpenter and Bithell (2000)

Given

Y^s, . . . , Y^s

B resamples of the dataset

1 B

b θ^ˆ(Y^s), b= 1..Bunivariate parameter estimates of interest

b θ^ˆ_q^sqth quantile of θ^ˆ(Y^s)s,

α = 1 − CL

^.θ − (θ ˆ ˆ^s

1−α/2

− θ^ˆ), θ^ˆ− (θ^ˆs

• θ^ˆ)^Σ

α/2

− − − Assumes that distribution of θ^ˆsθ^ˆis the same as the distribution of θ^ˆθ.

0.975 Intuition: 97.5% of the time, θ^ˆwill overestimate θ by less than θ^ˆs

− θ^ˆ,

0.975 so θ will be less than θ^ˆ− (θ^ˆs− θ^ˆ) only 2.5% of the time.

− Often has poor coverage.

Y ^s, . . . , Y ^s B resamples of the dataset

1 B

b θ^ˆ(Y^s), b= 1..Bunivariate parameter estimates of interest

b θ^ˆ_q^sqth quantile of θ^ˆ(Y^s)s,

α = 1 − CL

θ^ˆ(Y) − .θ^ˆ(Y^s) − θ^ˆ(Y)Σ ± z^s

b

B

b=1

b

 

^√v^arF(θ^ˆ),

b

where θ^ˆ(Y^s) = ^1ΣBθ^ˆ(Y^s) and

1−α/2

^

F θ

B

b=1

θ

b

− _B

b=1 ^θ

b )

. var (^ˆ) = ^1ΣB .^ˆ(Y^s) ^1ΣB ˆ(Y ^sΣ2

− Assumes sampling distribution of θ^ˆis symmetric and not too weird.

Intuition: Use bootstrap to debias and estimate the variance.

+ Has good coverage if it is.

^ + Can be useful if var(θ^ˆ) is hard to get analytically.

Y ^s, . . . , Y ^s B resamples of the dataset

1 B

b θ^ˆ(Y^s), b= 1..Bunivariate parameter estimates of interest

b θ^ˆ_q^sqth quantile of θ^ˆ(Y^s)s,

α = 1 − CL

+ Invariant to transformation.

^.θ^ˆs_α/2, θ^ˆsΣ

1−α/2 − Effectively assumes that there exists a monotone transformation g (θ) s.t.

g(θ^ˆ) ∼ N(g(θ), σ²)

g(θ^ˆs) ∼ N(g(θ^ˆ), σ²)

Intuition: If it exists we could get a 1 − α CI by taking

g⁻¹g(θ^ˆ) ± z^sσ, but g(θ^ˆ) + z^sσ ≡ g(θ^ˆs

), so we

1−α/2 _ˆs

1−α/2

1−α/2

can just use quantiles of θ directly.

⁾Performs poorly if θ^ˆdoesn’t behave that way (e.g., mean–variance relationship).

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals

Basic approaches

Refined approaches Demonstration

Bootstrap for Regression

Y ^s, . . . , Y ^s B resamples of the dataset

1 B

b θ^ˆ(Y^s), b= 1..Bunivariate parameter estimates of interest

b θ^ˆ_q^sqth quantile of θ^ˆ(Y^s)s,

α = 1 − CL

• A refinement of pivotal interval
• √ Adjusts for difference between var(θ^ˆs− θ^ˆ) and var(θ^ˆ− θ).

1. Estimate σˆ ≈ var_θ(θ^ˆ).
2. For each θ^ˆs, estimate σˆ^s≈

^.var _ˆs (θ^ˆ)

b

 

^{b b} θ_b⁾E.g., run bootstrap for every θ^ˆs.

⁾Very computationally intensive, unless a formula exists for σˆ.

1. Calculate t^s(θ^ˆs) = (θ^ˆs− θ^ˆ)/σˆ^sfor b= 1..B, t^sits qth quantile.

1. 1−α/2 α/2 The CI is then ^.θ^ˆ+ σˆt^s, θ^ˆ− σˆt^sΣ.

Y ^s, . . . , Y ^s B resamples of the dataset

1 B

b θ^ˆ(Y^s), b= 1..Bunivariate parameter estimates of interest

b θ^ˆ_q^sqth quantile of θ^ˆ(Y^s)s,

α = 1 − CL

• A refinement of percentile interval

+ Adjusts for skewness in θ^ˆand in how it changes with θ.

⁾ Loosely, assumes that a transformation g (θ) exists s.t.

g(θ^ˆ) ∼ N[g(θ) − b1 + ag(θ), 1 + ag(θ)²]

g(θ^ˆs) ∼ N[g(θ^ˆ) − b1 + ag(θ^ˆ), 1 + ag(θ^ˆ)²]

for a and b.

+ Works very well

− Can misbehave badly if α very small.

1. ^Then,qu

= Φ ,b−

s α/2

z 1+a(z^s

−b

−b)

,, q_l = Φ ,b−

_zs

1−α/2 1+a(z^s

−b

−b)

,, and the CI is

ˆ⁾More complicated afor parametric.

l

u^.θ^ˆ_q^s, θ^ˆ_q^sΣ.

α/2

1−α/2

Bootstrap

Bias Correction with Bootstrap Bootstrap Confidence Intervals

Basic approaches

Refined approaches Demonstration

Bootstrap for Regression