Posts Tagged ‘Statistics’

A Bayesian view of the Stein estimator

November 17, 2008

I’ve been attending Statistical Theory, a course in Part III Maths at Cambridge, taught by Richard Samworth. In the latest lecture, Richard showed the Stein estimator for the mean of multivariate Gaussian has lower expected loss (measured as the Euclidean distance between the true and estimated values of the mean) than the MLE for any value of \theta. From a frequentist perspective this appears completely unintuitive, but from a Bayesian perspective it appears much more reasonable.

Assume we observe vector X drawn from a multivariate normal of dimension p, with mean \theta and identity covariance matrix. The MLE of \theta is then just X, but the Stein estimator

\theta_s=\left( 1-\frac{p-2}{||X||^2} \right) X

The fact that this estimator performs better than the ML is termed shrinkage, because the estimator is shrunk towards 0.

How would a Bayesian approach this problem? First let’s put a Gaussian prior on \theta, so

\theta \sim N_p(\theta;0,\lambda^{-1}I)

where \lambda is a precision (inverse variance). In a fully Bayesian setting we would then put a Gamma prior on \lambda, but unfortunately we would then have to resort to sampling to infer the posterior over \theta. Assuming \lambda is known, then the posterior of \theta is

P(\theta|X,\lambda) \propto P(X|\theta) P(\theta|\lambda) = N_p(\theta;(1+\lambda)^{-1}X,(1+\lambda)^{-1})

Thus the expected value of \theta is

Now let’s find the MLE of \lambda. This is not ideal, but is tractable at least. To do this we’ll first integrate out \theta:

P(X|\lambda)=\int P(X|\theta) P(\theta|\lambda) d\theta

An unbiased estimate of \lambda_x is given by

Substituting for \lambda and rearranging gives
\lambda^{ML} = \left( \frac{|X|^2}{p-1}-1 \right)^{-1}

Substituting into the expression for E(\theta|X) above and rearranging gives
E(\theta|X) = \left( 1-\frac{p-1}{|X|^2} \right) X,
which is very close to the Stein estimate. I suspect that some choice of prior on \lambda would result in a MAP estimate which would give the p-2 term.

The conclusion is that an estimator which has unintuitively desirable properties in a frequentist framework, is intuitively a sensible estimator using under a Bayesian framework.