Posts Tagged ‘Machine learning’

A Bayesian view of the Stein estimator

November 17, 2008

I’ve been attending Statistical Theory, a course in Part III Maths at Cambridge, taught by Richard Samworth. In the latest lecture, Richard showed the Stein estimator for the mean of multivariate Gaussian has lower expected loss (measured as the Euclidean distance between the true and estimated values of the mean) than the MLE for any value of $\theta$. From a frequentist perspective this appears completely unintuitive, but from a Bayesian perspective it appears much more reasonable.

Assume we observe vector $X$ drawn from a multivariate normal of dimension p, with mean $\theta$ and identity covariance matrix. The MLE of $\theta$ is then just $X$, but the Stein estimator

$\theta_s=\left( 1-\frac{p-2}{||X||^2} \right) X$

The fact that this estimator performs better than the ML is termed shrinkage, because the estimator is shrunk towards 0.

How would a Bayesian approach this problem? First let’s put a Gaussian prior on $\theta$, so

$\theta \sim N_p(\theta;0,\lambda^{-1}I)$

where $\lambda$ is a precision (inverse variance). In a fully Bayesian setting we would then put a Gamma prior on $\lambda$, but unfortunately we would then have to resort to sampling to infer the posterior over $\theta$. Assuming $\lambda$ is known, then the posterior of $\theta$ is

$P(\theta|X,\lambda) \propto P(X|\theta) P(\theta|\lambda) = N_p(\theta;(1+\lambda)^{-1}X,(1+\lambda)^{-1})$

Thus the expected value of $\theta$ is
$E(\theta|X)=(1+\lambda)^{-1}X$

Now let’s find the MLE of $\lambda$. This is not ideal, but is tractable at least. To do this we’ll first integrate out $\theta$:

$P(X|\lambda)=\int P(X|\theta) P(\theta|\lambda) d\theta$
$=N_p(X;0,\lambda_x^{-1}I)$
where
$\lambda_x=\frac{\lambda}{1+\lambda}$.

An unbiased estimate of $\lambda_x$ is given by
$\lambda_x^{ML}=\frac{|X|^2}{p-1}$.

Substituting for $\lambda$ and rearranging gives
$\lambda^{ML} = \left( \frac{|X|^2}{p-1}-1 \right)^{-1}$

Substituting into the expression for $E(\theta|X)$ above and rearranging gives
$E(\theta|X) = \left( 1-\frac{p-1}{|X|^2} \right) X$,
which is very close to the Stein estimate. I suspect that some choice of prior on $\lambda$ would result in a MAP estimate which would give the $p-2$ term.

The conclusion is that an estimator which has unintuitively desirable properties in a frequentist framework, is intuitively a sensible estimator using under a Bayesian framework.

To sample or not to sample? Missing values in Gibbs sampling

November 12, 2008

I ran into an interesting debate last week. Consider the factor analysis model:

$y=Gx+\epsilon$

where y is an observed vector, G is the mixing matrix, x is a vector of latent variables and the last term is isotropic Gaussian noise. Assume we observe a sequence of n y’s drawn iid from the model. Then writing $Y=[y_1 \dots y_n], X=[x_1 \dots x_n]$ and $E=[\epsilon_1 \dots \epsilon_n]$ we have

$Y=GX+E$

A very naive way to infer the posterior $P(G,X|Y)$ would be to perform Gibbs sampling: successively sample $P(G|X,Y)$, then $P(X|G,Y)$. Assume conjugate Gaussian priors on G and X, this becomes particularly easy. Note that the likelihood function, assuming Gaussian noise with precision $\lambda_e$, is

$\left(\frac{\lambda_e}{2\pi}\right)^\frac{ND}{2}\exp{\left\{-\frac{\lambda_e}{2}[-tr((Y-GX)^T(Y-GX))]\right\}}$

Now assume however that some of the elements of Y are missing, at random. How should we cope with this? Two, superficially quite different approaches are possible.

In the first, the simplest way, would be to consider the unobserved elements of Y as latent variables. Then in our Gibbs sampling scheme, have initialized G and X, we simply sample $P(Y^u|G,X)$, i.e. set the unobserved elements of Y to the corresponding elements of $GX+E$ (sampling E as noise). Then our sampling steps for G and X are the same as before. This does not change the model structure in any way, and is a completely valid Gibbs sampling scheme.

The second approach is to exclude terms involving the missing values from the likelihood function. We can achieve this algebraically by element-wise multiplication of the (Y-GX) term by a binary matrix H. Element (i,j) of H is equal to one iff element (i,j) of Y was observed. Now the likelihood function becomes:

$\left(\frac{\lambda_e}{2\pi}\right)^\frac{N_o}{2}\exp{\left\{-\frac{\lambda_e}{2}[-tr(((Y-GX)\circ H)^T((Y-GX)\circ H)]\right\}}$

where $N_o$ is the number of observed elements of Y. This approach makes sampling G and X a little more tricky because H affects the covariance structure of the conditional distributions. To deal with the algebra (specifically to get rid of the troublesome Hadamard product) I use some ideas from Tom Minka’s matrix algebra notes.

Having had two pretty smart people arguing the case for both sides, we decided to give this some more thought. Both schemes are valid Gibbs samplers for the same model. The first is considerably easier to implement, but intuitively should not perform as well. Once we have a sample for the unobserved elements of Y, when we sample G and X the algorithm has no knowledge about which elements of Y are observed, so we must be introducing more noise into the algorithm than we would leaving these terms out.

Jurgen made the very good point that leaving the terms out of the likelihood function is equivalent to integrating out the unobserved variables. This leads to the intuition that the question of which approach to take is actually equivalent to the much broader question of how much to collapse a Gibbs sampler. This question has garnered a great deal of interest of late, and seems to be very much model dependent. If you have two highly correlated variables (such as G and X above) then integrating out one would seem very beneficial, since exploring the joint parameter space will be very slow otherwise, whereas Jurgen and Finale have found cases (specifically the linear Gaussian IBP model) where the uncollapsed Gibbs sampler is much faster.

I’ll report back again when I have some hard numbers to support these ideas!