Wednesday, October 7, 2009

Unbiased but Probably an Underestimate

Over at mind hacks, Vaughan asks, why it always seems worse than you think? It turns out that its proper Bayesian statistical reasoning. The frequentist constructs unbiased estimates in the limit of large data. For finite data sets these estimates might be unbiased in some sense, but that doesn't mean that the true value is going to be just as likely to be greater than the estimate or less than the estimate. The median estimate has this property, by definition, but in most cases of interest the median is significantly higher than the mean or the maximum a posterori estimate. This seems to be generically the case when estimating quantities which are small. For example, consider two cases where we are estimating a small quantity. In one case, I sample from a Poisson distribution with mean obtained from a gamma distribution. In the other case, I assume a beta prior (uniform prior in particular) for the probability p of binomially distributed random variable. Because these are conjugate priors the posterior distributions are also gamma and beta respectively and we can compare the probability that the true value of the associated means conditioned on observing K examples (normalized by N samples in the Binomial case) is greater than the maximum a posteriori or mean estimates.

Clearly both MAP and MEAN estimates of rare events are likely to be underestimates given finite data. The same is true for ML estimates. In the binomial case a uniform prior was used. In the poisson case the results are independent of the prior... unless i made a silly error putting this together in 15 minutes :) For the binomial case, a Jeffrey's prior also seems to have this property despite heavily favoring small probabilities...