Some Properties of Estimators

[Read HC 6.1.]
Let the random variable $X$ have a pdf (or probability function) that is of known functional form, but in which the pdf depends on an unknown parameter $\theta$ (which may be a vector) that may take any value in a set $\Theta$ (the parameter space). We can write the pdf as $f(x; \theta), \theta \in \Theta$. To each value $\theta \in \Theta$ there corresponds one member of the family. If the experimenter needs to select precisely one member of the family as being the pdf of the random variable, he needs a point estimate of $\theta$, and this is the subject of sections 2.1 to 2.4 of this chapter.

Of course, we estimate $\theta$ by some (appropriate) function of the observations $X_1, \ldots, X_n$ and such a function is called a statistic or an estimator. A particular value of an estimator, say t( $x_1, \ldots, x_n$), is called an estimate. We will be considering various qualities that a ``good'' estimator should possess, but firstly, it should be noted that, by virtue of it being a function of the sample values, an estimator is itself a random variable. So its behaviour for different random samples will be described by a probability distribution.

It seems reasonable to require that the distribution of the estimator be somehow centred with respect to $\theta$. If it is not, the estimator will tend either to under-estimate or over-estimate $\theta$. A further property that a good estimator should posssess is precision, that is, the dispersion of the distribution should be small. These two properties need to be considered together. It is not very helpful to have an estimator with small variance if it is ``centred'' far from $\theta$. The difference between an estimator $T=t(X_1, \ldots, X_n$) and $\theta$ is referred to as an error, and the ``mean squared error'' defined below is a commonly used measure of performance of an estimator.