Last modified: October 25, 2026

This article is written in: 🇺🇸

Point Estimation

A parameter is an unknown feature of a population or statistical model, such as a mean $\mu$, variance $\sigma^2$, probability $p$, or regression coefficient $\beta$. A point estimator is a rule or statistic computed from sample data and used to estimate that parameter.

Before the data are observed, an estimator such as $\hat\theta$ is a random variable because its value varies from sample to sample. After the data are observed, the resulting numerical value is called an estimate.

Estimator vs Estimate

Suppose $X_1,\ldots,X_n$ are sampled from a population with mean $\mu$. The sample mean

$$\bar X=\frac{1}{n}\sum_{i=1}^n X_i$$

is an estimator of $\mu$. If the observed sample gives $\bar x=12.4$, then $12.4$ is the corresponding estimate.

This distinction matters because properties such as bias, variance, and consistency describe the estimator and its sampling behavior, not a single realized estimate.

Bias

The bias of an estimator $\hat\theta$ is

$$\mathrm{Bias}(\hat\theta)=E[\hat\theta]-\theta.$$

An estimator is unbiased if

$$E[\hat\theta]=\theta.$$

In other words, across repeated samples generated under the model, its average value equals the parameter being estimated. This is a statement about the rule, not a guarantee that the estimate in the current sample is close to the target. A procedure that overestimates and underestimates by large amounts can still be unbiased if those errors average to zero.

Unbiasedness is useful, but it is not the only desirable property. An unbiased estimator with high variability may be less useful than a slightly biased estimator with substantially lower variance.

Variance and Standard Error

The sampling variance of an estimator is

$$\mathrm{Var}(\hat\theta),$$

which describes how much the estimator varies across repeated samples.

Its standard error is the standard deviation of its sampling distribution:

$$\mathrm{SE}(\hat\theta)=\sqrt{\mathrm{Var}(\hat\theta)}.$$

In practice, the true standard error often depends on unknown population parameters. We therefore estimate it from the observed data, producing an estimated standard error.

Two standard cases are

$$\mathrm{SE}(\bar X)=\frac{\sigma}{\sqrt n}, \qquad \mathrm{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}},$$

with $\sigma$ and $p$ replaced by sample estimates to get the estimated standard error.

A smaller standard error generally indicates a more precise estimator.

Mean-Squared Error

Bias and variance can be considered together through the mean-squared error:

$$\mathrm{MSE}(\hat\theta) = E[(\hat\theta-\theta)^2] = \mathrm{Var}(\hat\theta) + \mathrm{Bias}(\hat\theta)^2.$$

This decomposition makes the bias-variance trade-off explicit. An estimator can have some bias yet still achieve a smaller MSE if the reduction in variance is large enough.

To obtain the decomposition, write $\hat\theta-\theta=(\hat\theta-E[\hat\theta])+(E[\hat\theta]-\theta)$ and expand the square. The cross term has expectation zero because $E[\hat\theta-E[\hat\theta]]=0$. The remaining terms are the variance and squared bias. This derivation also shows why both terms must refer to the same target and sampling model.

Consistency

An estimator is consistent if it converges to the true parameter as the sample size grows. More formally, $\hat\theta_n$ is consistent for $\theta$ if, for every $\varepsilon>0$,

$$P(|\hat\theta_n-\theta|>\varepsilon)\to 0 \qquad \text{as } n\to\infty.$$

Informally, as more data become available, the estimator becomes increasingly concentrated near the true parameter.

Consistency is an asymptotic property. A consistent estimator can still perform poorly in small samples, so finite-sample properties such as bias and variance remain important.

One sufficient route to consistency is a mean-squared error tending to zero: Markov's inequality gives $P(|\hat\theta_n-\theta|>\varepsilon)\leq\mathrm{MSE}(\hat\theta_n)/\varepsilon^2$. Thus bias tending to zero together with variance tending to zero is sufficient. Unbiasedness by itself is not sufficient. For example, estimating $\mu$ by the first observation $X_1$ is unbiased for every sample size, but it ignores the additional observations and does not become concentrated as $n$ grows.

Method of Moments

The method of moments estimates parameters by equating sample moments with their corresponding theoretical population moments and solving for the unknown parameters.

For example, if a model satisfies

$$E[X]=g(\theta),$$

we can replace the population mean with the sample mean and solve

$$\bar X=g(\hat\theta).$$

If the model contains several unknown parameters, additional moments such as $E[X^2]$, $E[X^3]$, and their sample counterparts can be used to obtain enough equations to solve for them.

The method of moments is often simple and intuitive, although it does not necessarily produce the most efficient estimator.

Maximum Likelihood

Suppose the observed data are $x_1,\ldots,x_n$ and the statistical model has parameter $\theta$. Once the data have been observed, their joint probability mass function or probability density can be viewed as a function of $\theta$. This function is called the likelihood:

$$L(\theta;x_1,\ldots,x_n).$$

A maximum-likelihood estimator (MLE) chooses a parameter value that maximizes the joint mass or density evaluated at the observed data. For continuous data, a density value is not the probability of observing that exact sample, which is typically zero. It compares how the model distributes probability around the observations. The likelihood varies the parameter while holding the observations fixed:

$$\hat\theta_{\mathrm{MLE}} = \arg\max_\theta L(\theta;x_1,\ldots,x_n).$$

In practice, it is often easier to maximize the log-likelihood

$$\ell(\theta)=\log L(\theta),$$

because logarithms turn products into sums. Since the logarithm is strictly increasing, maximizing the log-likelihood gives the same parameter value as maximizing the likelihood.

Bernoulli Example

Suppose $X_1,\ldots,X_n$ are independent Bernoulli observations with success probability $p$:

$$X_i\sim\mathrm{Bernoulli}(p).$$

The likelihood is

$$L(p) = p^{\sum_i x_i} (1-p)^{n-\sum_i x_i}.$$

Let

$$S=\sum_{i=1}^n x_i$$

be the number of observed successes. Then the log-likelihood is

$$\ell(p) = S\log p + (n-S)\log(1-p).$$

For $0<S<n$, differentiating gives $\ell'(p)=S/p-(n-S)/(1-p)$. Equating this to zero yields $S(1-p)=(n-S)p$, hence $p=S/n$. The second derivative is negative, so this stationary point is a maximum. If $S=0$ or $S=n$, there is no interior stationary point; the maximum is at the boundary $p=0$ or $p=1$, respectively.

Maximizing this expression gives

$$\hat p_{\mathrm{MLE}} = \frac{S}{n} = \bar X.$$

Thus, for Bernoulli data, the maximum-likelihood estimate of $p$ is simply the observed proportion of successes.

Normal Example and the Variance Divisor

Suppose $X_1,\ldots,X_n$ are independent $N(\mu,\sigma^2)$ observations. Maximizing the log-likelihood

$$\ell(\mu,\sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i-\mu)^2$$

gives

$$\hat\mu_{\mathrm{MLE}}=\bar X, \qquad \hat\sigma^2_{\mathrm{MLE}}=\frac{1}{n}\sum_{i=1}^n (X_i-\bar X)^2.$$

The MLE of $\sigma^2$ divides by $n$ and is biased:

$$E[\hat\sigma^2_{\mathrm{MLE}}]=\frac{n-1}{n}\sigma^2.$$

The usual sample variance divides by $n-1$,

$$S^2=\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)^2,$$

and satisfies $E[S^2]=\sigma^2$. The $n-1$ reflects that $\bar X$ was estimated from the same data, which uses up one degree of freedom. For normal data the biased MLE actually has the smaller MSE of the two, which illustrates that unbiasedness is not the only goal.

Exponential Example: Estimators Can Be Biased

If $X_i\sim\mathrm{Exponential}(\lambda)$ with rate $\lambda$, then $E[X]=1/\lambda$. The method of moments sets $\bar X=1/\hat\lambda$, so

$$\hat\lambda_{\mathrm{MoM}}=\frac{1}{\bar X}.$$

The log-likelihood is $\ell(\lambda)=n\log\lambda-\lambda\sum_i x_i$, whose maximizer is the same value, $\hat\lambda_{\mathrm{MLE}}=1/\bar X$. For $n>1$, although $\bar X$ is unbiased for $1/\lambda$, its reciprocal is not unbiased for $\lambda$. A nonlinear transformation does not generally commute with expectation. Here the sum has a gamma distribution, whose inverse moment gives:

$$E\left[\frac{1}{\bar X}\right]=\frac{n}{n-1}\lambda.$$

The bias, $\lambda/(n-1)$, shrinks as $n$ grows, so $1/\bar X$ is biased but consistent. The corrected estimator $\frac{n-1}{n\bar X}$ is unbiased.

Both inverse-moment statements require $n>1$. At $n=1$, $E[1/X_1]$ diverges, so a finite bias correction cannot be justified by the displayed expectation. For consistency, the law of large numbers gives $\bar X\to1/\lambda$ in probability, and continuity of the reciprocal at the positive limit gives $1/\bar X\to\lambda$. This argument establishes consistency without claiming finite-sample unbiasedness.

Estimation Is Not Yet Inference

A point estimator provides a single value for estimating an unknown parameter. By itself, however, it does not tell us how uncertain that estimate is.

For example, two samples may produce the same point estimate but very different levels of precision if their sample sizes differ substantially.

To quantify uncertainty, we need additional information about the estimator's sampling distribution. Depending on the problem, this may come from an exact distribution, an asymptotic approximation, or a resampling method such as the bootstrap.

These ideas lead naturally to confidence intervals and hypothesis tests.

An uncertainty statement should identify the parameter being estimated, the sampling or probability model, and which sources of error the calculation includes.