Bayesian and frequentist statistics are two major approaches to statistical inference. Both use sample data to learn about an underlying population or data-generating process, but they differ in how they interpret probability, represent uncertainty, and draw conclusions about unknown parameters.
A useful distinction is:
| AspectFrequentistBayesian | ||
| Unknown parameter | Fixed but unknown | Represented by a probability distribution |
| Probability | Long-run behavior of repeated experiments | Quantifies uncertainty given a model and available information |
| Prior information | Not represented through a prior distribution | Encoded through a prior distribution |
| Main inferential output | Point estimates, confidence intervals, hypothesis tests | Posterior distributions, credible intervals, posterior probabilities |
| Data | Random before observation | Observed data are treated as fixed once collected |
| Parameter uncertainty | Described through sampling procedures | Described directly by the posterior distribution |
Neither framework is universally better. The appropriate choice depends on the problem, the model, the available information, and the type of conclusion we want to make.
A frequentist does not normally assign a probability distribution to a fixed parameter itself. Instead, probability statements describe the behavior of data, estimators, and procedures under repeated sampling.
In the frequentist interpretation, probability is connected to the long-run frequency of events under repeated trials. For example, if a fair coin is tossed many times, the proportion of heads is expected to approach $0.5$.
An estimator such as the sample mean $\bar{x}$ or sample proportion $\hat p$ varies from sample to sample. Its sampling distribution describes this variation and forms the basis of standard errors, confidence intervals, and hypothesis tests.
In hypothesis testing, a test statistic measures how far the observed data depart from what would typically be expected under the null hypothesis.
The p-value is the probability, assuming the null hypothesis and the statistical model are correct, of obtaining a test statistic at least as extreme as the one observed.
It is important that:
$$ \text{p-value} \neq P(H_0 \mid \text{data}) $$
A p-value does not give the probability that the null hypothesis is true.
These limitations are properties of particular methods and interpretations, not evidence that frequentist statistics is inherently unreliable.
Suppose the attribute of interest is represented by X, while O represents its absence. For illustration, imagine the following population:
Population:
O O X O O O X X O X
A random sample of four items gives:
Sample:
X O O X
The observed sample proportion is:
$$ \hat p = \frac{2}{4} = 0.50 $$
A frequentist can use $\hat p$ as a point estimate of the unknown population proportion $p$.
There is an important modeling detail here: if the entire 10-item population shown above were actually known, there would be no need to estimate its proportion—we could calculate it exactly. The example should therefore be understood as an illustration of sampling, with the full population shown only for intuition.
For the standard calculations below, we use the usual Bernoulli/binomial approximation. If the target really were a small finite population sampled without replacement, a finite-population or hypergeometric model would be more appropriate.
| StepEquationPlugging the numbers | ||
| Point estimate | $\hat p=x/n$ | $\hat p=2/4=0.50$ |
| Standard error | $SE(\hat p)=\sqrt{\hat p(1-\hat p)/n}$ | $\sqrt{0.5(1-0.5)/4}=0.25$ |
| 95% Wald confidence interval | $\hat p\pm z_{0.975}SE(\hat p)$, where $z_{0.975}=1.96$ | $0.50\pm1.96(0.25)\approx[0.01,0.99]$ |
| Null-hypothesis test $H_0\:p=p_0$ | $z=(\hat p-p_0)/\sqrt{p_0(1-p_0)/n}$ | For $p_0=0.5$, $z=0$ and the two-sided p-value is $1$ |
The Wald interval is shown because it follows directly from the familiar standard-error formula, but with only four observations it is not a reliable confidence interval. Methods such as the Wilson or exact binomial interval are preferable for such a small sample.
This illustrates an important frequentist idea: the estimate is based entirely on the observed sample, while its uncertainty is evaluated through the sampling behavior of the estimator.
Bayesian statistics represents uncertainty about unknown parameters using probability distributions.
Saying that a Bayesian parameter is "random" does not necessarily mean that the underlying physical quantity is changing randomly. Rather, a probability distribution is used to represent our uncertainty about its unknown value.
Bayes' theorem is the foundation of Bayesian inference:
$$ P(\theta \mid D) = \frac{P(D\mid\theta)P(\theta)} {P(D)} $$
where:
Because $P(D)$ does not depend on $\theta$, the relationship is often written as:
$$ \text{Posterior} \propto \text{Likelihood} \times \text{Prior} $$
The denominator,
$$ P(D) = \int P(D\mid\theta)P(\theta)\,d\theta, $$
normalizes the posterior so that it forms a valid probability distribution.
Bayesian probability can therefore be used to quantify uncertainty about events, hypotheses, predictions, and unknown parameters given a specified model and available information.
A prior distribution can represent previous studies, domain knowledge, physical constraints, or weak background information.
For example, when modeling snake lifespans, biological knowledge tells us that values near 10 or 20 years may be plausible for some species, while a lifespan of 1000 years is not. A prior can encode this information without claiming that we know the exact lifespan in advance.
Priors can vary in strength:
After observing data, Bayes' theorem combines the prior with the likelihood to produce the posterior distribution.
As the amount of informative data increases, the likelihood often has more influence on the posterior and the effect of a reasonable prior becomes smaller.
$$ P(\theta>0\mid D) $$
or
$$ P(a<\theta