Introduction to Probability

A probability model specifies possible outcomes and assigns probabilities to events made from those outcomes. The sample space tells us what can happen; the probability law tells us how likely those possibilities are. Listing outcomes does not make them equally likely. For example, two fair dice have equally likely ordered pairs, while the sums formed from those pairs have different probabilities because some sums can be formed in more ways.

Introductory Terminology

• A random experiment is a procedure or observation whose outcome is uncertain in the model, with a specified set of possible outcomes. Rolling dice, flipping coins, or drawing cards are repeatable examples. A probability model can also describe a one-time uncertain event; literal unlimited repetition is not required to define its sample space or probabilities.
• The sample space (S) represents the complete set of possible outcomes of an experiment. It can be finite or infinite, and either discrete or continuous. For example, for a 6-sided die, the sample space is $S = \{1, 2, 3, 4, 5, 6\}$. For a coin flip, the sample space is $S = \{\text{heads}, \text{tails}\}$. If two coins are flipped simultaneously, the sample space expands to $S = \{(\text{H}, \text{H}), (\text{H}, \text{T}), (\text{T}, \text{H}), (\text{T}, \text{T})\}$.
• An event (E) is any collection of outcomes drawn from the sample space $S$, forming a subset of $S$. An event can consist of a single outcome or multiple outcomes. For example, rolling an even number on a die corresponds to the event $E = \{2, 4, 6\}$, getting tails when flipping a coin is $E = \{\text{tails}\}$, and drawing a heart from a standard deck of cards is $E = \{2\heartsuit, 3\heartsuit, \dots, A\heartsuit\}$.

Example: Dice Rolling

When you roll a 6-sided (fair) die, the sample space is:

$$S = \{1, 2, 3, 4, 5, 6\}$$

And the possible events include:

• Odd Numbers = $\{1, 3, 5\}$
• Even Numbers = $\{2, 4, 6\}$
• Rolling at Least a 5 = $\{5, 6\}$
• Prime Numbers = $\{2, 3, 5\}$
• Rolling a Number Less Than 4 = $\{1, 2, 3\}$

Each event is a subset of the sample space and represents a particular outcome or set of outcomes that we are interested in observing. The probability of an event is a measure of how likely that event is to occur, calculated, when all outcomes are equally likely, by dividing the number of favorable outcomes (the size of event set) by the total number of possible outcomes (the size of sample space).

Additional Examples

• In traffic observation, a count of accidents in a month can be modeled on $S=\{0,1,2,\ldots\}$. A finite upper limit should follow from an actual constraint or an explicitly truncated model. The largest historical count alone is not a physical limit on future counts.
• The lifetime of a light bulb can be modeled on the positive real line, $S=\{t:t>0\}$. Recording whole hours rounds a continuous lifetime into categories. Stopping observation at 10,000 hours instead creates possible right censoring: a bulb still working at the cutoff has a lifetime exceeding the cutoff, rather than a recorded failure at 10,000 hours. Recording precision and censoring are different features of the experiment.
• For an individual's birth month, the sample space is $S = \{\text{January}, \text{February}, \dots, \text{December}\}$.
• For radioactive decay, suppose we record which one-second interval contains the first observed decay. Outcomes can be encoded as D, ND, NND, and so on, where N means no decay yet and D marks the first decay. This discrete recording model does not claim that physical decay occurs only at integer seconds. An unbounded sequence of N symbols can be included as a zero-probability outcome under a model in which eventual decay occurs almost surely.
• If we record only the sum of two dice, the sample space is $S = \{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12\}$, but these sums are not equally likely (see the example below).
• In the seating at a square table scenario, if two people sit at two different corners of the four corners, the sample space for their seating positions consists of the $4\cdot 3=12$ ordered pairs of distinct positions.
• When rolling two dice, the sample space of ordered pairs (first die, second die) can be laid out in a table:

For instance, (1,1) means the first die shows 1 and the second die shows 1, and (1,2) means the first die shows 1 and the second die shows 2. Each ordered pair represents one of the 36 equally likely outcomes.

Probability

A probability measure assigns an event $A$ a number between zero and one. Probability zero and one mean almost impossible and almost sure under the model; they need not mean logical impossibility or certainty. In a continuous uniform model on $[0,1]$, any specified point has probability zero even though one point is observed. In a finite sample space with all elementary outcomes equally likely, probabilities can be calculated by counting:

$$ P(A) = \frac{\text{Number of favorable outcomes}}{\text{Total number of possible outcomes}} $$

The sums over individual outcomes in the following formulas apply to discrete, countable sample spaces. Continuous models assign probability by integrating a density over a region, when a density exists. The event-level axioms apply to both types of model. In more advanced probability, events are restricted to measurable subsets so that their probabilities are well defined.

• Probability of an Outcome (P(s)): The probability of a particular outcome s is a measure of the likelihood that this outcome will occur when the experiment is performed. It is a value between 0 (impossible event) and 1 (certain event), satisfying:
• $0 \leq P(s) \leq 1$ for each outcome $s$.

• The sum of the probabilities of all outcomes equals 1, or $\sum_{s \in S} P(s) = 1$, where $S$ is the sample space.

• Probability of an Event (P(E)): The probability of an event E is determined by adding up the probabilities of the outcomes that compose E. Mathematically, this is represented by the formula:

$$ P(E) = \sum_{s \in E} P(s) $$

• A random variable X is a function that assigns a numerical value to each outcome in a sample space. For instance, X could represent the total number of points scored in a basketball game.
• The expected value of a random variable X gives a measure of the central tendency of the distribution of the random variable. It is computed as the weighted average of all possible values that the random variable can take on, with the weights being the probabilities of the values. The formula for the expected value is:

$$ E(X) = \sum_{s \in S} P(s) \cdot X(s) $$

Example: Full Calculation for Two Six-Sided Dice

Consider the random variable X which represents the sum of the numbers that come up when two fair six-sided dice are rolled. To calculate the expected value E(X), we must first enumerate the sample space and the associated probabilities.

The sample space S for the sum of two dice is:

$$S = \{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12\}$$

Which is better represented using a table:

The probability of each sum, given that there are 36 possible combinations when rolling two dice, is:

$$P(2) = 1/36, P(3) = 2/36, P(4) = 3/36, ..., P(7) = 6/36, ..., P(12) = 1/36$$

Now, we calculate the expected value E(X) by multiplying each sum by its probability and adding all the products together:

$$E(X) = (1/36)*2 + (2/36)*3 + (3/36)*4 + ... + (6/36)*7 + ... + (1/36)*12$$

When you perform this calculation, you'll find that E(X) equals 7, which is the average sum of rolling two six-sided dice.

Example: Cards

A standard deck of playing cards contains 52 cards, divided evenly into four suits: hearts, diamonds, clubs, and spades. Each suit has 13 cards consisting of numbers 2 through 10, a Jack, Queen, King, and Ace. If we draw a single card at random from a well-shuffled deck, we can calculate the probability of various events.

Probability of Drawing a Heart

The probability of drawing a card from any particular suit, like hearts, is given by the ratio of the number of cards in that suit to the total number of cards in the deck:

$$ P(\text{Heart}) = \frac{\text{Number of hearts}}{\text{Total number of cards}} = \frac{13}{52} = \frac{1}{4} $$

This is because there are 13 hearts and 52 cards in total, so the probability is $\frac{13}{52}$, which simplifies to $\frac{1}{4}$ or 25%.

Probability of Drawing a Face Card

Face cards are the Jack, Queen, and King of each suit. There are 3 face cards per suit, and 4 suits, so there are 12 face cards in total.

$$ P(\text{Face Card}) = \frac{\text{Number of face cards}}{\text{Total number of cards}} = \frac{12}{52} = \frac{3}{13} $$

Probability of Drawing an Ace

Since there are four Aces in a deck, one for each suit, the probability of drawing an Ace is:

$$ P(\text{Ace}) = \frac{\text{Number of Aces}}{\text{Total number of cards}} = \frac{4}{52} = \frac{1}{13} $$

Probability of Drawing a Card Lower than 5

For this event, “lower than 5” refers to the printed number cards 2, 3, and 4, excluding Aces and face cards. Four suits give 12 such cards. A different game might count an Ace as one, in which case there would be 16 cards and probability $4/13$. The event definition therefore needs to be specified separately from the numerical encoding used for the expected-value example below.

$$ P(\text{Card} < 5) = \frac{\text{Number of cards lower than 5}}{\text{Total number of cards}} = \frac{12}{52} = \frac{3}{13} $$

Expected Value of Card Numbers

If we assign numerical values to each card (Jack = 11, Queen = 12, King = 13, Ace = 1), we can calculate the expected value for the number on a card drawn at random from the deck. We'll denote this random variable as $X$.

$$ E(X) = \sum_{i=1}^{13} P(i) \cdot i $$

Where $P(i)$ is the probability of drawing a card with number $i$, which is $\frac{1}{13}$ for each card number since there is an equal number of each card number in the deck. Hence

$$ E(X) = \frac{1}{13}(1 + 2 + \cdots + 13) = \frac{1}{13}\cdot\frac{13\cdot 14}{2} = 7. $$

These probabilities and expected values provide the fundamental basis for a wide range of card games and can be used to calculate more complex odds based on game rules and strategies.

Example: Distinguishable Balls in Cells

Consider an experiment where r = 3 distinguishable balls, labeled a, b, and c, are to be placed in n = 3 cells. We'll enumerate the elementary events and then calculate the probabilities of several interesting events related to this setup.

Elementary events are unique arrangements of the balls in the cells. Since the balls are distinguishable, each different arrangement counts as a separate event.

We can define several events based on these arrangements:

• The event A, "At least two balls are in one cell," includes any arrangement where a single cell contains two or three balls.
• For event B, "The first cell is not empty," this event includes any arrangement where the first cell contains at least one ball.
• The event C, "Both A and B occur," refers to the intersection of events A and B, meaning that both the first cell is not empty and at least one cell contains two or more balls.
• In the event D, "Either A or B occurs," this event represents the union of events A and B, meaning that either the first cell is not empty, or at least two balls are in one cell, or both.

Probability Calculations

With 27 possible outcomes, each equally likely, the probabilities of the events are:

Equal likelihood here comes from independently assigning each labeled ball to one of three cells with probability $1/3$. There are $3^3=27$ assignments. The six assignments with one ball per cell are the $3!$ permutations of the three labels, so $A$ contains $27-6=21$ assignments. Cell one is empty in $2^3=8$ assignments, leaving $27-8=19$ for $B$. Every one-per-cell assignment has cell one occupied, so $A\cap B$ contains $19-6=13$ assignments. If $A$ fails, every cell is occupied and $B$ holds; hence $A\cup B$ is the whole sample space.

$$ P(A) = \frac{\text{Number of favorable outcomes for A}}{\text{Total outcomes}} = \frac{21}{27} $$

$$ P(B) = \frac{\text{Number of favorable outcomes for B}}{\text{Total outcomes}} = \frac{19}{27} $$

$$ P(C) = \frac{\text{Number of favorable outcomes for C}}{\text{Total outcomes}} = \frac{13}{27} $$

$$ P(D) = \frac{\text{Number of favorable outcomes for D}}{\text{Total outcomes}} = \frac{27}{27} = 1 $$

These values are consistent with the addition rule: $P(D) = P(A) + P(B) - P(C) = \frac{21 + 19 - 13}{27} = 1$.

These probabilities can be simplified:

$$ P(A) = \frac{7}{9}, \quad P(B) = \frac{19}{27}, \quad P(C) = \frac{13}{27}, \quad P(D) = 1 $$

Example: Indistinguishable Balls

In this scenario, we consider the placement of r = 3 indistinguishable balls into n = 3 cells. Since the balls are indistinguishable, the order in which they are placed does not matter, which reduces the number of elementary events compared to the case with distinguishable balls.

The elementary events are represented by the arrangements of the balls across the cells as follows:

Events of Interest:

• The event A, "At least two balls are in one cell," includes all arrangements where any cell contains two or more balls. This covers situations where a single cell has either two or three balls.
• For event B, "The first cell is not empty," the event consists of all arrangements where the first cell contains at least one ball. This means the first cell cannot be empty in any valid outcome.

Probability Calculations

If the 10 unique outcomes are treated as equally likely (the Bose-Einstein counting model), the probabilities are:

$$ P(A) = \frac{\text{Number of favorable outcomes for A}}{\text{Total unique outcomes}} = \frac{9}{10} $$

$$ P(B) = \frac{\text{Number of favorable outcomes for B}}{\text{Total unique outcomes}} = \frac{6}{10} = \frac{3}{5} $$

Note that this equal-likelihood assumption is a modeling choice. If the balls are physically placed independently into cells (the Maxwell-Boltzmann model), the 10 arrangements above are not equally likely: "ooo in cell 1" has probability $\frac{1}{27}$, "oo in one cell and o in another" has probability $\frac{3}{27}$, and "one ball in each cell" has probability $\frac{6}{27}$. Under that model $P(A)=\frac{21}{27}=\frac{7}{9}$ and $P(B)=\frac{19}{27}$, the same values as in the distinguishable-ball example.