Last modified: December 23, 2023

This article is written in: 🇺🇸

Covariance is a fundamental statistical measure that quantifies the degree to which two random variables change together. It indicates the direction of the linear relationship between variables:

• A positive covariance implies that as one variable increases, the other tends to increase as well.
• A negative covariance suggests that as one variable increases, the other tends to decrease.
• A zero covariance indicates no linear relationship between the variables.

Definition

The covariance between two random variables $X$ and $Y$ is defined as the expected value (mean) of the product of their deviations from their respective means:

$$\text{Cov}(X,Y)=E\left[\right(X-{\mu }_X\left)\right(Y-{\mu }_Y\left)\right]$$

Where:

• $\text{Cov}(X,Y)$ is the covariance between $X$ and $Y$.
• $E$ denotes the expected value operator.
• ${\mu }_X=E\left[X\right]$ is the mean of $X$.
• ${\mu }_Y=E\left[Y\right]$ is the mean of $Y$.

Alternative Expression

By expanding the definition and applying the linearity properties of expectation, covariance can also be expressed as:

$$\text{Cov}(X,Y)=E\left[XY\right]-E\left[X\right]E\left[Y\right]$$

Derivation:

1. Start with the definition:

$$\text{Cov}(X,Y)=E\left[\right(X-{\mu }_X\left)\right(Y-{\mu }_Y\left)\right]$$

1. Expand the product inside the expectation:

$$\text{Cov}(X,Y)=E[XY-X{\mu }_Y-{\mu }_{X}Y+{\mu }_X{\mu }_Y]$$

1. Use the linearity of expectation:

$$\text{Cov}(X,Y)=E\left[XY\right]-{\mu }_{Y}E\left[X\right]-{\mu }_{X}E\left[Y\right]+{\mu }_X{\mu }_Y$$

1. Recognize that ${\mu }_X=E\left[X\right]$ and ${\mu }_Y=E\left[Y\right]$:

$$\text{Cov}(X,Y)=E\left[XY\right]-{\mu }_Y{\mu }_X-{\mu }_X{\mu }_Y+{\mu }_X{\mu }_Y=E\left[XY\right]-{\mu }_X{\mu }_Y$$

Thus, we arrive at:

$$\text{Cov}(X,Y)=E\left[XY\right]-E\left[X\right]E\left[Y\right]$$

Interpretation

• Positive Covariance ($\text{Cov}(X,Y)>0$): Indicates that $X$ and $Y$ tend to increase or decrease together.
• Negative Covariance ($\text{Cov}(X,Y)<0$): Indicates that when $X$ increases, $Y$ tends to decrease, and vice versa.
• Zero Covariance ($\text{Cov}(X,Y)=0$): Suggests no linear relationship between $X$ and $Y$.

Important Note:

• If $X$ and $Y$ are independent, then $\text{Cov}(X,Y)=0$.
• However, a covariance of zero does not necessarily imply independence. Variables can be uncorrelated (zero covariance) but still dependent in a non-linear way.

Properties of Covariance

I. Symmetry:

$$\text{Cov}(X,Y)=\text{Cov}(Y,X)$$

II. Linearity in Each Argument:

For constants $a$ and $b$, and random variables $X$, $Y$, and $Z$:

$$\text{Cov}(aX+bY,Z)=a\text{Cov}(X,Z)+b\text{Cov}(Y,Z)$$

III. Covariance with Itself (Variance Relation):

The covariance of a variable with itself is the variance of that variable:

$$\text{Cov}(X,X)=\text{Var}\left(X\right)$$

IV. Scaling:

If $a$ and $b$ are constants:

$$\text{Cov}(aX,bY)=ab\text{Cov}(X,Y)$$

V. Addition of Constants:

Adding a constant to a variable does not affect the covariance:

$$\text{Cov}(X+c,Y)=\text{Cov}(X,Y)$$

VI. Relationship with Correlation:

Covariance is related to the correlation coefficient ${\rho }_{XY}$:

$${\rho }_{XY}=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$$

Where ${\sigma }_X$ and ${\sigma }_Y$ are the standard deviations of $X$ and $Y$, respectively.

Sample Covariance

When working with sample data, the sample covariance between two variables $X$ and $Y$ is calculated as:

$$s_{XY}=\text{Cov}(X,Y)=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})$$

Where:

• $n$ is the number of observations.
• $X_i$ and $Y_i$ are the $i$-th observations of variables $X$ and $Y$.
• $\overline{X}$ and $\overline{Y}$ are the sample means of $X$ and $Y$.

Note: The denominator $n-1$ provides an unbiased estimate of the covariance for a sample drawn from a population.

Example: Calculating Covariance Step by Step

Let's calculate the covariance between two variables $X$ and $Y$ using the following dataset:

Step 1: Calculate the Sample Means

Compute the mean of $X$ and $Y$:

$$\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{1+2+3}{3}=\frac{6}{3}=2$$

$$\overline{Y}=\frac{1}{n}\sum _{i=1}^n{Y}_i=\frac{2+4+6}{3}=\frac{12}{3}=4$$

Step 2: Compute the Deviations from the Mean

Calculate $(X_i-\overline{X})$ and $(Y_i-\overline{Y})$:

Step 3: Calculate the Product of Deviations

Compute $(X_i-\overline{X})(Y_i-\overline{Y})$:

Step 4: Sum the Products of Deviations

Compute the sum:

$$\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})=2+0+2=4$$

Step 5: Calculate the Sample Covariance

Use the sample covariance formula:

$$s_{XY}=\text{Cov}(X,Y)=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})$$

Since $n=3$:

$$s_{XY}=\frac{1}{3-1}\times 4=\frac{1}{2}\times 4=2$$

Interpretation:

• The positive covariance of $2$ indicates that $X$ and $Y$ tend to increase together.
• Since the data points lie perfectly on a straight line ($Y=2X$), the covariance reflects a perfect positive linear relationship.

Step 6: Calculate the Variances (Optional)

For completeness, calculate the variances of $X$ and $Y$:

Variance of $X$

$$s_{XX}=\text{Var}\left(X\right)=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2$$

Compute $(X_i-\overline{X}{)}^2$:

Sum:

$$\sum _{i=1}^n(X_i-\overline{X}{)}^2=1+0+1=2$$

Compute variance:

$$s_{XX}=\frac{1}{2}\times 2=1$$

Variance of $Y$

Similarly, compute $(Y_i-\overline{Y}{)}^2$:

Sum:

$$\sum _{i=1}^n(Y_i-\overline{Y}{)}^2=4+0+4=8$$

Compute variance:

$$s_{YY}=\text{Var}\left(Y\right)=\frac{1}{2}\times 8=4$$

Step 7: Calculate the Correlation Coefficient (Optional)

The correlation coefficient $r_{XY}$ standardizes the covariance, providing a dimensionless measure of the strength and direction of the linear relationship:

$$r_{XY}=\frac{s_{XY}}{\sqrt{s_{XX}\times s_{YY}}}=\frac{2}{\sqrt{1\times 4}}=\frac{2}{2}=1$$

Interpretation:

• A correlation coefficient of $1$ indicates a perfect positive linear relationship between $X$ and $Y$.
• This makes sense since $Y=2X$ in the dataset.

Plot:

Limitations of Covariance

I. Scale Dependence:

• Covariance values are affected by the units of measurement of the variables.
• For example, measuring height in meters vs. centimeters will change the covariance.

II. Comparison Difficulties:

• Because covariance is not standardized, comparing covariances across different datasets or variables with different scales is challenging.
• This is why the correlation coefficient, which standardizes covariance, is often used.

III. Not a Measure of Strength:

• The magnitude of covariance does not directly indicate the strength of the relationship.
• A large covariance could be due to large variances rather than a strong relationship.

IV. Linear Relationships Only:

• Covariance measures only linear relationships.
• It does not capture non-linear dependencies between variables.