Correlation

Correlation is a statistical measure that quantifies the strength and direction of the linear relationship between two variables. It is a fundamental concept in statistics, enabling researchers and analysts to understand how one variable may predict or relate to another. The most commonly used correlation coefficients are the Pearson correlation coefficient and the Spearman rank correlation coefficient.

Understanding Correlation

• A positive correlation occurs when, as one variable increases, the other tends to rise as well.
• In contrast, a negative correlation happens when an increase in one variable results in the other decreasing.
• Lastly, a zero correlation indicates that there is no linear relationship between the variables.

Important Note: Correlation does not imply causation. A high correlation between two variables does not mean that one variable causes changes in the other.

Pearson Correlation Coefficient ($r$)

The Pearson correlation coefficient measures the strength and direction of the linear relationship between two continuous variables. It is sensitive to outliers and summarizes only linear association. It can be computed for any data with positive variances; normality is needed only for exact small-sample inference (confidence intervals and tests) about the population correlation.

Definition

The Pearson correlation coefficient $r$ is defined as:

$$ r = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} $$

Where:

• $\text{Cov}(X, Y)$ is the covariance between variables $X$ and $Y$.
• $\sigma_X$ and $\sigma_Y$ are the standard deviations of $X$ and $Y$, respectively.

The same formula is used for the population correlation (with population moments) and for the sample correlation (with sample moments; the factors $1/(n-1)$ cancel between numerator and denominator).

Alternative Formula:

$$ r = \frac{\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})}{\sqrt{\sum_{i=1}^{n} (X_i - \bar{X})^2} \sqrt{\sum_{i=1}^{n} (Y_i - \bar{Y})^2}} $$

Where:

• $n$ is the number of observations.
• $X_i$ and $Y_i$ are the $i$-th observations of $X$ and $Y$.
• $\bar{X}$ and $\bar{Y}$ are the sample means of $X$ and $Y$.

Interpretation

• $r = 1$: Perfect positive linear correlation (all points on a line with positive slope).
• $r = -1$: Perfect negative linear correlation.
• $r = 0$: No linear correlation.

General Guidelines:

These cut-offs are conventional rules of thumb, not universal; what counts as "strong" depends on the field.

Example: Calculating Pearson's $r$

Dataset

Consider the following data on the number of hours studied ($X$) and test scores ($Y$):

Step 1: Calculate the Means

$$ \bar{X} = \frac{1}{5}(1 + 2 + 3 + 4 + 5) = \frac{15}{5} = 3 $$

$$ \bar{Y} = \frac{1}{5}(50 + 60 + 70 + 80 + 90) = \frac{350}{5} = 70 $$

Step 2: Calculate Deviations and Products

Compute $(X_i - \bar{X})$, $(Y_i - \bar{Y})$, and their products:

Step 3: Compute the Pearson Correlation Coefficient

$$ r = \frac{\sum_{i=1}^{5} (X_i - \bar{X})(Y_i - \bar{Y})}{\sqrt{\sum_{i=1}^{5} (X_i - \bar{X})^2} \sqrt{\sum_{i=1}^{5} (Y_i - \bar{Y})^2}} = \frac{100}{\sqrt{10} \times \sqrt{1000}} $$

Compute the denominators:

$$ \sqrt{10} \approx 3.1623, \quad \sqrt{1000} \approx 31.6228 $$

Compute $r$:

$$ r = \frac{100}{3.1623 \times 31.6228} = \frac{100}{100} = 1 $$

Visualization

Pearson's $r = 1$ indicates a perfect positive linear relationship between hours studied and test scores. As the number of hours studied increases, the test score increases proportionally.

Spearman Rank Correlation Coefficient ($r_s$)

The Spearman rank correlation coefficient measures the strength and direction of the monotonic relationship between two variables. It is the Pearson correlation computed on the ranks of the data. It is a non-parametric measure, meaning it does not assume a specific distribution for the variables and is less sensitive to outliers.

Definition

The Spearman correlation coefficient $r_s$ is defined as:

$$ r_s = 1 - \frac{6 \sum_{i=1}^{n} d_i^2}{n(n^2 - 1)} $$

Where:

• $d_i = R(X_i) - R(Y_i)$ is the difference between the ranks of $X_i$ and $Y_i$.
• $R(X_i)$ and $R(Y_i)$ are the ranks of $X_i$ and $Y_i$, respectively.
• $n$ is the number of observations.

This shortcut formula is exact only when there are no ties. With ties, assign average ranks and compute the Pearson correlation of the ranks instead.

Calculation Steps

1. Assign Ranks to the data points in $X$ and $Y$ separately.
2. Compute the Differences of Ranks $d_i$.
3. Square the Differences $d_i^2$.
4. Compute $r_s$ using the formula.

Example: Calculating Spearman's $r_s$

Using the same dataset:

Step 1: Assign Ranks

Since the data is already ordered, the ranks correspond to the order of observations.

Step 2: Compute Differences of Ranks

Calculate $d_i = R(X_i) - R(Y_i)$:

Step 3: Compute Spearman's $r_s$

$$ r_s = 1 - \frac{6 \times 0}{5(5^2 - 1)} = 1 - 0 = 1 $$

Interpretation

Spearman's $r_s = 1$ indicates a perfect positive monotonic relationship between hours studied and test scores.

When to Use Spearman's $r_s$

• When data is ordinal.
• When the relationship between variables is monotonic but not necessarily linear.
• When there are outliers that might affect Pearson's $r$.
• When the variables are not normally distributed.

Comparison of Pearson's $r$ and Spearman's $r_s$

• Pearson's $r$ measures the strength of a linear relationship.
• Spearman's $r_s$ measures the strength of a monotonic relationship.
• Both coefficients range from -1 to 1.
• $r_s = \pm 1$ whenever the relationship is perfectly monotonic, whereas $r = \pm 1$ requires an exact linear relationship.
• Spearman's $r_s$ is less sensitive to outliers and does not require the assumption of normality.

Correlation of Two Random Variables

For two random variables $X$ and $Y$ with positive variances, the population correlation coefficient $\rho_{XY}$ is defined as:

$$ \rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} $$

Where:

• $\text{Cov}(X, Y)$ is the covariance between $X$ and $Y$.
• $\sigma_X$ and $\sigma_Y$ are the standard deviations of $X$ and $Y$.

Properties

Interpretation

• $\rho_{XY} > 0$: Positive linear relationship.
• $\rho_{XY} < 0$: Negative linear relationship.
• $\rho_{XY} = 0$: No linear relationship.

Example with Random Variables

Suppose $X$ and $Y$ are random variables with the following properties:

• $\mu_X = \mathbb{E}[X] = 5$
• $\mu_Y = \mathbb{E}[Y] = 10$
• $\sigma_X = \sqrt{\text{Var}(X)} = 2$
• $\sigma_Y = \sqrt{\text{Var}(Y)} = 4$
• $\text{Cov}(X, Y) = 6$

Compute the correlation coefficient:

$$ \rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} = \frac{6}{2 \times 4} = \frac{6}{8} = 0.75 $$

Interpretation:

• A correlation coefficient of $0.75$ indicates a strong positive linear relationship between $X$ and $Y$ by the conventional guidelines above.
• The covariance is consistent with the bound $|\text{Cov}(X,Y)|\le\sigma_X\sigma_Y=8$.

Important Considerations

Correlation vs. Causation

• Correlation does not imply causation. A high correlation between two variables does not mean that one variable causes changes in the other.
• A confounding variable (or lurking variable) may influence both correlated variables, creating a statistical association without a direct causal link. For example, ice cream sales and drowning incidents are positively correlated, but neither causes the other — the confounding variable is hot weather, which independently increases both.
• Spurious correlations can arise purely by chance, especially when testing many variable pairs simultaneously or when working with small sample sizes. Always consider the underlying mechanism before drawing conclusions from observed correlations.

Outliers

• Pearson's $r$ is sensitive to outliers, which can distort the correlation coefficient.
• Spearman's $r_s$ is more robust to outliers due to ranking.

Non-linear Relationships

• Variables can have a strong non-linear relationship but a low Pearson correlation coefficient.
• If the relationship is monotonic (for example $Y=e^X$), Spearman's $r_s$ will detect it.
• If the relationship is non-monotonic (for example $Y=X^2$ with $X$ symmetric about zero), both coefficients can be near zero even though $Y$ is a function of $X$.

Assumptions of Pearson's $r$

1. Linearity: the relationship between $X$ and $Y$ should be approximately straight, because $r$ measures only linear association.
2. No extreme outliers or leverage points, which can dominate $r$.
3. Approximate bivariate normality (and independent observations) is needed for the usual $t$-test and Fisher-$z$ confidence interval for $\rho_{XY}$, but not for computing $r$ itself.
4. Constant spread (homoscedasticity) is not required for $r$ to be defined, but a funnel-shaped scatter means a single $r$ summarizes the relationship poorly.

If these conditions are violated, Pearson's $r$ may not be an appropriate summary, and a rank-based or other measure may be preferable.