Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are foundational concepts in linear algebra, with extensive applications across various domains such as physics, computer graphics, and machine learning. These concepts are instrumental in decomposing complex matrix transformations, thereby simplifying numerical computations.

Definitions

An eigenvector of a square matrix $A$ is a non-zero vector $v$ that, when multiplied by $A$, results in a scaled version of $v$. The scalar factor is the eigenvalue corresponding to that eigenvector. In mathematical terms, this relationship is described as:

$$ A v = \lambda v $$

where:

• $A$ is a square matrix,
• $v$ is an eigenvector of $A$,
• $\lambda$ is the corresponding eigenvalue.

Procedure for Finding Eigenvalues and Eigenvectors

1. Eigenvalues: Eigenvalues are calculated by solving the characteristic equation, formulated as $det(A - \lambda I) = 0$. Here, $I$ is the identity matrix of the same dimension as $A$, and $det(\cdot)$ denotes the determinant. The roots of this equation yield the eigenvalues.

2. Eigenvectors: Upon finding each eigenvalue, its corresponding eigenvectors are obtained by substituting the eigenvalue into the equation $(A - \lambda I)v = 0$, followed by computing the null space.

Example

Consider a 2x2 matrix:

$$ A = \begin{bmatrix} 4 & 1 \\ 2 & 3 \\ \end{bmatrix} $$

1. Solve the characteristic equation $det(A - \lambda I) = 0$, which gives $\lambda^2 - 7\lambda + 10 = 0$. The roots of this equation are $\lambda_1 = 2$ and $\lambda_2 = 5$, representing the eigenvalues.

2. To find the corresponding eigenvectors:

3. For $\lambda_1 = 2$, solve the equation $(A - 2I)v = 0$, which yields the eigenvector $v_1 = [1, -2]$.

4. For $\lambda_2 = 5$, solve the equation $(A - 5I)v = 0$, which yields the eigenvector $v_2 = [1, 1]$.

Applications

• Eigenvalues and eigenvectors are used in PCA for dimensionality reduction, enabling data to be represented in fewer dimensions without significant loss of information.
• They are employed in solving systems of linear differential equations, providing a simplified approach to understanding dynamic systems.
• In quantum mechanics, eigenvalues and eigenvectors of operators correspond to physically measurable quantities and their associated states, respectively.

Limitations

• Eigenvalues and eigenvectors are defined only for square matrices. A real matrix may have complex eigenvalues, and not every square matrix has a full set of linearly independent eigenvectors (defective matrices).
• For large matrices, the computation of eigenvalues and eigenvectors can be computationally demanding (about $O(n^3)$ operations for a dense $n \times n$ matrix), potentially slowing down analyses or algorithm implementations.
• Forming the characteristic polynomial and solving for its roots, as in the hand calculation above, is numerically unreliable for larger matrices, especially with repeated eigenvalues; library routines use methods such as the QR algorithm instead.