Last modified: October 08, 2024

This article is written in: 🇺🇸

ARMA, ARIMA and SARIMA Models

ARMA, ARIMA, and SARIMA are models commonly used to analyze and forecast time series data. ARMA (AutoRegressive Moving Average) combines two ideas: using past values to predict current ones (autoregression) and smoothing out noise using past forecast errors (moving average). ARIMA (AutoRegressive Integrated Moving Average) builds on ARMA by adding a step to handle trends in non-stationary data through differencing. SARIMA (Seasonal ARIMA) takes it a step further by accounting for repeating seasonal patterns. These models are practical and versatile for working with time series data that show trends, noise, or seasonal effects.

Autoregressive Moving Average (ARMA) Models

ARMA models combine autoregressive (AR) and moving average (MA) components to model time series data exhibiting both autocorrelation and serial dependence.

Mathematical Definition of ARMA Models

An ARMA(p,q) model is defined by:

Xt=c+p∑i=1ϕiXt−i+ϵt+q∑j=1θjϵt−j

or, equivalently, using the backshift operator B:

ϕ(B)Xt=c+θ(B)ϵt

where:

• ϕ(B)=1−ϕ1B−ϕ2B2−⋯−ϕpBp
• θ(B)=1+θ1B+θ2B2+⋯+θqBq

Stationarity of AR Processes

An AR(p) process is stationary if all the roots of the characteristic polynomial ϕ(B)=0 lie outside the unit circle in the complex plane. This condition ensures that the time series has a constant mean and variance over time.

Invertibility of MA Processes

An MA(q) process is invertible if all the roots of θ(B)=0 lie outside the unit circle. Invertibility allows the MA process to be expressed as an infinite AR process, ensuring a unique representation and facilitating parameter estimation.

Infinite Order Representations

AR(∞) Representation of MA Processes:

An MA process can be expressed as an infinite-order AR process:

Xt=∞∑k=1πkXt−k+ϵt

MA(∞) Representation of AR Processes:

An AR process can be expressed as an infinite-order MA process:

Xt=∞∑k=0ψkϵt−k

Example: ARMA(1,1) Process

Consider the ARMA(1,1) model:

Xt=ϕXt−1+ϵt+θϵt−1

Let ϕ=0.7, θ=0.2, and ϵt is white noise.

Simulation

To analyze this process, we simulate a large number of observations using statistical software (e.g., R or Python) to approximate its properties.

set.seed(500)
data <- arima.sim(n = 1e6, list(ar = 0.7, ma = 0.2))

Converting ARMA to Infinite Order Processes

AR(∞) Representation:

(1−ϕB)Xt=(1+θB)ϵt

Xt=(1−ϕB)−1(1+θB)ϵt

Xt=[1+ϕB+ϕ2B2+…](1+θB)ϵt

Multiplying the series:

Xt=[1+(ϕ+θ)B+(ϕ2+ϕθ)B2+…]ϵt

MA(∞) Representation:

Xt=1+θB1−ϕBϵt=[1+ψ1B+ψ2B2+…]ϵt

Calculating ψ coefficients:

ψk=ϕk+θϕk−1

Theoretical Autocorrelations

The autocorrelation function (ACF) for an ARMA(1,1) process is:

ρk=ϕk(1+ϕθ1+2ϕθ+θ2)

Calculations:

ρ1=0.7(1+0.7×0.21+2×0.7×0.2+0.22)≈0.777

ρ2=0.7×ρ1≈0.544

ρ3=0.7×ρ2≈0.381

Results and Interpretation

• The autocorrelations computed from the simulated data closely match the theoretical values, validating the model.
• The ARMA(1,1) model captures both the short-term dependencies (MA component) and the longer-term autocorrelation (AR component).

Autoregressive Integrated Moving Average (ARIMA) Models

ARIMA models generalize ARMA models to include differencing, allowing them to model non-stationary time series data.

Mathematical Definition of ARIMA Models

An ARIMA(p,d,q) model is defined by:

ϕ(B)(1−B)dXt=c+θ(B)ϵt

where:

• ϕ(B): Autoregressive polynomial.
• θ(B): Moving average polynomial.
• d: Order of differencing required to achieve stationarity.

Determining Differencing Order

• Unit Root Tests: Tests like the Augmented Dickey-Fuller (ADF) test assess whether differencing is needed.
• Visual Inspection: Time series plots reveal trends or changing variance.
• ACF Analysis: A slow decay in the ACF suggests non-stationarity.

Fitting ARIMA Models: Numerical Example

Suppose we have a time series Xt exhibiting an upward trend.

Step 1: Differencing

First-order differencing is applied to achieve stationarity:

Yt=(1−B)Xt=Xt−Xt−1

Step 2: Model Identification

Analyzing the differenced series Yt:

• ACF: Significant spikes at lag q suggest an MA(q) component.
• PACF: Significant spikes at lag p suggest an AR(p) component.

Assume ACF suggests MA(1) and PACF suggests AR(1).

Step 3: Parameter Estimation

Fit an ARIMA(1,1,1) model:

(1−ϕB)(1−B)Xt=c+(1+θB)ϵt

Estimate ϕ, θ, and c using MLE.

Step 4: Model Diagnostics

• Residual Analysis: Plot residuals to check for randomness.
• Ljung-Box Test: Confirm absence of autocorrelation in residuals.
• Information Criteria: Compare AIC and BIC values for different models.

Step 5: Forecasting

Use the fitted model to forecast future values:

$$\hat{X}{t+h} = c + \phi \hat{X}{t+h-1} + \theta \hat{\epsilon}_{t+h-1}$$

Seasonal ARIMA Processes (SARIMA)

Seasonal ARIMA (SARIMA) models are widely used for time series data exhibiting both trend and seasonal behaviors.

Mathematical Formulation

A SARIMA(p,d,q×P,D,Q)s model incorporates both non-seasonal and seasonal factors:

ΦP(Bs)ϕp(B)(1−Bs)D(1−B)dXt=ΘQ(Bs)θq(B)ϵt

Non-seasonal components in time series models include several terms:

• The autoregressive (AR) polynomial, represented as ϕp(B), captures the influence of past values on the current value.
• The differencing order, d, represents the number of times the series is differenced to achieve stationarity.
• The moving average (MA) polynomial, θq(B), models the impact of past forecast errors on the current value.

Seasonal components extend these concepts to capture repeating patterns:

• The seasonal AR polynomial, ΦP(Bs), accounts for autoregressive effects at seasonal lags.
• The seasonal differencing order, D, specifies the number of seasonal differences needed to remove seasonal trends.
• The seasonal MA polynomial, ΘQ(Bs), models the impact of seasonal forecast errors.
• The seasonal period, s, defines the length of the seasonal cycle (e.g., 12 for monthly data).

The backshift operator is used to reference previous values in the series:

• For non-seasonal lags, BXt=Xt−1, indicating a one-period shift backward.
• For seasonal lags, BsXt=Xt−s, referencing values from the same season in prior cycles.

Examples of SARIMA Models

Example 1: SARIMA(1, 0, 0)(1, 0, 0)12

Model Equation:

(1−ϕ1B)(1−Φ1B12)Xt=ϵt

• Non-seasonal AR(1): ϕ1BXt
• Seasonal AR(1) with period 12: Φ1B12Xt
• Interpretation: Current value depends on the previous value and the value from 12 periods ago.

Example 2: SARIMA(0, 1, 1)(0, 1, 1)4

Model Equation:

(1−B)(1−B4)Xt=(1+θ1B)(1+Θ1B4)ϵt

• First-order Non-seasonal Differencing: (1−B)Xt
• First-order Seasonal Differencing with period 4: (1−B4)Xt
• Non-seasonal MA(1): θ1Bϵt
• Seasonal MA(1): Θ1B4ϵt

Simplification and Expansion

Non-seasonal Differencing:

(1−B)Xt=Xt−Xt−1

Seasonal Differencing:

(1−Bs)Xt=Xt−Xt−s

Combining Differencing:

(1−B)(1−Bs)Xt=Xt−Xt−1−Xt−s+Xt−s−1

Stationarity and Invertibility Conditions

• Stationarity: All roots of ϕp(B) and ΦP(Bs) polynomials must lie outside the unit circle.
• Invertibility: All roots of θq(B) and ΘQ(Bs) polynomials must lie outside the unit circle.

Seasonal Differencing

• Removes seasonal trends to achieve stationarity.

First-order Seasonal Differencing:

∇sXt=Xt−Xt−s

Second-order Seasonal Differencing:

∇2sXt=Xt−2Xt−s+Xt−2s

Autocorrelation Function (ACF) of SARIMA Processes

The ACF of a SARIMA model displays patterns reflecting both seasonal and non-seasonal behavior.

Example: SARIMA(0, 0, 1)(0, 0, 1)12

Model Specification:

Xt=ϵt+θ1ϵt−1+Θ1ϵt−12+θ1Θ1ϵt−13

Parameters:

• θ1=0.7
• Θ1=0.6

Error Term:

ϵt: White noise with mean zero and variance σ2

Calculating Autocovariances

I. Variance (γ0):

γ0=Var(Xt)=σ2(1+θ21+Θ21+θ21Θ21)

II. Covariance at Lag 1 (γ1):

γ1=Cov(Xt,Xt−1)=σ2θ1(1+Θ21)

III. Covariance at Lag 12 (γ12):

γ12=σ2Θ1(1+θ21)

IV. Covariance at Lag 13 (γ13):

γ13=σ2θ1Θ1(1+θ1Θ1)

Calculating Autocorrelations

ACF at Lag k:

ρk=γkγ0

• Significant spikes at lags that are multiples of the seasonal period (s) indicate seasonal correlations.
• Non-seasonal correlations are observed at lower lags.

Example: Monthly Airline Passenger Data

Consider a time series consisting of monthly airline passenger data. This particular time series is characterized by two distinct features: an upward trend indicating an increase in the number of passengers over time, and a seasonal pattern that repeats every 12 months, typically linked to factors such as holiday travel or seasonal tourism.

Steps to Model and Forecast the Time Series:

I. Decompose the Time Series into Trend, Seasonal, and Residual Components:

• Time series decomposition methods such as classical decomposition and STL decomposition can be applied to analyze the structure of a time series.
• The decomposition process breaks down the original time series into three distinct components to enhance interpretability.
• The trend component reveals the long-term direction of the data, indicating whether it is increasing, decreasing, or remaining stable over time.
• The seasonal component highlights periodic patterns in the data, occurring at consistent intervals such as months, quarters, or years.
• The residual component accounts for random fluctuations or noise that are not explained by the trend or seasonal patterns.

II. Detrend the Time Series Using Differencing or Transformation:

• Methods such as logarithmic transformation and differencing are useful for stabilizing the mean and variance of a time series.
• Differencing involves subtracting the current value from the previous value, which helps remove trends and make the series stationary. Seasonal differencing adjusts for periodic patterns by subtracting the value from the same period in the previous cycle.
• Transformations like logarithmic, square root, or Box-Cox are applied to stabilize variance and reduce heteroscedasticity in the time series. These techniques make it easier to model and analyze the data effectively.

III. Fit an Appropriate Model that Accounts for Both Trend and Seasonality:

• Models such as Seasonal ARIMA (SARIMA) and Exponential Smoothing State Space Model (ETS) are effective for forecasting time series data with seasonal patterns.
• SARIMA extends the ARIMA model by incorporating seasonal terms, making it suitable for data with repeating seasonal behavior. It is defined by the parameters (p, d, q) for the non-seasonal part and (P, D, Q)s for the seasonal part, where "s" represents the seasonal frequency.
• ETS combines error, trend, and seasonal components, making it useful for handling time series with stable seasonal patterns. It adapts to variations in the data and is flexible in modeling additive or multiplicative effects.

IV. Forecast Future Values Using the Fitted Model:

• Use the model to forecast future values.
• This step involves generating predictions based on the model's understanding of the trend and seasonal patterns in the historical data.
• Ensure to provide confidence intervals for these predictions to understand the potential variability in the forecasts.

• The original data is presented in blue, and the trend component is shown in orange, allowing for a clear visualization of long-term trends over time.
• The seasonal component is depicted in green, illustrating the recurring patterns that occur periodically within the data.
• The differenced series is represented in purple, demonstrating how differencing can transform the data to remove trends and achieve stationarity.
• The forecast versus actual values plot combines the original series in blue and the forecasted values in red. Confidence intervals are shaded in black to indicate the forecast's uncertainty.