Long-Memory Dynamic Factor Model (LMDFM): Scalable Non-Stationary DFM

The Long-Memory Dynamic Factor Model (LMDFM) is a scalable non-stationary (i.e. time-varying) DFM algorithm. All model coefficients and outputs are estimated by an implementation of spectral principal components analysis (spectral PCA) framework with conjugate two-dimensional discrete Fourier transform (C2D-DFT) technique.

The model estimation methodology underlying LMDFM is detailed step by step in "Introduction to Non-Stationary Dynamic Factor Models Estimated by Spectral PCA with Conjugate 2D-DFT", https://github.com/i4cast/aws/blob/main/long_memory_dynamic_factor_model/publication/Non-stationary_DFMs_estimated_by_spectral_PCA_and_conjugate_2D-DFT.pdf .

The LMDFM modeling is to make (1) dynamic factor analysis of observed multiple (vector) time-series, (2) multi-step multivariate forecasts of vector time-series, and (3) multi-step forecasts of multivariate volatility (variance-covariance) of vector time-series.

The LMDFM algorithm estimates (a) non-stationary dynamic factor loadings matrixes, (b) time-varying vector autoregressive (VAR) coefficients of common factors, (c) non-stationary variances and vector autocovariances of factors, and (d) variances of stochastic components.

Inputs into, outputs from, and functions of, the LMDFM algorithm are described in details in "help(LMDFM)", a Python module help document, https://github.com/i4cast/aws/blob/main/long_memory_dynamic_factor_model/notebook/help(LMDFM ).

INPUT data taken by LMDFM algorithm are multiple, or large number of, time-series. All time-series data are stored in a CSV (comma separated value) text data table, in a format of a CSV text data string or a CSV text data file. Each row of the table contains values of an individual time-series. Row header is the label or symbol of the time-series. Each column contains values of all time-series at a given moment in time. Column header is the time-index or time-stamp of the moment in time. The first data column is for the earliest time and the last column for the most recent time.

The current version of LMDFM requires equally spaced time-indexes. An example of CSV text data table containing multiple time-series input data for LMDFM algorithm is: https://github.com/i4cast/aws/blob/main/long_memory_dynamic_factor_model/input/Weekly_VTS_6Yr.csv .

OUTOUTs from LMDFM algorithm are multi-step forecasts of multivariate time-series, or of multivariate volatility (variance-covariance matrix). Additional outputs include a variety of matrixes of model coefficients/parameters, as well as common and idiosyncratic components of all time-series. Output data in format of CSV text data tables can be used to make quick review by loading them into a spreadsheet application. Output data in format of JSON text data strings can be used as input data for further analysis by other models/algorithms.

A set of examples of various outputs from LMDFM algorithm can be review by opening CSV or JSON files shown at: https://github.com/i4cast/aws/tree/main/long_memory_dynamic_factor_model/output/ .

A step-by-step, item-by-item, demo notebook exemplifying how to use LMDFM algorithm on Amazon AWS Sagemaker platform is developed for users of LMDFM: https://github.com/i4cast/aws/blob/main/long_memory_dynamic_factor_model/notebook/demo_lmdfm_aws.ipynb .

Here are examples of observed big datasets: Measurements and indicators of multinational economies. Prices of financial or physical or other items continuously traded in markets. Measurements of natural or engineering processes. Social or political trends. Scores of sports. etc.

The LMDFM algorithm estimates, but does not model, idiosyncratic components of observed time-series. To model the balancing idiosyncratic component time-series, we developed YWpcAR (Yule-Walker-PCA Autoregression) algorithm. Therefore, other models built by i4cast LLC, such as LMVAR, DFVCM and DFbVIF algorithms, take both LMDFM and YWpcAR as prerequisite underlying models, analyze and forecast common components by LMDFM and, at the same time, analyze and forecast all individual idiosyncratic time-series by YWpcAR.