Principal fitted components as an optimization problem over the Grassmann manifold

Abstract

High-dimensional data where the number of predictor is large, possibly larger than the number of observations, have become increasingly popular in regression analysis. The presence of multicollinearity, in high-dimensional data is inevitable. It is known, that multicollinearity poses a number of series issues to classical statistical techniques, such as the Ordinary Least Squares (OLS) estimator. Thus, regularization techniques are needed. Various regularization methods have been proposed in the literature. These techniques are divided into two classes; (1) Penalized Regression (PR) methods and (2) Dimension Reduction (DR) methods. The aim of this dissertation is to analyze a recent class of DR methodology proposed by Cook (2007), namely Principal Fitted Components (PFC), in which Cook introduced a class of inverse regression models, equipped to capture information about high-dimensional predictors which are non-linearly related to the response. The focus will be on a special member of this class of DR methods known as the Extended PFC (EPFC). It will be shown that Maximum Likelihood (ML) estimation under this model results in a constrained optimization problem, which can be reformulated as an unconstrained optimization problem on the Grassmann manifold. It has been documented in the literature, that these models were actually aimed to cater for low-dimensional data. However, since the interest in this dissertation is in high-dimensionality, an adaptation to the ‘GrassmannOptim’ package, an R-package which implements the EPFC model to low-dimensional data shall be proposed in this dissertation, in order to extend the use of this EPFC model to high-dimensionality. The proposed modification, can be seen as an amalgamation of penalized regression and the EPFC model, which will therefore be called Penalized EPFC (PEPFC). A number of simulated data and a real data shall be used in order to assess numerically the performance of the proposed model.