Tests and variables selection on regression analysis for massive datasets

According to Lindley’s paradox, most point null hypotheses will be rejected when the sample size is too large. In this paper, a two-stage block testing procedure is proposed for massive data regression analysis. New variables selection criteria incorporating with classical stepwise procedure are als...

Full description

Saved in:
Bibliographic Details
Published in:Data & knowledge engineering 2007-12, Vol.63 (3), p.811-819
Main Authors: Fan, Tsai-Hung, Cheng, Kuang-Fu
Format: Article
Language:English
Subjects:
Hypothesis testing
Lindley’s paradox
Massive data
Regression analysis
Stepwise selection
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:According to Lindley’s paradox, most point null hypotheses will be rejected when the sample size is too large. In this paper, a two-stage block testing procedure is proposed for massive data regression analysis. New variables selection criteria incorporating with classical stepwise procedure are also developed to select significant explanatory variables. Our approach is not only simple in computation for massive data but also confirmed by the simulation study that our approach is more accurate in the sense of achieving the nominal significance level for huge data sets. A real example with moderate sample size verifies that the proposed procedure is accurate compared with the classical method, and a huge real data set is also demonstrated to select appropriate regressors.
ISSN:0169-023X
1872-6933
DOI:10.1016/j.datak.2007.05.001