Selective Inference for Hierarchical Clustering

Classical tests for a difference in means control the Type I error rate when the groups are defined a priori. However, when the groups are instead defined via clustering, then applying a classical test yields an extremely inflated Type I error rate. Notably, this problem persists even if two separat...

Full description

Saved in:
Bibliographic Details
Published in:Journal of the American Statistical Association 2024-01, Vol.119 (545), p.332-342
Main Authors: Gao, Lucy L., Bien, Jacob, Witten, Daniela
Format: Article
Language:English
Subjects:
Americans
Cluster analysis
Clustering
data collection
Difference in means
Errors
Gene sequencing
Hypothesis testing
Inference
Null hypothesis
Post-selection inference
sequence analysis
Statistics
testing
Tests
Type I error
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Classical tests for a difference in means control the Type I error rate when the groups are defined a priori. However, when the groups are instead defined via clustering, then applying a classical test yields an extremely inflated Type I error rate. Notably, this problem persists even if two separate and independent datasets are used to define the groups and to test for a difference in their means. To address this problem, in this article, we propose a selective inference approach to test for a difference in means between two clusters. Our procedure controls the selective Type I error rate by accounting for the fact that the choice of null hypothesis was made based on the data. We describe how to efficiently compute exact p-values for clusters obtained using agglomerative hierarchical clustering with many commonly used linkages. We apply our method to simulated data and to single-cell RNA-sequencing data. Supplementary materials for this article are available online.
ISSN:0162-1459
1537-274X
1537-274X
DOI:10.1080/01621459.2022.2116331