Identifying outlier loci in admixed and in continuous populations using ancestral population differentiation statistics

• Published in: Molecular ecology 2016-10, Vol.25 (20), p.5029-5042
• Main Authors: Martins, Helena, Caye, Kevin, Luu, Keurcien, Blum, Michael G. B., François, Olivier
• Format: Article
• Language: English
• Subjects: Adaptation; Adaptation, Biological - genetics; admixed populations; Arabidopsis - genetics; Arabidopsis thaliana; Botany; Computer Simulation; Gene Frequency; Gene loci; Genetic Loci; Genetics, Population - methods; genome scans for selection; Genome, Human; Genomics - methods; Humans; inference of population structure; Models, Genetic; Polymorphism, Single Nucleotide; population differentiation tests; Population genetics; Selection, Genetic; Statistics
• ISSN: 0962-1083; 1365-294X
• DOI: 10.1111/mec.13822

Finding genetic signatures of local adaptation is of great interest for many population genetic studies. Common approaches to sorting selective loci from their genomic background focus on the extreme values of the fixation index, FST, across loci. However, the computation of the fixation index becomes challenging when the population is genetically continuous, when predefining subpopulations is a difficult task, and in the presence of admixed individuals in the sample. In this study, we present a new method to identify loci under selection based on an extension of the FST statistic to samples with admixed individuals. In our approach, FST values are computed from the ancestry coefficients obtained with ancestry estimation programs. More specifically, we used factor models to estimate FST, and we compared our neutrality tests with those derived from a principal component analysis approach. The performances of the tests were illustrated using simulated data and by re‐analysing genomic data from European lines of the plant species Arabidopsis thaliana and human genomic data from the population reference sample, POPRES.