Cross-evaluation of metrics to estimate the significance of creative works

• Published in: Proceedings of the National Academy of Sciences - PNAS 2015-02, Vol.112 (5), p.1281-1286
• Main Authors: Wasserman, Max, Zeng, Xiao Han T., Amaral, Luís A. Nunes
• Format: Article
• Language: English
• Subjects: Creativity; Effectiveness; Estimating techniques; Motion pictures; Physical Sciences; Social Sciences
• ISSN: 0027-8424; 1091-6490
• DOI: 10.1073/pnas.1412198112

In a world overflowing with creative works, it is useful to be able to filter out the unimportant works so that the significant ones can be identified and thereby absorbed. An automated method could provide an objective approach for evaluating the significance of works on a universal scale. However, there have been few attempts at creating such a measure, and there are few “ground truths” for validating the effectiveness of potential metrics for significance. For movies, the US Library of Congress’s National Film Registry (NFR) contains American films that are “culturally, historically, or aesthetically significant” as chosen through a careful evaluation and deliberation process. By analyzing a network of citations between 15,425 United States-produced films procured from the Internet Movie Database (IMDb), we obtain several automated metrics for significance. The best of these metrics is able to indicate a film’s presence in the NFR at least as well or better than metrics based on aggregated expert opinions or large population surveys. Importantly, automated metrics can easily be applied to older films for which no other rating may be available. Our results may have implications for the evaluation of other creative works such as scientific research. Significance Whether it is Hollywood movies or research papers, identifying works of great significance is imperative in a modern society overflowing with information. Through analysis of a network constructed from citations between films as referenced in the Internet Movie Database, we obtain several automated metrics for significance. We find that the best automated method can identify significant films, represented by selection to the US National Film Registry, at least as well as the aggregate rating of many experts and far better than the rating of a single expert. We hypothesize that these results may hold for other creative works.