Extracting Structured Labor Market Information from Job Postings with Generative AI

• Published in: Digital government (New York, N.Y. Online) N.Y. Online), 2024-07
• Main Authors: Howison, Mark, Ensor, William O., Maharjan, Suraj, Parikh, Rahil, Sengamedu, Srinivasan H., Daniels, Paul, Gaither, Amber, Yeats, Carrie, Reddy, Chandan K., Hastings, Justine S.
• Format: Article
• Language: English
• ISSN: 2691-199X; 2639-0175
• DOI: 10.1145/3674847

Labor market information is an important input to labor, workforce, education, and macroeconomic policy. However, granular and real-time data on labor market trends are lacking; publicly available data from survey samples are released with significant lags and miss critical information such as skills and benefits. We use generative Artificial Intelligence to automatically extract structured labor market information from unstructured online job postings for the entire U.S. labor market. To demonstrate our methodology, we construct a sample of 6,800 job postings stratified by 68 major occupational groups, extract structured information on educational requirements, remote-work flexibility, full-time availability, and benefits, and show how these job characteristics vary across occupations. As a validation, we compare frequencies of educational requirements by occupation from our sample to survey data and find no statistically significant difference. Finally, we discuss the scalability to collections of millions of job postings. Our results establish the feasibility of measuring labor market trends at scale from online job postings thanks to advances in generative AI techniques. Improved access to such insights at scale and in real-time could transform the ability of policy leaders, including federal and state agencies and education providers, to make data-informed decisions that better support the American workforce.