Data Scientists in Software Teams: State of the Art and Challenges

The demand for analyzing large scale telemetry, machine, and quality data is rapidly increasing in software industry. Data scientists are becoming popular within software teams, e.g., Facebook, LinkedIn and Microsoft are creating a new career path for data scientists. In this paper, we present a lar...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on software engineering 2018-11, Vol.44 (11), p.1024-1038
Main Authors: Kim, Miryung, Zimmermann, Thomas, DeLine, Robert, Begel, Andrew
Format: Article
Language:English
Subjects:
Best practices
Data science
Demand analysis
development roles
industry
Interviews
Scientists
Sociology
Software
Software engineering
State of the art
Statistics
Telemetry
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The demand for analyzing large scale telemetry, machine, and quality data is rapidly increasing in software industry. Data scientists are becoming popular within software teams, e.g., Facebook, LinkedIn and Microsoft are creating a new career path for data scientists. In this paper, we present a large-scale survey with 793 professional data scientists at Microsoft to understand their educational background, problem topics that they work on, tool usages, and activities. We cluster these data scientists based on the time spent for various activities and identify 9 distinct clusters of data scientists, and their corresponding characteristics. We also discuss the challenges that they face and the best practices they share with other data scientists. Our study finds several trends about data scientists in the software engineering context at Microsoft, and should inform managers on how to leverage data science capability effectively within their teams.
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2017.2754374