MIDST: an enhanced development environment that improves the maintainability of a data science analysis

• Published in: International journal of information systems and project management 2020-01, Vol.8 (3), p.5-22
• Main Authors: Saltz, Jeffrey S., Crowston, Kevin, Heckman, Robert, Hegde, Yatish
• Format: Article
• Language: English
• Subjects: Algorithms; Data science; data science development environment; Experiments; maintainability; Project management; Reproducibility; Science; Scientists; Software development; Software engineering; Software upgrading; visual programming
• ISSN: 2182-7788; 2182-7796; 2182-7788
• DOI: 10.12821/ijispm080301

With the increasing ability to generate actionable insight from data, the field of data science has seen significant growth. As more teams develop data science solutions, the analytical code they develop will need to be enhanced in the future, by an existing or a new team member. Thus, the importance of being able to easily maintain and enhance the code required for an analysis will increase. However, to date, there has been minimal research on the maintainability of an analysis done by a data science team. To help address this gap, data science maintainability was explored by (1) creating a data science maintainability model, (2) creating a new tool, called MIDST (Modular Interactive Data Science Tool), that aims to improve data science maintainability, and then (3) conducting a mixed method experiment to evaluate MIDST. The new tool aims to improve the ability of a team member to update and rerun an existing data science analysis by providing a visual data flow view of the analysis within an integrated code and computational environment. Via an analysis of the quantitative and qualitative survey results, the experiment found that MIDST does help improve the maintainability of an analysis. Thus, this research demonstrates the importance of enhanced tools tohelp improve the maintainability of data science projects.