Undulate: A framework for data-driven software engineering enabling soft computing

• Published in: Information and software technology 2022-12, Vol.152, p.107039, Article 107039
• Main Authors: Asikainen, Timo, Männistö, Tomi
• Format: Article
• Language: English
• Subjects: Continuous experimentation; Data-driven software engineering; Dimensional database; Multilevel modelling; Soft computing
• ISSN: 0950-5849; 1873-6025
• DOI: 10.1016/j.infsof.2022.107039

Especially web-facing software systems enable the collection of usage data at a massive scale. At the same time, the scale and scope of software processes have grown substantively. Automated tools are needed to increase the speed and quality of controlling software processes. The usage data has great potential as a driver for software processes. However, research still lacks constructs for collecting, refining and utilising usage data in controlling software processes. The objective of this paper is to introduce a framework for data-driven software engineering. The Undulate framework covers generating, collecting and utilising usage data from software processes and business processes supported by the software produced. In addition, we define the concepts and process of extreme continuous experimentation as an exemplar of a software engineering process. We derive requirements for the framework from the research literature, with a focus on papers inspired by practical problems. In addition, we apply a multilevel modelling language to describe the concepts related to extreme continuous experimentation. We introduce the Undulate framework and give requirements and provide an overview of the processes of collecting usage data, augmenting it with additional dimensional data, aggregating the data along the dimensions and computing different metrics based on the data and other metrics. The paper represents significant steps inspired by previous research and practical insight towards standardised processes for data-driven software engineering, enabling the application of soft computing and other methods based on artificial intelligence. •We constructed Undulate, a novel framework for systematically collecting data from software and business processes, storing the data in a dimensional database, thus creating a basis for data-driven software engineering.•We defined the general principles of how soft computing can be used to exercise automated control on and provide decision support for software engineering processes, such as continuous experimentation.•We extended and applied a multilevel modelling language to define key concepts in software experimentation to provide a scientifically rigorous conceptual basis for the Undulate framework.