Software Process Evaluation from User Perceptions and Log Data

• Published in: Journal of software : evolution and process 2022-04, Vol.34 (4), p.n/a
• Main Authors: Vavpotič, Damjan, Bala, Saimir, Mendling, Jan, Hovelja, Tomaž
• Format: Article
• Language: English
• Subjects: case study; evaluation; Managers; Perceptions; Software; software development methodology; software development tool logs; Software development tools; stakeholder perceptions
• ISSN: 2047-7473; 2047-7481
• DOI: 10.1002/smr.2438

Companies often claim to follow specific software development methodologies (SDM) when performing their software development process. These methodologies are often supported by dedicated tools that keep track of work activities carried out by developers. The purpose of this paper is to provide a novel approach that integrates analytical insights from both the perceptions of SDM stakeholders and software development tools logs to provide SDM improvement recommendations. This paper develops a new process improvement approach that combines two significantly different sources of data on the same phenomenon. First, it uses a questionnaire to gather software development stakeholder SDM perceptions (managers and developers). Second, it leverages process mining to analyze software development tools logs to obtain additional information on software development activities. Finally, it develops recommendations based on concurrent analysis of both sources. Our novel process improvement approach is evaluated in three directions: Does the presented approach (RQ1) enable managers to gain additional insights into employees' performance, (RQ2) deliver additional insights into project performance, and (RQ3) enable development of additional SDM improvement recommendations? We find that integrated analysis of software development perception data and software development tools logs opens new possibilities to more precisely identify and improve specific SDM elements. The evaluation of our novel process improvement approach follows a single case study design. Our approach can only be used in enterprises in which software development tools logs are available. The study should be repeated in different cultural settings. We practically show how concurrently analyzing data about developer SDM perceptions and event log data from software development tools enables management to gain additional insights in the software development process regarding the performance of individual developers. The main theoretical contribution of our paper is a novel process improvement approach that effectively integrates data from management and developer perspectives and software development tools logs. To understand how the different elements composing software development methodologies (SDM) are perceived versus how they are actuated in practice, this paper develops a novel approach that combines subjective understanding of the involved people and factual analyses from trace data. Our findings lead to