Do concern mining tools really help requirements analysts? An empirical study of the vetting process

• Published in: The Journal of systems and software 2019-10, Vol.156, p.181-203
• Main Authors: Rago, Alejandro, Diaz-Pace, J. Andres, Marcos, Claudia
• Format: Article
• Language: English
• Subjects: Crosscutting concern; Empirical study; Human behavior; Requirements engineering; Tool support; Use case specifications
• ISSN: 0164-1212; 1873-1228
• DOI: 10.1016/j.jss.2019.06.073

•Empirical evaluation of concern vetting with 55 subjects in 3 systems and 3 tools.•Time does not have an effect on the final quality achieved by analysts.•The initial quality of a solution is a crucial factor to achieve satisfactory results.•The concern mining tools exhibited performance variations, although they were not statistically significant in our case-studies.•Analysts work best using solutions with high recall rather than with high precision. Software requirements are often described in natural language because they are useful to communicate and validate. Due to their focus on particular facets of a system, this kind of specifications tends to keep relevant concerns (also known as early aspects) from the analysts’ view. These concerns are known as crosscutting concerns because they appear scattered among documents. Concern mining tools can help analysts to uncover concerns latent in the text and bring them to their attention. Nonetheless, analysts are responsible for vetting tool-generated solutions, because the detection of concerns is currently far from perfect. In this article, we empirically investigate the role of analysts in the concern vetting process, which has been little studied in the literature. In particular, we report on the behavior and performance of 55 subjects in three case-studies working with solutions produced by two different tools, assessed in terms of binary classification measures. We discovered that analysts can improve “bad” solutions to a great extent, but performed significantly better with “good” solutions. We also noticed that the vetting time is not a decisive factor to their final accuracy. Finally, we observed that subjects working with solutions substantially different from those of existing tools (better recall) can also achieve a good performance.