Automatic creation of acceptance tests by extracting conditionals from requirements: NLP approach and case study

Acceptance testing is crucial to determine whether a system fulfills end-user requirements. However, the creation of acceptance tests is a laborious task entailing two major challenges: (1) practitioners need to determine the right set of test cases that fully covers a requirement, and (2) they need...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of systems and software 2023-03, Vol.197, p.111549, Article 111549
Main Authors: Fischbach, Jannik, Frattini, Julian, Vogelsang, Andreas, Mendez, Daniel, Unterkalmsteiner, Michael, Wehrle, Andreas, Henao, Pablo Restrepo, Yousefi, Parisa, Juricic, Tedi, Radduenz, Jeannette, Wiecher, Carsten
Format: Article
Language:English
Subjects:
Acceptance testing
Acceptance tests
Automatic creations
Automatic test case creation
Case-studies
Causality extraction
Language processing
Natural language processing
Natural language processing systems
Natural languages
Requirement engineering
Requirements engineering
Software testing
Test case
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Acceptance testing is crucial to determine whether a system fulfills end-user requirements. However, the creation of acceptance tests is a laborious task entailing two major challenges: (1) practitioners need to determine the right set of test cases that fully covers a requirement, and (2) they need to create test cases manually due to insufficient tool support. Existing approaches for automatically deriving test cases require semi-formal or even formal notations of requirements, though unrestricted natural language is prevalent in practice. In this paper, we present our tool-supported approach CiRA (Conditionals in Requirements Artifacts) capable of creating the minimal set of required test cases from conditional statements in informal requirements. We demonstrate the feasibility of CiRA in a case study with three industry partners. In our study, out of 578 manually created test cases, 71.8 % can be generated automatically. Additionally, CiRA discovered 80 relevant test cases that were missed in manual test case design. CiRA is publicly available at www.cira.bth.se/demo/. •Functional requirements often describe system behavior by conditionals.•We present an approach capable of extracting conditionals in fine-grained form.•We show how conditional extraction can help to create acceptance tests automatically.•We demonstrate the feasibility of our approach in a case study with three companies.•Out of 578 manually created test cases, 71.8% could be generated automatically.
ISSN:0164-1212
1873-1228
1873-1228
DOI:10.1016/j.jss.2022.111549