RETORCH: A Cost and Resource aware Model for E2E Testing in the Cloud

• Published in: The Journal of systems and software 2025-03, Vol.221, p.112237, Article 112237
• Main Authors: Augusto, Cristian, Morán, Jesús, Bertolino, Antonia, de la Riva, Claudio, Tuya, Javier
• Format: Article
• Language: English
• Subjects: Cloud computing; E2E testing; End-to-end testing; Software engineering; Software testing; Test optimization
• ISSN: 0164-1212
• DOI: 10.1016/j.jss.2024.112237

•The RETORCH* model that enables representing cloud and test configurations and estimating the cost of executing the E2E test suite.•An approach to optimize the cost-efficiency of E2E test execution based on the cloud and test configuration model.•E2E test execution configuration depends on tester objectives, but the RETORCH* model allows for informed and fine-grained selection. Moving testing to the Cloud overcomes time/resource constraints by leveraging an unlimited and elastic infrastructure, especially for testing levels like End-to-End (E2E) that require a high number of resources and/or execution time. However, it introduces new challenges to those already faced on-premises, like selecting the most suitable Cloud infrastructure and billing scheme. We propose the RETORCH* test execution model that estimates and compares the monetary cost of executing an E2E test suite with different Cloud alternatives, billing schemes, and test configurations. RETORCH* goes beyond the mere cost billed, and selects the solution that best aligns with the test team strategy using the data of on-premises prior executions and the tester's experience. This cost is broken down into the cost incurred to execute the test suite (testing cost) and possible unused infrastructure (overprovisioning cost). Based on these distinct costs, the test team can compare different Cloud and test configurations. RETORCH* has been evaluated using a real-world application's E2E test suite. We analyze how the different decisions taken when the suite is migrated to the Cloud impact the cost, highlighting how RETORCH* can help the tester during Cloud and test configuration to make a more informed decision.