Developer Testing in the IDE: Patterns, Beliefs, and Behavior

Software testing is one of the key activities to achieve software quality in practice. Despite its importance, however, we have a remarkable lack of knowledge on how developers test in real-world projects. In this paper, we report on a large-scale field study with 2,443 software engineers whose deve...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on software engineering 2019-03, Vol.45 (3), p.261-284
Main Authors: Beller, Moritz, Gousios, Georgios, Panichella, Annibale, Proksch, Sebastian, Amann, Sven, Zaidman, Andy
Format: Article
Language:English
Subjects:
Androids
Developer testing
field study
Humanoid robots
Java
JUnit
KaVE FeedBag
Programming environments
Programming languages
Servers
Software
Software development
Software testing
test-driven development (TDD)
Testing
testing effort
TestRoots WatchDog
unit tests
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Software testing is one of the key activities to achieve software quality in practice. Despite its importance, however, we have a remarkable lack of knowledge on how developers test in real-world projects. In this paper, we report on a large-scale field study with 2,443 software engineers whose development activities we closely monitored over 2.5 years in four integrated development environments (IDEs). Our findings, which largely generalized across the studied IDEs and programming languages Java and C#, question several commonly shared assumptions and beliefs about developer testing: half of the developers in our study do not test; developers rarely run their tests in the IDE; most programming sessions end without any test execution; only once they start testing, do they do it extensively; a quarter of test cases is responsible for three quarters of all test failures; 12 percent of tests show flaky behavior; Test-Driven Development (TDD) is not widely practiced; and software developers only spend a quarter of their time engineering tests, whereas they think they test half of their time. We summarize these practices of loosely guiding one's development efforts with the help of testing in an initial summary on Test-Guided Development (TGD), a behavior we argue to be closer to the development reality of most developers than TDD.
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2017.2776152