Automatically finding bugs in a commercial cyber-physical system development tool chain with SLforge

• Main Authors: Chowdhury, Shafiul Azam, Mohian, Soumik, Mehra, Sidharth, Gawsane, Siddhant, Johnson, Taylor T., Csallner, Christoph
• Format: Conference Proceeding
• Language: English
• Subjects: Computer bugs; cyber-physical system; Cyber-physical systems; differential testing; Numerical models; Object oriented modeling; Simulink; Software and its engineering > Software creation and management > Software verification and validation > Software defect analysis > Software testing and debugging; Software and its engineering > Software organization and properties > Software system structures > Software system models > Model-driven software engineering; Software packages; Testing; tool chain bug
• ISSN: 1558-1225
• DOI: 10.1145/3180155.3180231

Cyber-physical system (CPS) development tool chains are widely used in the design, simulation, and verification of CPS data-flow models. Commercial CPS tool chains such as MathWorks' Simulink generate artifacts such as code binaries that are widely deployed in embedded systems. Hardening such tool chains by testing is crucial since formally verifying them is currently infeasible. Existing differential testing frameworks such as CyFuzz can not generate models rich in language features, partly because these tool chains do not leverage the available informal Simulink specifications. Furthermore, no study of existing Simulink models is available, which could guide CyFuzz to generate realistic models. To address these shortcomings, we created the first large collection of public Simulink models and used the collected models' properties to guide random model generation. To further guide model generation we systematically collected semi-formal Simulink specifications. In our experiments on several hundred models, the resulting SLforge generator was more effective and efficient than the state-of-the-art tool CyFuzz. SLforge also found 8 new confirmed bugs in Simulink.