Don’t Look UB: Exposing Sanitizer-Eliding Compiler Optimizations

• Published in: Proceedings of ACM on programming languages 2023-06, Vol.7 (PLDI), p.907-927, Article 143
• Main Authors: Isemann, Raphael, Giuffrida, Cristiano, Bos, Herbert, van der Kouwe, Erik, Gleissenthall, Klaus von
• Format: Article
• Language: English
• Subjects: Security and privacy; Software security engineering
• ISSN: 2475-1421; 2475-1421
• DOI: 10.1145/3591257

Sanitizers are widely used compiler features that detect undefined behavior and resulting vulnerabilities by injecting runtime checks into programs. For better performance, sanitizers are often used in conjunction with optimization passes. But doing so combines two compiler features with conflicting objectives. While sanitizers want to expose undefined behavior, optimizers often exploit these same properties for performance. In this paper, we show that this clash can have serious consequences: optimizations can remove sanitizer failures, thereby hiding the presence of bugs or even introducing new ones. We present LookUB, a differential-testing based framework for finding optimizer transformations that elide sanitizer failures. We used our method to find 17 such sanitizer-eliding optimizations in Clang. Next, we used static analysis and fuzzing to search for bugs in open-source projects that were previously hidden due to sanitizer-eliding optimizations. This led us to discover 20 new bugs in Linux Containers, libmpeg2, NTFS-3G, and WINE. Finally, we present an effective mitigation strategy based on a customization of the Clang optimizer with an overhead increase of 4%.