Abstract
Evaluating cyber defense security mechanisms and techniques is a challenging and manual process. To effectively compare these, security analysts use synthetic benchmarks comprised of simple or artificial test cases, a combination of known vulnerabilities that provide coverage for different classes of attacks, or manually injected bugs in programs of interest [21]. As such, security mechanisms adapt their techniques to guarantee good coverage for the synthetic benchmarks but provide few guarantees of their efficacy beyond that. Furthermore, these generic test cases may fail to represent the security mechanisms' users' environment and expectations. We need automated techniques for injecting realistic and verifiable vulnerabilities to ensure accurate evaluation and comparison of competing security techniques on real-world programs. In our Tunable Cyber Defensive Mechanisms SBIR, we have developed Aikido, a new technique and system for automatically generating and injecting realistic vulnerabilities into real-world applications. Aikido operates on any existing C program, allowing users to create vulnerabilities and evaluate security mechanisms on the applications in which users are most interested. Aikido uses targeted symbolic execution to discover program paths that could be used to generate vulnerabilities. The program paths (i.e., symbolic constraints) are then modified using information from formal methods (e.g., using SMT solvers) to generate and inject new code at the source level that is provably vulnerable (e.g., the system can prove that the generated conditions along a specific program path can lead to a vulnerability). Using previously learned bug patterns, Aikido obfuscates the code to look like native vulnerable code. To enable Aikido to generate vulnerabilities deep inside complex applications, a scenario that is difficult for existing symbolic execution engines to solve due to path explosion and over-constraining paths, it uses goal-directed branch enforcement to select only the relevant conditions required to reach a specific program path. Aikido is implemented as a compiler pass and runtime component in the LLVM Compiler Infrastructure, a set of Clang-based tools, and an orchestration and automation system written in Python. To ease integration and wide use, Aikido has built-in support for Google's OSS-Fuzz system, thus enabling It to seamlessly support any project that employs OSS-Fuzz. We evaluated Aikido on standalone, manually onboarded applications and over 200 OSS-Fuzz applications. Aikido can automatically generate vulnerabilities for many applications. We also identified several important avenues for further development.