A Comprehensive Benchmark on Java Cryptographic API Misuses

Misuses of cryptographic APIs are prevalent in existing real-world Java code. Some open-sourced and commercial cryptographic vulnerability detection tools exist that capture misuses in Java program. To analyze their efficiency and coverage, we build a comprehensive benchmark named CryptoAPI-Bench that consists of 171 unit test cases. The test cases include basic cases and complex cases. We assess four tools i.e., SpotBugs, CryptoGuard, CrySL, and Coverity using CryptoAPI-Bench and show their relative performance.