Pros
- Extremely extensive evaluation
- Has (a little) explanation
- Evaluate off-the-shelf and self-trained LLMs
- Parameter Sweeping
Gadgets
We note that restricting our evaluation to short, localized patches does not necessarily harm the validity of our analysis, as prior work has found that security patches tend to be more localized, have fewer source code modifications, and tend to affect fewer functions, compared to non-security bugs >>>