Enhancing Code Security Through Open-Source Large Language Models: A Comparative Study

Significant advances in the language processing field are providing new innovations, including the ability to analyze code for weaknesses. Typically, analyzing code security is performed by tools that use known vulnerable patterns, which may not adequately represent the intricacies of vulnerabilities in real-world projects. Such tools can fail to detect non-standard weaknesses in code samples, potentially leading to a loss of personal and financial information for end users of the code. Using language-based models to detect weaknesses that would have otherwise been missed by the currently available analysis tools is a promising new avenue of vulnerability detection. In this research, we employ 25 different models to evaluate the security of code samples. Using an existing dataset of insecure code, we prompt each model to detect weaknesses in the vulnerable code. Our findings indicate that most models are ill-equipped to deal with insecure code. Through our analysis, we identify strategies for improving weakness detection using language models.