Researchers tested popular open-source models against a benchmark for detecting IDOR vulnerabilities. GLM 5.2, from Zhipu AI, outperformed Claude Code. The test used the same dataset and prompt as previous evaluations of frontier coding agents.
The evaluation aimed to assess the models' ability to identify Insecure Direct Object Reference vulnerabilities. GLM 5.2 achieved a 39% F1 score, surpassing Claude Code's 32%. The cost per vulnerability found was approximately $0.17.
GLM 5.2's performance was notable, given its open-weight architecture. It still trailed behind Semgrep's proprietary model. The results suggest that open-source models are closing the gap with their proprietary counterparts.
The test's findings have significant implications for the development of vulnerability detection tools. GLM 5.2's success demonstrates the potential of open-source models in this area.
The difference in performance between GLM 5.2 and Claude Code can be attributed to various factors, including the models' architectures and training data.
As the cybersecurity landscape continues to evolve, the performance of models like GLM 5.2 will be crucial in determining the effectiveness of vulnerability detection tools.
What is IDOR vulnerability detection? IDOR detection involves identifying Insecure Direct Object References, a type of security vulnerability. GLM 5.2 demonstrated strong performance in this area.
How does GLM 5.2 compare to other models? GLM 5.2 outperformed Claude Code but trailed Semgrep's proprietary model. Its performance highlights the potential of open-source models.
What are the implications of these findings? The results suggest that open-source models are becoming increasingly viable alternatives to proprietary ones, potentially changing the cybersecurity landscape.