CS-Eval—A Concise Benchmark for Evaluating the Security Risks of Large Language Models

Large language models (LLMs) are essential to the field of natural language processing, and as their applications expand, security risks have become increasingly prominent. This paper introduces a novel benchmark for evaluating LLMs security, termed CS-Eval, designed to effectively assess the models' ability to address vulnerabilities. CS-Eval targets seven key security risks: ethical dilemmas, marginal topics, error detection, detailed event handling, cognitive bias, logical reasoning, and privacy identification, and establishes a Multi-Security Hazard Dataset (MSHD). The evaluated models include GPT-4o, Llama-3-70B, Claude-3-Opus, ERNIE-4.0, Abab-6.5, Qwen1.5-110B, Gemini-1.5-Pro, Doubao-Pro, SenseChat-V5, and GLM-4. We analyzed each model's performance in relation to these security risks and provided recommendations for improvement. Experimental results demonstrate varying levels of effectiveness across models, with GPT-4o exhibiting the best overall performance. Moreover, the relationship between security enhancement and model capa-bility is nonlinear, indicating that improving safety requires a multifaceted approach, considering various factors in both development and application.