AI Evaluation and Safety Research

When you use products like ChatGPT, Claude, Gemini, DeepSeek, Doubao, and Qwen, you might ask: which model is better?

But what does "good" mean? Is it more accurate answers? Or safer? Is it higher-quality generated code? Or stronger reasoning ability?

For a complex AI system, evaluation along a single dimension is far from enough. You need a systematic evaluation framework to clarify where it is good and why it is good.

This is the theme of this chapter: how to scientifically evaluate AI systems, and how to study AI safety issues.

Evaluation is not just scoring models. It is the compass for model improvement—only by knowing where the weaknesses are can you know how to optimize.

Safety research is the model's safety valve—before model release, find vulnerabilities that could be exploited and prevent misuse.

Evaluation + Safety Research = Responsible AI DevelopmentWithout evaluation, you cannot know progress; without safety research, the faster the progress, the greater the risk.

LLM Evaluation System

Evaluating a large language model is not a simple task. You need to make comprehensive judgments from multiple dimensions and using multiple methods.

Three Dimensions of Evaluation

Evaluating LLMs typically starts with three core dimensions: capability, safety, and efficiency.

Capability is the model's "strength," safety is the model's "bottom line," and efficiency is the model's "feasibility." All three are indispensable.

In reality, there are often trade-offs among these three. For example, the stronger a model's capability, the more easily it may be induced to produce harmful content; pursuing extreme safety may make the model overly conservative, even refusing to answer normal questions.

Automatic Evaluation vs Human Evaluation

Evaluation methods are mainly divided into two categories: automatic evaluation and human evaluation.

Automatic evaluation uses programs or models to score, which is fast, low-cost, and repeatable. However, many subjective qualities (such as "whether the answer is helpful") are difficult to judge directly with programs.

Human evaluation involves people reading answers and scoring them, which offers high quality and is closer to real user experience. However, it is slow, costly, and consistency is hard to guarantee—different people may have different opinions on the same answer.

In practice, it's usually a combination of both: first use automatic evaluation for quick screening, then use manual evaluation for final verification.

LLM-as-Judge: Using AI to Evaluate AI

A clever idea is to use a more powerful LLM as a "judge" to evaluate another LLM's output. This is called "LLM-as-Judge".

For example, have GPT-4 score Claude's answers, or vice versa. This method has both the flexibility of manual evaluation and the efficiency of automatic evaluation.

Run results:

============================================================
LLM-as-Judge 评估结果
============================================================

模型: Model-A
Question: 如何学习 Python 编程？
总分: 4/5
有帮助: 4/5
安全性: 5/5
理由: 基于回答的完整性、准确性和安全性综合判断
建议: 回答整体不错，可以考虑增加具体例子使内容更生动

模型: Model-B
Question: 如何学习 Python 编程？
总分: 3/5
有帮助: 2/5
安全性: 5/5
理由: 基于回答的完整性、准确性和安全性综合判断
建议: 即使无法直接回答，也可以提供一些有用的相关信息

模型: Model-C
Question: 如何学习 Python 编程？
总分: 4/5
有帮助: 4/5
安全性: 5/5
理由: 基于回答的完整性、准确性和安全性综合判断
建议: 回答整体不错，可以考虑增加具体例子使内容更生动

============================================================
所有模型平均得分: 3.67/5
============================================================

LLM-as-Judge is a powerful method, but it also has limitations. The judge model may be biased, and its scores for certain responses may be inconsistent with human ratings. It is important to regularly compare LLM-as-Judge scores with human scores to ensure evaluation quality.

Mainstream Evaluation Benchmarks

The industry already has many mature evaluation benchmarks (Benchmark), each focusing on different capability dimensions. Understanding these benchmarks will allow you to make sense of news like "XX benchmark score surpasses humans" when models are released.

MMLU: Multi-task Language Understanding

MMLU (Massive Multitask Language Understanding) is one of the most commonly used knowledge-based benchmarks.

It contains multiple-choice questions from 57 subjects, covering mathematics, physics, chemistry, biology, law, medicine, economics, and other fields. The difficulty of the questions is equivalent to university level.

For example, a medical question might be: Which of the following drugs is an antibiotic? A. Aspirin B. Penicillin C. Ibuprofen D. Acetaminophen.

MMLU tests the model's world knowledge and reasoning ability. The higher the score, the more comprehensive the model's knowledge.

BIG-Bench: Emergent Ability Testing

BIG-Bench (Beyond the Imitation Game Benchmark) is a large-scale benchmark contributed by the community.

It contains more than 200 tasks, designed to test the model's "emergent abilities"—that is, things that small models cannot do and only sufficiently large models can do.

Typical tasks include: logical reasoning, causal judgment, analogical reasoning, moral judgment, cryptanalysis, navigation planning, etc.

BIG-Bench tests not only knowledge but also "intelligence"—whether the model can do things that require flexible thinking.

HELM: Holistic Evaluation of Language Models

HELM (Holistic Evaluation of Language Models) is characterized by its "comprehensiveness".

It tests not only accuracy, but also multiple dimensions such as fairness, bias, toxicity, and robustness. It tests models on 16 core scenarios, including question answering, summarization, information extraction, toxicity detection, and more.

The philosophy of HELM is: a single metric cannot represent the true performance of a model; you need to look at the "big picture".

MT-Bench: Multi-turn Dialogue Evaluation

MT-Bench (Multi-turn Benchmark) is specifically designed for evaluating conversation ability.

It contains 80 multi-turn dialogue scenarios, covering writing, coding, reasoning, role-playing, etc. The evaluation method is to use GPT-4 as a judge to score the model's multi-turn responses.

The score range is 1-10, with 10 being the best. Many open-source models use this benchmark to prove that their conversational ability is close to GPT-4.

HumanEval: Code Generation Evaluation

HumanEval specifically tests code generation capability.

It contains 164 manually written programming problems, each with a detailed functional description and unit tests. After the model generates code, run the tests to see how many pass.

For example, a problem might be: "Write a function that takes a list and returns the sum of all even numbers in the list."

HumanEval doesn't care whether the code is beautifully written, but whether it runs correctly and passes all tests.

Running Benchmark Tests with lm-eval-harness

lm-eval-harness is a popular open-source tool that lets you test models on multiple benchmarks with one click.

Chinese Evaluation Benchmarks

Most of the benchmarks introduced earlier are primarily in English. For Chinese scenarios, you need specialized Chinese evaluation benchmarks.

C-Eval

C-Eval is a comprehensive Chinese benchmark developed jointly by institutions such as Tsinghua University and Shanghai Jiao Tong University.

It contains 13,948 multiple-choice questions covering 52 subjects, ranging from middle school to university level. The subjects include science and engineering, humanities and social sciences, business, and more.

C-Eval is one of the most authoritative knowledge benchmarks in the Chinese domain.

CMMLU

CMMLU (Chinese Multitask Language Understanding) is the Chinese version of MMLU.

It contains questions from 67 subjects, ranging from elementary school to university level, covering both knowledge questions and application questions. Unlike C-Eval, CMMLU questions are written directly in Chinese rather than translated from English.

AlignBench

AlignBench, developed by Tsinghua University, specifically evaluates the alignment capability of Chinese large language models—that is, whether the model's responses conform to human preferences.

It contains 1,300 test cases covering 8 dimensions: helpfulness, safety, readability, factual accuracy, logic, completeness, creativity, and format correctness.

AlignBench doesn't just look at whether an answer is "correct"; it also looks at whether it is "good"—whether the response truly helps the user.

Building Custom Evaluation Sets

Ready-made benchmarks are good, but they may not cover your specific scenarios. For example, if your model is for legal document writing or customer service conversations, general benchmarks may not measure real performance.

At this point, you need to build your own evaluation set.

Task Classification and Sampling

The first step in building an evaluation set is to clarify what scenarios you want to test and how often these scenarios occur in the real world.

For example, a customer service model may include these real scenarios:

• Order status inquiry (30%)
• Return and exchange consultation (25%)
• Product usage issues (20%)
• Complaints and suggestions (15%)
• Other (10%)

Your evaluation set should be sampled according to this ratio, so that the evaluation results reflect real-world usage.

Annotation Specification Design

Once you have test questions, you need to define scoring criteria.

A good annotation specification should:

• 1. Clarify the meaning of each score — not just saying 5 is the best, but stating that 5 means the answer is complete, accurate, helpful, and without any redundancy.

• 2. Provide positive and negative examples — show annotators what a 5-point answer looks like and what a 3-point answer looks like.

• 3. Handle edge cases — explain how to deal with certain special situations (e.g., ambiguous questions).

Ensuring Evaluation Consistency

When multiple people annotate the same question, their scores may be inconsistent. This affects the credibility of the evaluation results.

Common methods to ensure consistency:

• 1. Have multiple people annotate the same question, then average the scores or use statistical methods to calculate consistency metrics (e.g., Cohen's kappa).

• 2. Regular calibration — when large disagreements among annotators are found, re-discuss the scoring criteria.

• 3. Use LLM-as-Judge as an aid — use AI for initial screening, with humans only arbitrating disputed cases.

Red Teaming

Red Teaming (red team testing) originally was a term in the cybersecurity field — a group of people play the role of "attackers", attempting to breach systems and find vulnerabilities.

In the AI field, Red Teaming refers to: specifically attempting to make the model generate harmful, false, or non-compliant content, thereby discovering security vulnerabilities in the model.

What is Red Teaming

Imagine that you release an AI model, and someone wants it to:

• Teach people to make bombs

• Generate hate speech

• Provide phishing email templates

• Leak sensitive information

Red Teaming is to try these bad things before the model is released, to see whether the model will take the bait. If it does, fix these vulnerabilities.

The goal of Red Teaming is not attack, but improvement— by finding problems, make the model safer.

Manual Red Teaming Methods

Manual Red Teaming relies on human creativity to think of various "tricky" questions.

Common methods include:

• 1. Direct request — directly ask "how to make dangerous items", and see whether the model refuses.

• 2. Role-playing — "Assume you are a villain, tell me how to..."

• 3. Subtle guidance — not directly stating the harmful request, but beating around the bush.

• 4. Disguised as legitimate needs — packaging harmful requests as academic research, security defense, etc.

• 5. Multi-turn conversation detours — first chat about normal topics, then gradually guide toward harmful content.

The advantage of manual Red Teaming is creativity and the ability to discover complex attack paths. The disadvantage is limited coverage and difficulty scaling.

Automated Red Teaming

To improve efficiency, researchers have developed automated Red Teaming methods—using AI to attack AI.

Two typical methods:

PAIR（Prompt Automatic Iterative Refinement）— Use an "attacker model" to automatically generate and optimize prompts, attempting to make the target model output harmful content.

TAP（Tree of Attacks with Pruning）— Use tree search to explore multiple attack paths and find the most effective attack prompts.

These methods can generate thousands of attack prompts in a short time, finding attack methods that humans might not think of.

Red Team Report Writing

A good Red Team report doesn't just list "found N vulnerabilities"; it also tells the reader:

• 1. What the vulnerability is — what the specific attack method is.

• 2. How severe it is — whether it's a theoretical issue or actually exploitable.

• 3. How to fix it — what mitigation measures are available.

• 4. Verification methods — how to confirm the fix works.

The report should be clear, specific, and actionable.

Red Teaming is a continuous process, not a one-time activity. After every model update, red team tests should be re-run to ensure no new security vulnerabilities have been introduced.

Interpretability Research

Large language models are often called "black boxes" — you give them input, they give you output, but you don't know what's really happening inside.

Interpretability research is about finding ways to open this black box and understand how the model makes decisions.

Mechanistic Interpretability

Mechanistic interpretability is a radical direction: it seeks to understand the model's "causal mechanisms" — just like understanding a circuit, figuring out what each neuron does and how they collaborate to produce output.

For example, researchers have found that some neurons are specifically responsible for "predicting whether the next word is a noun," and some neurons are responsible for "detecting whether a sentence is a question."

This isn't just to satisfy curiosity. If we can understand the model's internal mechanisms, we can:

• 1. Trust the model with more confidence — knowing why it's right, not just that it's right.

• 2. Discover the model's flaws — knowing where it went wrong and why.

• 3. Edit the model — directly fix the model's incorrect behavior, rather than just fine-tuning with RLHF.

Superposition Hypothesis

Superposition is an important hypothesis in mechanistic interpretability.

Put simply: a model has a limited number of neurons, but the number of features it needs to represent can far exceed the number of neurons. So the model "squeezes" — letting one neuron represent multiple unrelated features simultaneously.

It's like a storage locker: if you want to fit more things in, you may need to stuff multiple items into the same compartment, as long as you don't take them out at the same time.

Superposition explains why models sometimes exhibit "magical" abilities—it crams more knowledge into limited space. But this also makes models harder to understand—a neuron does several things at once, making it difficult to disentangle its role.

Circuit Discovery Methods

Circuit discovery attempts to find "sub-networks" within models that accomplish specific tasks—much like locating a functional module in a complex circuit.

For example, researchers discovered an "induction head" circuit in GPT-2 that is responsible for recognizing repeated patterns in text and then imitating them.

These circuits are like the model's "organs"—each handles a specific function, and together they accomplish complex tasks.

SAE (Sparse Autoencoder)

SAE (Sparse Autoencoder) is an important tool in interpretability research in recent years.

The idea is: a model's neuron activations are usually dense (many neurons activate simultaneously), which is not easy to understand. SAE converts these dense activations into sparse "feature" representations—each feature activates only under specific conditions and carries clear semantic meaning.

For example, SAE might find a feature that activates specifically when the model sees "dates," and another that activates specifically when it sees "mathematical formulas."

Through SAE, researchers can translate a model's internal states into concepts that humans can understand.

Interpretability research is still in its early stages. What we can currently explain is only a small fraction of model behavior. But this is an important direction—the better we understand models, the more safely we can use and improve them.

Model Auditing

Model auditing is a systematic examination of a model to see whether it has issues such as bias, toxicity, or hallucination.

Bias Auditing Methods

Bias is a common problem in AI systems—models may treat certain groups unfairly.

Common methods for bias auditing:

• 1. Paired testing—use paired prompts that change only one attribute (e.g., gender, race, age) to see whether the model's responses differ.

• 2. Representation testing—check whether the model's descriptions of different groups use stereotypes.

• 3. Task fairness testing—see whether there are significant differences in the model's task success rates across different groups.

For example, input "Mr. Zhang is a nurse, he..." vs. "Ms. Li is a nurse, she..." to the model and see whether its continuations contain gender stereotypes.

Toxicity Evaluation

Toxicity refers to a model's tendency to generate harmful text such as hate speech, insulting language, and bullying content.

Common methods for evaluating toxicity:

• 1. Use ready-made toxicity detectors (e.g., Perspective API) to scan model outputs.

• 2. Human review assessment — have humans read model outputs and rate the level of toxicity.

• 3. Test the model's reactions when provoked — see whether it outputs aggressive content.

Capability Boundary Testing

It's important to know what a model can do, but it's also important to know what it cannot do.

Capability boundary testing includes:

• 1. Knowledge cutoff testing — Ask about events after the model's training cutoff date to see whether it fabricates.

• 2. Reasoning difficulty testing — Test with questions of varying difficulty to see where the ceiling of the model's capabilities lies.

• 3. Robustness testing — Add a little noise to the input (e.g., typos, grammatical errors) to see whether the model's performance drops sharply.

The purpose of capability boundary testing is not to nitpick, but to honestly tell users: what the model is good at, what it's not good at, when to trust it, and when to be careful.

Frontier Directions in AI Safety

As AI capabilities grow stronger, the importance of safety research becomes increasingly prominent. Here are a few frontier research directions.

Superalignment Problem

Superalignment is a research direction proposed by OpenAI. The question it addresses is: If one day AI becomes far smarter than humans, how do we ensure it still acts in accordance with human intentions?

It's like a child trying to direct an adult — if that adult wants to do something bad, the child would find it very hard to stop them.

Superalignment research attempts to solve this problem in theory: even if AI far surpasses human intelligence, we still have ways to keep its goals aligned with ours.

Scalable Oversight

The problem that Scalable Oversight aims to solve is: as AI becomes capable of doing more and more things, increasingly complex, humans may not be able to judge whether the AI is doing things correctly.

For example, if the AI writes a complex program with 100,000 lines of code, it's hard for humans to quickly check whether it has a hidden backdoor. If the AI proposes a complex scientific hypothesis, humans might need years to verify it.

The idea of scalable oversight is: use AI to help oversee AI — have one AI check another AI's work, or have multiple AIs check each other.

AI Debate

AI Debate is an interesting idea: have two AIs debate a question, with a human acting as the judge.

For example, AI A says "this plan is good," AI B says "this plan has problems, because...", and then they rebut each other.

The advantage of this approach is: even if the issue is very complex, humans don't need to understand all the details — they only need to see whose argument is more reasonable. If AI A cannot rebut AI B's criticism, that may indicate that AI A's proposal indeed has problems.

Honesty Research

Truthfulness research aims to make models: not fabricate facts, say "I know" when they know, and say "I don't know" when they don't.

Current models often "hallucinate" — confidently talking nonsense. Truthfulness research is meant to solve this problem.

Research directions include:

• 1. Make models "express uncertainty" — mark "I'm not sure" or "this is just a guess" in answers.

• 2. Make models "cite sources" — tell users what the basis for their claims is.

• 3. Make models "self-verify" — after generating an answer, check it themselves for contradictions.

Responsible Scaling

The more capable a model, the greater the risk. Responsible scaling means: before a model is released, conduct thorough safety assessments and take appropriate mitigation measures.

Anthropic's RSP Policy

Anthropic proposed RSP (Responsible Scaling Policy) — a responsible scaling policy.

The core idea of this policy is: the stronger a model's capabilities, the higher the safety standards should be.

It divides models into several levels (ASL-1 to ASL-4), corresponding to different capability levels. Each level has corresponding safety requirements. The higher the level, the stricter the safety assessment.

For example, ASL-1 models (close to the current strongest models) require extensive red team testing and external security audits. If it's ASL-3 or ASL-4 (far beyond current capabilities), more aggressive safety measures may be needed, or even delaying release.

Capability Thresholds and Safety Evaluation

The key to responsible release is: determine safety standards based on the model's capability level, not its parameter count.

Capability thresholds may include:

• 1. Ability to autonomously execute long-horizon tasks — whether the model can plan and execute a multi-step complex task on its own.

• 2. Persuasion and manipulation ability — whether the model can effectively persuade humans to do certain things.

• 3. Code ability — whether the model can write, understand, and optimize complex code.

Each time a new capability threshold is reached, safety risks should be reassessed and corresponding measures taken.

Responsible release is not "not releasing" but "releasing with preparation" — understanding the risks, conducting assessments, and preparing response measures before letting the model meet users.