Evaluations & Red Teaming

Definition

Evaluations measure a system against stated criteria. Red teaming deliberately probes ways it may fail or be misused, complementing ordinary performance checks.

An evaluation needs a defined target, decision purpose and interpretation of results. Red teaming adds deliberate attempts to find failures. A successful exercise improves knowledge of observed behavior; it does not establish safety in every future context.

What it covers

Target

The model, agent, tools and complete workflow.

Cases

Expected use, edge cases and adversarial attempts.

Evidence

Results tied to versions, conditions and limitations.

Choose units that match the workflow: answer accuracy, prohibited tool calls, approval compliance, tenant separation, recovery and user impact. Include ordinary failures as well as adversarial cases.

In practice

Practical design patterns by Min.

• Define success and unacceptable behavior before testing.
• Include multi-step actions and permission boundaries.
• Re-run relevant cases after changing models, tools or policy.

Retain versioned cases, expected outcomes and scoring rules. Record the model, tools, data and policy configuration that produced the result.

Use findings to make a release decision, assign remediation and rerun relevant cases after changes. Keep remaining gaps visible rather than compressing them into one reassuring average.

Working example

Why the controls matter

A support agent can be tested on legitimate cases, ambiguous instructions and malicious retrieved pages. Separate a wrong answer from a blocked unauthorized action and a successfully committed harmful action.

Common pitfalls

• Testing only answer quality.
• Treating a passing sample as proof of universal safety.

Testing only the model omits integration behavior. Include the services that validate, approve and execute its proposals.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• What decision will this evidence inform?
• Which failure paths are untested?
• What changes invalidate previous results?