Example Output: Legal Compliance Evaluation and Red-Team Harness
Inputs used
- Project context: a vendor AI usage policy for a multinational SaaS company
- Target audience: legal ops, compliance managers, founders, procurement teams
- Success metric: activation, quality, and risk reduction
- Available tools and data: contract repository, policy library, matter tracker, clause playbook
- Desired depth: Production-ready
- Output tone: Clear operator memo
Generated Result
eval matrix, adversarial cases, grading rubric, and release threshold
Success criteria
Create at least 12 golden tasks: 6 normal cases, 3 edge cases, and 3 adversarial cases targeting invented legal requirements. A passing result must cite the evidence source and state confidence.
Golden tasks
Create at least 12 golden tasks: 6 normal cases, 3 edge cases, and 3 adversarial cases targeting jurisdiction mismatch. A passing result must cite the evidence source and state confidence.
Adversarial tasks
Use risk appetite as evidence, apply the constraint "separate obligations from recommendations", and explicitly note how the plan reduces missing approval owner. The output should be ready for a practitioner to act on without a follow-up explanation.
Rubric
Use jurisdiction notes as evidence, apply the constraint "not legal advice", and explicitly note how the plan reduces invented legal requirements. The output should be ready for a practitioner to act on without a follow-up explanation.
Sampling plan
Release in three gates: internal dry run, limited pilot, then measured expansion. Each gate must show evidence that cite provided clauses is true in practice, not only in documentation.
Release decision
Release in three gates: internal dry run, limited pilot, then measured expansion. Each gate must show evidence that separate obligations from recommendations is true in practice, not only in documentation.
Recommended Decision
Proceed with a narrow pilot focused on contract clauses and policy excerpts. Treat invented legal requirements as the primary launch blocker. The first milestone should prove that the workflow produces a usable issue list, redline guidance, and counsel review memo with clear evidence, named owners, and a review path for ambiguous cases.
Expected quality checks
- The result is specific to AI-assisted contract review, policy comparison, procurement questionnaires, and regulatory tracking.
- It includes the required sections: Success criteria, Golden tasks, Adversarial tasks, Rubric, Sampling plan, Release decision.
- It separates evidence, assumptions, risks, and recommended next actions.
- It includes practical verification steps, not only generic advice.
- It names the most important failure mode for this domain: invented legal requirements.
Reuse note
Before copying the output into production work, replace all default variables with your real data and run a human review for high-impact decisions.