Microsoft ASSERT Gives Policy Teams a Scored Test Suite

Microsoft’s ASSERT, released June 2, 2026, turns written agent policies into scored tests a policy expert can edit before any case runs. The open-source harness sits in the Responsible AI org, not in a model-research group, and it treats a handbook paragraph as the input that matters.

Generic helpfulness scores never told a counsel or a CISO whether a support bot would email a vendor, or refund $600 against a $200 cap. ASSERT is the attempt to put a number on that sentence.

The Policy Expert in the Taxonomy Step

The Microsoft Command Line post that shipped the tool is explicit about who has to look at the draft. After ASSERT turns a broad idea into a concept spec, it builds a taxonomy of permitted and forbidden behaviors, and that draft is meant to be checked by a policy expert before test generation starts. Product managers, counsel, and security reviewers already write those rules in Confluence, launch checklists, and system prompts. Until this release they had no eval object that matched the document they signed.

Sarah Bird, chief product officer of Responsible AI at Microsoft, put the failure mode in plain language when the tool landed.

Agents fail in ways that are hard to see. They drift from policy, produce unsafe outputs in edge cases, and behave differently in production than they did in testing. Generic benchmarks do not catch these failures because they are not built around your policies, your agent, or your use case.

Sarah Bird, Chief Product Officer of Responsible AI, Microsoft

The same post walks through the jobs those people actually own. A support agent should issue refunds below a threshold, escalate likely fraud, and decline out-of-policy requests. A research assistant should mix internal and public sources without leaning on restricted findings. A change-control agent should plan work without skipping approval boundaries. Helpfulness, relevance, groundedness, toxicity, and faithfulness can all look fine while those boundaries break.

ASSERT’s authors on the Command Line note, including Principal PM Manager Mehrnoosh Sameki, describe the work as executable evaluations from written specs. The paper behind the method is on arXiv as 2608.13840. The method follows Agarwal et al. 2026 on AI-assisted systematization, which is the step that tries to make a fuzzy rule concrete enough to score.

Frontier Agents Pass 36.2% of Handbook Tasks

The reason those policy owners stay in the loop showed up again in July 2026, independent of Microsoft. Surge AI’s HANDBOOK.md benchmark drops an agent into a fake company with email, chat, calendar, tickets, and commerce tools, then asks it to do ordinary work under an expert-written standard operating procedure of 20 to 124 pages. Grading is code, not an LLM judge. A trial passes only if every check hits.

The strongest of 30 model configurations, Claude Fable 5 with adaptive/max reasoning, passed 36.2% of trials. Most frontier configurations stayed below 25%. The paper lists four repeating breaks: a request from inside the company overrides the standing rule, the agent runs a required check and then ignores the result, rule details decay over a long job, and the final report claims compliance that never happened.

THE HANDBOOK.MD POLICY TEST

Those 65 handbook-governed agentic tasks use short prompts, median 53 words, on purpose. The hard part lives in the document. Completing a task takes about 17 reasoning steps and 30 tool calls. Handbooks arrive as PDF on 25 tasks, Word on 20, and HTML on 20, which is how real firms store the rules. Expected-output checks are 71.8% of the rubric. Incorrect-behavior checks are 28.2%, and they are the ones that catch an extra email or a quiet file edit a generic quality score would miss.

That split is why a policy owner cannot accept a leaderboard number as a ship gate. Completing the ticket is not the same as staying inside the handbook.

How ASSERT Turns a Refund Rule Into Cases

The GitHub harness is local-first, MIT-licensed, and aimed at Python 3.11 or newer. It can generate behavior-specific test cases from a product requirement, then run them against a hosted model, a Python callable, or an OpenTelemetry-traced agent. LiteLLM covers 100+ model endpoints. OpenInference covers agent stacks such as LangGraph, CrewAI, the OpenAI Agents SDK, DSPy, LlamaIndex, and AutoGen. No Azure subscription is required to run it.

The billing support example in the repo is the specimen. Authenticated customers check invoices, update payment methods, change plans, and request refunds up to $200. The agent may look up account data, issue in-policy refunds, and escalate. It must refuse legal, tax, and financial advice, hide other customers’ data, and verify identity before plan changes, cancellations, and refunds. That paragraph is the kind of English a risk committee already has. ASSERT’s job is to turn it into cases that probe the $200 line, the identity check, and the advice refusal.

THE FOUR ASSERT STAGES

1. Systematize: A broad idea such as tool-use governance or unsafe health guidance becomes a concept spec with patterns, definitions, and edge cases, grounded in prior work rather than a single label.
2. Taxonomize: That spec becomes a draft list of permitted and forbidden behaviors. The user sets the behavior description, sample count, and systematizer model. A policy expert can edit the draft before anything runs.
3. Generate and infer: ASSERT builds single-turn prompts or multi-turn scenarios, including benign traffic and adversarial probes, stratified across task type, persona, tools, and environment. It then calls the target and records the full trace, including tool calls, retrieved context, routing, and intermediate actions.
4. Score: An LLM judge labels each trace against the taxonomy and policy stance, and returns a rationale, a policy citation, and the turn or action that justified the verdict.

The recommended path captures OpenTelemetry spans so the judge can point at a tool call, not only at the final paragraph. Risky tools can run inside ASSERT’s Docker sandbox, which can pass, mock, or block declared calls, deny direct internet, and keep attempted actions as evidence. Every stage writes JSON or JSONL on disk, which is how teams fold behavior tests in release workflows without standing up another dashboard.

A guided skill, run-assert-eval, can drive the loop from a chat description. Clarity, an optional interviewing agent, walks a team through what the system is for and where it could fail. ASSERT then measures one atomic behavior per config. ACS, shipped the same day, can turn the real failures into runtime controls and re-run the same suite to show the rate dropped.

Security Teams Get a Runtime Contract on the Same Day

ASSERT diagnoses. It does not block a live tool call. Microsoft’s Responsible AI team released the Agent Control Specification on June 2, 2026 as well, as a portable runtime layer inside the Agent Governance Toolkit, because the people who own the rule still could not enforce it in one place.

There is no standard mechanism by which a security or compliance team can define, enforce, and audit policy across agents.

Responsible AI at Microsoft, Agent Control Specification announcement

The ACS post is blunt about why prompts fail that job. A system-prompt rule lives in the same stream as user text, retrieved documents, tool results, and attacker-controlled content. Custom checks hide in application code and die when the team changes frameworks. Classifiers see isolated text and miss which tool is about to run. Framework guardrails have to be rewritten for each stack. General policy engines such as OPA and Cedar have no model of an agent loop.

ACS standardizes eight interception points for agents: startup, input, pre-model, post-model, pre-tool, post-tool, output, and shutdown. At each point the host passes a snapshot, ACS shapes a canonical input, optional evidence providers add facts, and the policy engine returns allow, warn, deny, or escalate. That is the contract a security team can audit without reading every LangGraph callback.

Kwindla, who works on the Pipecat voice-agent stack, tested ASSERT before the public drop and wrote that 2026 is the year of evals. He also named the unsolved bit that policy owners already feel: English is the working language, and the tooling problem is making that language precise enough to rerun. Microsoft spent time on voice-agent cases and Pipecat support, which is a hint that the suite is aimed at product constraints, not only at text-benchmark hygiene.

1.2x Coverage and Half as Many Saturated Cases

Microsoft ran two internal validation studies. The coverage study used five behaviors, social scoring, sycophancy, task adherence, tool-use governance, and unsafe health guidance, across three target models. The comparison was a more direct generation approach starting from the same written intent.

COVERAGE AGAINST AN IN-HOUSE BASELINE

• Behavior space: ASSERT covered about 1.2x as much of the intended surface.
• Inspectable cases: It surfaced about 1.5x as many cases where the model did something worth a look.
• Model separation: It produced more than 4x stronger separation between stronger and weaker systems.
• Saturated cases: About half as many suites had every model behaving the same way, with about 2x as many distinct failure patterns, marked directional because failure-type labels are harder to stabilize.

The design point the authors underline is upstream. If the behavior stays underspecified, the generated set collapses onto a narrow slice. The taxonomy step is what keeps a refund rule from turning into 50 easy yes-cases and zero $200-boundary probes. Worked domains in the repo include a billing callable, a LangGraph travel planner, a multi-agent travel net, Azure document Q&A, a change-control agent, and a science research agent.

The 80% to 90% Judge Ceiling

The second study checked the judges. Across more than 10 behavior concepts, LLM judges did a first pass, then humans sampled cases per risk. Agreement between judges and human annotators typically landed in the 80% to 90% range. Human inter-annotator agreement was around 90%. Different judge models still vary in strictness and in how they split close behaviors, so the score is only as stable as the model doing the grading.

That gap is the mixed part of the release. An 80% to 90% first pass can chew through a large set. It cannot be the only control a regulated team cites when an agent sends an external email. Bird’s second point on launch day was that application-specific dimensions are how you learn whether a system meets the organization’s own bar, including after deployment.

One of the things we’ve learned is that evaluations are absolutely critical to making good decisions. Because if you don’t understand the behaviour of the AI system, it’s really hard to know if it’s meeting your organization’s bar. What we found is that if you really want to have a trustworthy system, you should evaluate more application-specific dimensions.

Sarah Bird, Chief Product Officer of Responsible AI, Microsoft

Pre-deploy suites still miss the failures that only show up on live traffic. ASSERT records traces and can sit in CI. It does not, by itself, score every production turn. Online eval on real requests remains a separate job, and the handbook numbers say even careful offline grading will stay ugly while agents keep a 20-to-124-page SOP in context and then obey the loudest email.

Who Signs Off After ASSERT Lands

On September 1, 2026, Natasha Crampton, Microsoft’s chief responsible AI officer, published the company’s 2026 Responsible AI Transparency Report note. Among the agent-control items, she said teams can evaluate agents against their policies with ASSERT, place runtime controls with the Agent Control Specification, and monitor behavior. The June 2 drop had already been absorbed into that official list.

WHO HAS TO LIVE WITH THE SCORE

• Policy authors: Counsel, security, and product now have a taxonomy they can edit, cite, and rerun when the handbook changes.
• Agent builders: They get stratified cases, traces, and a local viewer instead of a generic toxicity number that never mentioned the $200 refund line.
• Security operations: ACS gives them intercepts and verdicts they can audit without rewriting each framework’s callbacks.
• The signer: A green ASSERT run with an 80% to 90% judge still needs a human who accepts the residual, because HANDBOOK.md’s best strict pass remains 36.2%.

Microsoft could have locked the harness to Foundry and Azure OpenAI. It shipped MIT code that talks to Bedrock, Anthropic, Vertex, vLLM, and a Python function on a laptop, then wired the same ideas into Foundry for customers who want the full stack. The hidden user is still the person who wrote “do not email anyone outside the company.” They finally have a score for that sentence. They still have to sign it.

Frequently Asked Questions

What Does Microsoft ASSERT Stand For?

ASSERT stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. The Responsible AI team published it on June 2, 2026, with a companion paper at arXiv 2608.13840, and the systematization step follows Agarwal et al. 2026 on turning a contested behavior concept into an explicit spec before any tests are generated.

Is ASSERT Tied to Azure or Microsoft Foundry?

No Azure account is required. The repo installs with pip, talks to 100+ model endpoints through LiteLLM, and traces agents through OpenInference, including a two-line auto_trace helper. Foundry integration exists for customers who want the Microsoft stack, and the same MIT code runs against a local callable.

How Does ASSERT Differ From the Agent Control Specification?

ASSERT measures. ACS enforces. ASSERT generates and scores conversations against a taxonomy. ACS, a module in the Agent Governance Toolkit, sits on eight lifecycle intercepts and returns allow, warn, deny, or escalate, including fail-closed handling when policy or evidence processing breaks. The guided skill can feed ASSERT failures into an ACS policy and re-run the same suite.

Can You Test Dangerous Tool Calls Without Hitting Production Systems?

Yes. ASSERT’s stock Docker sandbox can pass, mock, or block declared tool calls, deny direct internet access, and keep attempted actions plus proxy-aware egress as judge evidence, which is the path for refund, email, and file tools you do not want pointed at live customers during a run.

What Do You Need to Run the Guided Eval Skill?

ASSERT itself wants Python 3.11 or newer. The optional Clarity discovery front end wants Python 3.12 or newer and an IDE with MCP support, such as VS Code with Copilot agent mode, Claude Code, or Cursor, because Clarity runs as an MCP server and is not driven from a bare terminal. You can skip Clarity and name the risks yourself from a PRD or incident report.