Skip to main content

Notebook

The workspace for one evaluation program. It connects the intended behavior, study design, runs, evidence, and outputs.

Sprint

A versioned execution of an evaluation. Its scenarios, variants, mappings, and evidence rules are frozen at launch so the result remains interpretable later.

Signal

A defined behavior or response pattern that the evaluation can observe or score. Signals should be specific enough that two reviewers can apply the same rule.

Scenario

A situation presented consistently across variants and evaluation arms. Scenarios serve as the shared comparison blocks in a study.

Variant

One product experience, prompt, workflow, or configuration being compared. Variant keys identify the comparison arms.

APM

An AI-based participant model used to simulate responses to the frozen study. APM evidence is labeled modeled because it is not direct observation of people.

Human validation

A study arm that collects and reviews participant responses. Human evidence becomes authoritative only after the required independent review and adjudication steps complete.

Signal Gap

The difference between the human contrast and the APM contrast across the same scenarios. It shows where modeled behavior and observed human behavior align or diverge.

Sufficiency gate

One rule used to decide whether the evidence can support a claim. Some gates block completion or suppress a claim; others qualify how the result should be interpreted.

Refinement brief

A frozen, versioned statement about what the evidence supports, what remains uncertain, and what changes the team can test or implement.