The strongest CiCi evaluations begin with a decision, not a long list of things that might be interesting to measure.
Start with the decision
Write down:
- The workflow you may change
- The user behavior you want to support
- The decision the evaluation will inform
- The evidence you would need before testing, shipping, or communicating the change
Define observable signals
Signals should describe what the evaluation can distinguish. Good signals are:
- Specific to one behavior or judgment
- Applicable across both variants
- Interpretable under every scenario where they are used
- Paired with a clear rule for
observed, not_observed, or unclear
unclear means the available response does not settle the signal under the applicable rule. It is different from having no rule or no measurement.
Select scenarios
Include scenarios that cover the decisions and failure modes the product actually encounters. Avoid changing several dimensions at once unless that combination is itself the product condition you need to study.
Define variants
Variants should differ in ways you can name and implement. Record the prompt, context, interface, workflow, guardrail, retrieval, or model configuration change represented by each arm.
Decide whether human validation is needed
Use human validation when the decision depends on observed comprehension, trust, action, escalation, or another response that simulation alone cannot establish.
You can still use APM-only evidence to prepare a candidate change for testing. CiCi will label it accordingly and prevent it from being presented as observed human evidence. 
