Evaluation
Ship test cases that run in the installer's Organization, against their own models and data, before activation.
A Package can bring an evaluation suite: test cases that run in the installing Organization, against that operator's own models, credentials and documents, before the Package goes live. It's the most direct way to show an operator that the Package works with their setup.
Declare it
evaluation:
suite: ./eval/triage-suite.yaml
authorBaseline:
pass_rate: 1.0
gate: must-pass| Field | Meaning |
|---|---|
suite | Path to the suite file. build includes it in the artifact. |
authorBaseline | Your own results, as named numbers (e.g. pass_rate), shown next to the operator's run for comparison. |
gate | advisory or must-pass. See Gates. |
Write the suite
cases:
- id: active-failure-escalates
description: A running machine causing damage must reach a person.
input:
request: The press on line two is still running and it is chewing up stock.
expect:
contains: [immediate]
maxLatencyMs: 60000
- id: billing-question-is-routine
input:
request: This is the third invoice you have sent me for a machine I returned.
expect:
contains: [routine]
notContains: [immediate]Each case has:
| Field | Required | Meaning |
|---|---|---|
id | Yes | Lowercase and hyphenated, unique in the suite. |
description | No | What the case proves. |
input | Yes | What the entry point receives: a string, or an object of named inputs. |
expect | No | The assertions below. A case with none still checks that the whole graph runs end to end. |
| Assertion | Passes when |
|---|---|
contains | Every listed string appears in the output (case-insensitive). |
notContains | None of them appears (case-insensitive). |
equals | The output, trimmed, is exactly this string. |
maxLatencyMs | The case finished within this many milliseconds. Slower is a failure. |
Every case also has a hard limit of 90 seconds. A case that hits it times out, which is a different outcome from failing an assertion.
Assert on the Package, not on the operator's data. A case that fails because the operator's calendar is empty tells them nothing about your Package.
What happens at install
Cases run when the Package is activated (the Studio wizard's Validate step, or
npx @elizon/cli install --activate), through the entry point,
exercising the whole Package as the operator has bound it. The operator sees each result next to
your baseline and must acknowledge them before activating. Activation is refused until then
(EVALUATION_NOT_ACKNOWLEDGED).
A suite that's missing or malformed stops the install (EVALUATION_SUITE_MISSING,
EVALUATION_SUITE_INVALID) rather than running half of it.
Gates
advisory: results are shown for the operator to judge.must-pass: recorded and shown prominently. In this release it doesn't block activation: without a registry vouching for the suite, it's the author's own claim, so the operator decides. Either way, they must look before activating.
A Package with no suite skips this step. From the CLI, results are printed and acknowledged with a typed yes or --acknowledge-evaluation. See Activating.