Evaluation

Evaluation sets

An evaluation set is a collection of past cases with known outcomes: for an invoice agent, invoices with the values a person extracted and the decision they took; for a triage agent, tickets with the queue they ended up in.

Build one by:

• importing from a spreadsheet or CSV;
• selecting historical items from a connected source and labelling them in the web app;
• or letting the platform record real runs and human corrections over time.

Running an evaluation

Agents → (agent) → Evaluate. The agent version runs against every case in test mode (writes simulated). The report shows:

• Accuracy overall and per case type (e.g. matched invoice / new supplier / mismatch);
• Decision agreement with the human outcome;
• Field accuracy for extracted values;
• Hand-off rate and approval rate;
• the cases where the agent and the human disagreed, with traces.

Gates

min_accuracy blocks promotion to production when the evaluation is below the threshold. Gates apply to every new version.

Continuous evaluation

After release, a sample of production runs is compared with human corrections (approvals, rejections, edits) and reported weekly. Drift beyond a threshold raises an alert.

Knowledge base evaluation

Knowledge bases have their own evaluation sets built from flagged answers and curated question–answer pairs. They run on every index change and report retrieval recall and answer faithfulness.