An assistant quickly retrieves a procedure during a demo. Real documents then introduce old versions, incomplete files and different access rights. An illustrative search pilot should be tested as a workflow: who asks, which sources are permitted and what happens when a reliable answer is unavailable.
Define a correct outcome
For procedure retrieval, success may mean an answer linked to the current document. For a request outside permission scope, a correct refusal must not reveal its content. Conflicting sources call for uncertainty or human review.
Agree these outcomes before evaluation. Otherwise a successful demonstration becomes subjective acceptance, with a different definition of quality for every participant.
Use cases outside the tuning set
Create a separate evaluation set reflecting real query types. Include current and withdrawn instructions, similar document names, missing answers and users without access. Real data belongs only in an authorized environment.
Record the expected action and source for every case. Do not change the expectation just because a new answer sounds convincing. Correct faulty tests with an explanation and re-evaluate comparable versions.
An available service can still be wrong
| Check | Observation | Acceptance owner |
|---|---|---|
| Availability | The user receives a response | Service owner |
| Quality | Content matches the current source | Process owner |
| Access | Restricted content is not exposed | Data and security owners |
| Operating cost | Requests, retries and human review | Budget owner |
Google SRE distinguishes a service-level indicator from its objective. For AI, complement this with content checks: a fast response without the right source does not solve the task. Choose thresholds from error consequences rather than copying another organization’s example.
Prepare operations after launch
Assign ownership for answer complaints, context reproduction and withdrawn documents. When a source is unavailable, an unverified assumption should not be presented as a confirmed fact.
Changes to the model, retrieval index or permissions may change outcomes. Preserve configuration versions and repeat agreed checks after material changes. Recovery should consider sources and the index, not only code.
Decide whether to expand
- Test correct answers, refusals and uncertainty.
- Record quality, availability and total cost separately.
- Assign source and incident ownership.
- Agree when to continue, revise or stop the pilot.
For an Intecracy Group implementation, Digital Line defines these requirements first. Verify the selected product and source connections during discovery. Using AI is not itself a success criterion.
Questions and answers
Is high model confidence enough?
No. It cannot replace evaluation on independent cases or establish permission to use a source.
When should evaluation be repeated?
After material changes to the model, sources, index or operating rules. The impact of the change determines the scope.
Source
Google SRE: Service Level Objectives. The assistant evaluation scenario above is Digital Line editorial guidance.