AI quality testing: illustrative scenario
An illustrative quality workflow: test source support, citation accuracy, and regressions before releasing an AI research feature.
Illustrative scenario, not client results. This describes a possible approach, not a completed engagement. No measured outcomes or delivery timelines are claimed.
The situation
A team wants to offer customers an AI research feature. Fluent answers are easy to demonstrate. Knowing whether the answer is supported by the underlying documents is harder, especially when models or source material change.
A possible first pilot
Choose one narrow research task and collect approved questions, source documents, and expert-reviewed answers. Include examples where the correct behaviour is to say that the source does not answer the question.
A test harness would check whether cited documents exist, whether citations support the claims made, and whether an update introduces regressions. Automated checks would help experts find problems; model-generated grades would need calibration against human review.
What to measure
Establish the current error rate on the agreed sample. Track unsupported claims, incorrect citations, missed relevant information, and the time experts spend reviewing results. Keep the sample and scoring method visible so a score cannot be mistaken for a guarantee across all future questions.
These checks can help a team decide whether a useful new feature is ready for a limited trial. They do not replace professional judgment or prove that a system is error-free.
The expansion decision
Expand the test set and release boundary only when the results justify it. A first pilot would not certify a whole legal research product or promise a particular reduction in hallucinations.