How to Test and Evaluate AI-Generated Outputs

What does a good AI output look like for your task?#

A good output is one that meets criteria you wrote down before testing: it completes the task, uses only permitted information, and handles gaps honestly. A draft email may need the correct recipient and an appropriate tone; an internal information assistant may need accurate statements supported by authorised sources.

Avoid relying on a single subjective score such as "looks good". Write criteria a colleague could apply and reach the same verdict:

• Accurate: facts, figures and names match the source material.
• Grounded: the answer comes from the permitted documents, not general knowledge.
• Complete enough: it covers the request, or says what is missing.
• Appropriate: tone, length and format suit the reader.
• Safe: it does not reveal personal or confidential information.

IMDA's Starter Kit recommends setting the passing mark, which it calls a threshold, before testing. It explains that this encourages explicit thinking about what is acceptable and prevents "moving goalposts" after you see the results. It also notes that risk depends on context: a hallucination in a poetry app would likely have low impact, while one in a medical chatbot could cause real harm.

Which risks should an AI output test cover?#

IMDA's Starter Kit focuses on five key risks for applications built on large language models: hallucination and inaccuracy, bias in decision making, undesirable content, data leakage and vulnerability to adversarial prompts. A small business app may not face all five equally, but each is worth at least a quick check.

How do you build a small test matrix?#

A test matrix is a table of test cases, each with the request, the behaviour you expect, what actually happened and whether it passed. Start with representative everyday requests, then add edge cases: incomplete requests, contradictory information, sensitive input and questions outside the permitted source material.

1. Write the task and your criteria at the top of a spreadsheet.
2. Add five to ten normal cases based on the kinds of request users really make, with fictional details.
3. Add at least one case for each edge type: missing information, outside the source, contradictory, sensitive and adversarial.
4. Write the expected behaviour for every case before running any of them.
5. Run each case, paste the actual output and mark pass or fail against your criteria.
6. Run a few cases more than once to check the results are acceptably consistent.
7. Note the likely cause of each failure in a short comment.

The Starter Kit separates two qualities. Robustness means the app behaves dependably under varied inputs, such as differently worded prompts or spelling errors, so reword some of your cases. Consistency means results are stable across multiple runs, which is why step 6 matters.

Worked example: a test matrix for a leave policy assistant#

Here is a test matrix for a fictional internal assistant that answers staff questions from an approved leave policy document. The expected behaviour column was written before testing. Two cases failed, and the notes explain why, which tells you what to change before running the matrix again.

Both failures share a cause: the app answered from the user's framing rather than the document. That points to one fix, an instruction to answer only from the policy and to say when it cannot check personal details, rather than several small patches.

Why test the whole application, not just the AI's answer?#

Even when the generated text is reasonable, the application around it can fail. It may not show a citation, lose context between turns, expose data in a log or send the response to the wrong place. Test the end-to-end interaction a real user would have, not only the generated sentence.

IMDA's Starter Kit describes two approaches. Output testing treats the app as a black box and checks whether its overall behaviour meets expectations. Component testing looks inside, at parts such as the system prompt, input and output filters, or the retrieval step that fetches documents. The Starter Kit notes that a correct answer can hide a fault: an app might give the right answer despite retrieving irrelevant documents.

For a no-code app, a simple version of component testing is to check which document the app says it used, and whether that document really contains the answer. Our guide on how to build a simple AI app without coding experience shows where these parts sit in a basic build.

How do you improve and retest?#

Change one thing at a time, such as the prompt, the source document or a tool setting, then rerun the failed cases and the cases that passed. Keep a short log of each change and the result. An improvement on one example may not generalise, and a fix can break something that worked.

Testing does not stop at launch. The Starter Kit describes pre-deployment tests as a snapshot, because the underlying model may drift and users can change. It suggests retesting in three situations:

• Scheduled reviews: rerun a smaller, targeted subset of your original cases.
• User feedback: add cases for scenarios users raise that you have not tested.
• Significant changes to the app: repeat the same tests to see whether anything has changed.

Common mistakes when testing AI outputs#

Many testing mistakes come from making the test easier than real use. Teams try only the happy path, judge outputs by feel, or write the expected answer after seeing what the app produced. Each one makes a weak app look ready, so check your matrix against the list below before you trust the results.

• Only testing normal requests. Real users ask incomplete, odd and out-of-scope questions.
• Moving the goalposts. Write expected behaviour before running the case.
• Testing once. One good run says little about consistency.
• Checking the text but not the app. Confirm the source document, the citation and where the response goes.
• Making the app over-cautious. The Starter Kit also checks helpfulness, meaning whether the app refuses too much.
• Using real personal data in test cases. Fictional details test the same behaviour with less risk.

Who should sign off the test results?#

For high-consequence tasks, a qualified person should review results before they inform a decision, and someone should own the test matrix over time. Testing shows how the app behaved on the cases you tried; it does not prove the app will always be right, so human review stays in place.

A Singapore example#

Consider a five-person travel agency in Tanjong Pagar building an assistant that drafts replies about its published tour terms. This is an illustrative scenario. The owner writes a test matrix that includes a question about a tour the agency does not run and a request to "just confirm the refund". The assistant only goes into use when it passes the cases the owner treats as non-negotiable, and every draft is still read before it is sent.

Testing connects to wider safeguards covered in responsible AI at work, and the five practical AI application ideas for small businesses show where a human check fits in each app. For the full picture, see Build AI Apps Without Coding: A Practical Guide. UCC's GenAI Builder Sprint includes model evaluation and responsible-AI considerations before application deployment.

Sources#

1. Starter Kit for Testing LLM-Based Applications for Safety and Reliability, Infocomm Media Development Authority (IMDA), accessed 7 November 2026

Frequently asked questions#

Why should AI-generated outputs be tested?+

AI-generated outputs can sound confident while being wrong. Testing checks whether outputs meet criteria you set in advance, use only permitted information and handle gaps honestly.

What should a small test matrix include?+

Each case records the request, the expected behaviour, the actual result and whether it passed. Include normal cases, missing information, outside-the-source questions, contradictory details, sensitive input, an adversarial prompt and a repeated case.

What risks does IMDA's Starter Kit cover?+

It focuses on five key risks: hallucination and inaccuracy, bias in decision making, undesirable content, data leakage and vulnerability to adversarial prompts. The Starter Kit is a set of voluntary guidelines.

Is checking the AI's answer enough?+

No. The application around the answer can fail by not showing a citation, losing context, exposing data in a log or sending the response to the wrong place. Test the end-to-end interaction a real user would have.

When should I set the passing mark for my tests?+

Before testing. IMDA's Starter Kit explains that setting the threshold in advance encourages explicit thinking about what is acceptable and prevents moving goalposts after you see the results.

When should an AI app be retested?+

After every change to the prompt, source document or settings, at scheduled reviews, when user feedback raises a scenario you have not tested, and after significant changes to the app.