Testing and Improving Your Prompts: Building Your Library

A prompt that worked once is not necessarily a good prompt. It may have worked because of a lucky draw, an easy example or a forgiving topic. In this lesson you will learn to treat prompts like small tools: define what "good" means, test them on several cases, diagnose problems, and store the best ones in a library you can rely on.

So far you have learned how to build and shape prompts. Now comes the habit that separates occasional users from confident ones: evaluating and improving. This is also the backbone of your final project in chapter 6.

Step 1: Define what "good" means

You cannot improve what you have not defined. Before testing a prompt, write three to five evaluation criteria, phrased so that you can answer yes or no, or give a quick score. Common criteria:

• Accuracy: are the facts faithful to the source? Nothing invented?
• Completeness: is every required element present?
• Format: does it follow the requested structure and length?
• Tone and style: does it suit the reader?
• Usefulness: could you use it with little or no editing?
• Safety: no sensitive data revealed, no risky advice, no unfair assumptions.

For a "summarize meeting notes" prompt, criteria might be: contains all decisions (completeness), no invented names or dates (accuracy), under 150 words (format), neutral tone (style).

Step 2: Build a small test set

One successful run tells you very little. Create a test set of three to five inputs that represent your real use, including awkward ones:

• a typical, straightforward case;
• a short or poorly written input;
• a long or messy one;
• a case with missing information;
• an unusual case (an unexpected language, a tricky request, an input with an instruction hidden inside the text).

Use non-sensitive or anonymized material for your tests. Keep the test set in your notebook so you can rerun it every time you change the prompt. That way you know whether a change really helped, or just fixed one case while breaking another.

Step 3: Run, score, compare

Run your prompt on each test input and score the results against your criteria. A simple grid is enough:

Test case	Accuracy	Completeness	Format	Tone	Notes
1. Typical notes	OK	OK	OK	OK	Good
2. Very short notes	OK	Fail	OK	OK	Invented a deadline
3. Messy notes	OK	OK	Fail	OK	Ignored the table

Because outputs vary from run to run, run the same input two or three times if consistency matters. If the results differ widely, the prompt is probably under-specified.

Step 4: Diagnose and fix

Most problems fall into a few categories, each with a typical remedy.

Symptom	Likely cause	What to try
Answers too vague or generic	Missing context, audience or purpose	Add context, a concrete example, a precise task verb
Off-topic or missing the point	Ambiguous task, too many requests	State the goal first; split into steps; add "focus on..."
Too long or padded	No length limit	Give a number and unit; limit structure; "no introduction"
Wrong format	Format only described loosely	Provide a template or an example output
Invented facts	No source, no rule for gaps	Provide the source, "use only the text," "say not stated," ask for quotes
Wrong tone	Tone described vaguely	Name the reader and give a sample of the desired style
Ignores a rule	Rule buried or contradicted	Move it up, make it short, repeat it at the end, remove conflicts
Inconsistent results	Under-specified prompt	Add examples, fix the template, reduce open choices

Step 5: Know when a prompt is good enough

A prompt is ready for your library when it passes your criteria on most of the test set, and the failures are understood and acceptable. For low-stakes tasks, that bar can be modest. For tasks with real consequences, the bar is higher and you should keep a final human review no matter what.

Also be honest about when a prompt is the wrong tool. If, after several rounds, the task keeps failing (exact calculations, up-to-date facts, a decision requiring professional judgment), the right fix may be a different tool, extra material, or doing that part yourself.

Step 6: Build your prompt library

Good prompts are worth keeping. Store each one with enough information that you (or a colleague) can use it in six months:

• Name: a clear title ("Meeting notes to action items").
• Purpose: what it does and when to use it.
• The prompt, with variable parts in square brackets.
• Expected input and output: what you provide, what you get.
• Limits and checks: what to verify, what data must never be pasted.
• Version and date, and a one-line change log.
• Tool used, since another tool may behave differently.

A notebook, a spreadsheet or a notes app works well. Name prompts consistently, group them by task type (write, summarize, learn, organize) and review the library from time to time: delete what you no longer use, and update what no longer works as well.

Your turn: evaluate a prompt

Exercise 1. Pick one prompt you wrote in this course. Write four evaluation criteria and three test inputs (one typical, one with missing information, one messy). Run the prompt on each and fill in a small scoring grid.

Exercise 2 (project). Based on your results, make one targeted change to the prompt (one thing only), rerun the three tests, and decide whether to keep the change. Then write the library entry with the fields listed above.

See a solution

Example for a "meeting notes to action items" prompt:

• Criteria: every action has an owner or "to be defined" (completeness); no invented names or dates (accuracy); follows "Who - What - By when" (format); fewer than 10 items (usefulness).
• Tests: (1) normal notes; (2) notes where nobody is named; (3) disorganized notes with side conversations.
• Likely finding: on test 2, the model invents owners. Single change: add "If no owner is named, write 'to be defined'; never guess."
• Re-test: the invention disappears; keep v2.

Library entry: name, purpose, prompt v2 with [notes] as the variable, input and output description, limits ("check owners and dates against the notes; do not paste confidential names"), date and change log ("v2: rule for missing owners").