BM
Bashar ModallalSOFTWARE · AI · SYSTEMS
← All field notes

AI / الذكاء الاصطناعي

Few-Shot Prompting: Teaching an AI Through Examples and Counterexamples

Few-shot prompts use examples to show a task’s labels, input patterns, and output format. Learn how counterexamples and ordering clarify the decision boundary.

Four illustrated prompt cards show two examples, a counterexample, and a final output connected in sequence.

Few-Shot Prompting: Teaching an AI Through Examples and Counterexamples

Few-shot prompting places a small number of input-output demonstrations in a request, then asks a model to handle a new case. The examples provide context; they do not update the model's parameters. Brown et al. evaluated this setting across language tasks and also reported important limitations. The practical question is not how many examples to paste, but which pattern they make clear.

What examples show

Demonstrations can communicate the label set, the kind of inputs to expect, and the requested output format. Min et al. found that these facets can matter even when demonstration labels are replaced in controlled experiments. A model may imitate a pattern without learning the intended rule, so examples should support a clear written instruction.

Start with a zero-shot baseline. For example:

Classify each support request as scheduling, billing, or other. Return only the class name.

Then add short, varied cases:

"Can we move Thursday's review to Friday?" → scheduling
"The invoice lists the same charge twice." → billing
"The dashboard's export button is missing." → other

These examples establish the label vocabulary and response shape. They do not cover every request. Choose representative, unambiguous cases; avoid near-duplicates that repeat one side of the decision.

Counterexamples clarify boundaries

A counterexample isolates a tempting but incorrect cue. Compare:

"Move the Friday review to Monday." → scheduling
"The Friday invoice has the wrong total." → billing

The shared day name is not the rule; the requested action is. A useful contrast pair changes one meaningful feature while keeping the rest similar. If several features change, it becomes harder to tell which distinction the examples illustrate.

State the decision rule in words too: choose scheduling for requests to arrange or change a meeting; choose billing for questions about invoices, payments, or charges. If a case could reasonably fit two labels, revise the taxonomy or specify a tie-break rule instead of hoping an example will settle it.

Keep format stable and test ordering

Few-shot behavior can depend on example order. Lu et al. found large ordering effects on the text-classification tasks they examined, and a useful order for one model did not necessarily transfer to another. This does not mean every prompt needs extensive permutation search. It means a single favorable ordering is not proof of robustness.

Use the same labels, separators, and answer position in every demonstration. Put the instruction first, examples next, and the new input last. If order may matter, compare a small documented set of permutations on held-out cases. Do not tune on the final evaluation examples and then treat them as independent evidence.

A practical workflow

  1. Write the rule and allowed outputs before selecting examples.
  2. Establish a zero-shot baseline.
  3. Add examples only to clarify a missing distinction.
  4. Include ordinary cases and carefully chosen boundary cases.
  5. Test unseen, ambiguous, and out-of-scope inputs.
  6. Record the exact prompt and compare a few orders if needed.
  7. Provide an "uncertain" or "other" path rather than forcing a label without evidence.

Few-shot prompting specifies a task through context; it does not guarantee dependable learning. Clear rules, informative contrasts, and evaluation on unseen cases make behavior easier to inspect. For consequential workflows, add validation and human review rather than relying on examples alone.

Arabic edition: https://basharmodallal.com/blog/few-shot-prompting-examples-counterexamples-ar

References

#LLM

YOUR TAKE

What did you think?

Choose one reaction for this article.

Was this useful?

A cookie helps limit repeat responses. Your feedback is private.

Share this noteLinkedIn ↗X ↗Email ↗

Keep exploring.