You do not have an AI system. You have a prompt you keep retyping.

,
By Cheri Stockton, Chief Technical Therapist, Hot Hand Media.

A prompt is not a system, which is why your AI gives you a clean answer one day and a confidently wrong one the next, with nothing changed on your end but the roll of the dice. A system pins down the format, the source, and the escape hatch for when the model does not know. That difference is what makes an output hold up twice.

  • A prompt is a single request, while a system is the set of rules that forces the same request to behave the same way every time.
  • AI models are non-deterministic by design, so a retyped prompt is a hope, not a guarantee.
  • The “that is absolutely not what happened” summary failure comes from ungrounded output, not from a bad personality in the tool.
  • Specifying an exact output format and grounding answers in the source text are structural fixes a sentence cannot provide.
  • Letting the model say “I do not know” removes the pressure that makes it invent facts.
  • You trust a system because you tested it, not because it sounded right once.

You looked at the AI summary and you said, nope, that is absolutely not what happened. The meeting did not go that way. The email did not say that. So you quietly stopped trusting the tool, not with a decision, just a slow drift. Here is the thing worth saying plainly. A prompt is not a system, and what you have been doing is retyping one prompt and expecting it to act like infrastructure.

A prompt is not a system because a prompt is a single request you make to a model, while a system is the fixed set of rules, format constraints, and source grounding that forces that request to produce the same quality of result every single time you run it. The prompt is the words. The system is everything around the words that keeps the model honest. When people say they have an AI workflow, what they usually have is a sentence they paste into a chat box and a little prayer that today is a good day.

Why does the same prompt give different answers?

The same prompt gives different answers because AI models are non-deterministic, meaning the model samples from a range of likely responses rather than returning one fixed output, so identical input can produce varied results from one run to the next without anything changing on your side. This is not a bug you can scold out of the tool. It is how the technology works. OpenAI’s own documentation treats prompting as part art and part science precisely because output varies run to run. If your process depends on the model guessing the same way twice, your process is luck wearing a lab coat.

A retyped prompt is a hope. A system is a test you already passed.

So the question is not “how do I write the perfect prompt.” The question is “how do I make a good result repeatable.” Those are different jobs. One is phrasing. The other is engineering. You can get a brilliant answer from a sloppy prompt once. You cannot get it twice without structure.

What separates a good prompt from one that just repeats?

A good prompt produces a correct result once, while a repeatable prompt produces that result reliably because it specifies an exact output format, constrains the model with examples, and grounds its answers in source material instead of asking the model to improvise from memory. The gap between them is not better wording. It is structure the words sit inside. A good writer can draft a sharp sentence. A system writer builds the rails that sentence runs on.

Consider how these two approaches actually differ in practice.

The retyped prompt The actual system
You rephrase it a little each time The instruction is fixed and reused verbatim
Output format changes run to run Output format is specified and enforced
Model answers from memory Model answers from the source text you give it
No way to say “I do not know” The model is allowed to decline
You spot errors by feel, later You test outputs against known answers

Look at the right column. None of it is a cleverer sentence. All of it is a decision you make once and lock in. Anthropic’s guidance on increasing output consistency lands on the same point. Repeatable output comes from format, examples, and grounding. Not from typing the request with more feeling.

Why did the summary get it so wrong?

The summary got it wrong because it was ungrounded, meaning the model generated a plausible-sounding account from its own prediction rather than quoting the actual transcript, so it produced confident language that had no anchor in what truly happened in your meeting or your document. Confidence is not accuracy. The model is built to sound sure. That is the trap. A fluent wrong answer reads better than a careful right one, which is exactly why it costs you trust so quietly.

The fix is not a better personality. It is grounding. Anthropic’s notes on reducing hallucinations point at two moves that matter. Let the model say it does not know. And make summaries quote the source word for word instead of paraphrasing from thin air. When the model is forced to point at a line in the transcript, it stops making things up, because there is nowhere to hide.

Confidence is the easiest thing for a model to fake and the hardest thing for a reader to doubt.

Grounding in one sentence

Grounding is the practice of requiring the model to base its answer on specific source material you provide, rather than on its own stored prediction of what a plausible answer looks like. That is the whole trick behind summaries you can trust.

How do you turn a prompt into a system?

You turn a prompt into a system by writing down the four things a good run depends on, which are the exact output format you want, the source the model must use, an explicit permission to say “I do not know,” and a small set of test cases you check new outputs against before you rely on them. That is it. No stack to buy. Just decisions you stop re-making every morning.

Here is a simple way to think about the work, using a rhythm I keep coming back to. Note, Eliminate, Refine, Deliver.

  1. Note what a correct output actually looks like. Write one real example by hand. You cannot test against a standard you never defined.
  2. Eliminate the freedom the model does not need. Pin the format. Pin the source. Remove the room where it drifts.
  3. Refine by running the same input a few times and comparing. If results wobble, your constraints are too loose.
  4. Deliver only once it holds. Then reuse the exact instruction, verbatim, not a fresh paraphrase each time.

This is what OpenAI’s writing on evaluation best practices is getting at. Evals are structured tests that keep accuracy intact even though the model is non-deterministic. A real system measures. A retyped prompt only hopes. If you have never once checked a new output against a known right answer, you do not have a system yet. You have a habit.

The beautiful part is that none of this requires new software. You can build a working version of all four steps inside the chat window you already use. The platform matters less than the discipline. If you want more structured help running this kind of work, the community is where people share the setups that held up for them, right?

The word “deterministic” comes from the same root as “determine,” meaning to set a firm boundary. An AI model being non-deterministic is literal. By design, nothing fixes its answer to one boundary unless you add that boundary yourself.

Almost every time I open a client’s AI setup expecting to find a workflow, I find a single sentence saved in a notes app and a person who retypes it from memory anyway. They are not lazy. They just never crossed the line from writing a prompt to building the thing around the prompt. The moment we define one real example of a correct output and force the model to quote the source, the “nope, that is not what happened” moments drop off a cliff. Not because the model got smarter. Because we stopped letting it guess.

Why does AI give different answers to the same prompt?

Because AI models are non-deterministic and sample from a range of likely responses. Identical input can produce different output run to run. You reduce the swing by pinning the output format and grounding the answer in source text, which narrows the range the model is allowed to pick from.

How do I stop AI from making things up in summaries?

Require it to quote the source word for word and give it explicit permission to say it does not know. Most fabricated summaries happen when the model paraphrases from memory. When it must point at an actual line in the document, it has far less room to invent.

What is the difference between a prompt and an AI system?

A prompt is a single request you type. A system is the fixed format, source grounding, decline option, and test cases that make that request behave the same way every time. The prompt is the words. The system is the structure that keeps the words honest.

Do I need special software to build an AI system?

No. You can build the four core pieces, format, grounding, a decline option, and test cases, inside the chat tool you already use. The discipline matters more than the platform. Software helps at scale, but the structure is what makes any single output repeatable.

Why did I stop trusting my AI tool?

Probably because it was confidently wrong once and you never got a clear reason why. Ungrounded output sounds certain even when it is false. That quiet mismatch erodes trust faster than an obvious error, because a fluent wrong answer is harder to catch than a clumsy one.

How do I know if my prompt is actually repeatable?

Run the same input several times and compare the outputs. If they wobble in format or facts, your constraints are too loose. A repeatable prompt holds its shape across runs. If you have never tested it this way, you do not know yet, you are assuming.

Pick your single most-used prompt. The one you lean on. Then do the small thing most people skip. Write down one real example of what a correct output looks like, pin the format, tell the model it is allowed to say it does not know, and run it twice tomorrow on the same input. Watch whether it holds. If it drifts, you found your system gap, and that is good news, because now you know where to tighten.

Save this and run your best prompt again tomorrow. See if it holds.

Add Hot Hand Media as a Preferred Source

… and see more of my posts in your Google results.

Your business should not run on chaos.

Tell me what you are trying to fix and I will tell you whether Hot Hand Media is the right fit. No pitch deck. No salesperson voice.