How Large Language Models Read Your Prompt: Tokens, Probability, and Context

Key takeaway & what you'll build

A language model reads your prompt as tokens and predicts the most likely continuation, one token at a time. It has no memory beyond what's in the conversation, it can't check facts on its own, and it fills every gap with the most common answer. Write prompts that leave fewer gaps, and check anything that matters.

You'll build: Your own prompt library

Prediction, not lookup

A large language model doesn't look answers up. It reads your prompt and predicts, one small piece at a time, what text is most likely to come next given everything it learned in training.

Those small pieces are tokens: roughly a short word or part of a longer one. "Prompt engineering" is about three tokens. Models read, think, and are priced in tokens, which is why long prompts and long answers cost more.

Four things that follow from this

1. Gaps get filled with the average

If your prompt doesn't say who the audience is, the model writes for the most typical audience. If it doesn't say how long, you get a typical length. Every unanswered question is answered with the most common option.

2. It only knows what's in the conversation

Within a chat, the model sees the earlier messages. Outside it, it remembers nothing about you unless the tool gives it that context. (In Ace, Brand Profiles and your knowledge vault do this; "Make AI Sound Like You" covers them.)

3. Confident isn't the same as correct

Because the model predicts plausible text, it can produce a fluent answer that's wrong, such as an invented statistic, quote, or source. This is often called hallucination. Ask for sources you can check, give it the facts you need it to use, and verify anything that matters.

4. Wording steers the result

The model responds to everything in the prompt, including tone and examples. A casual prompt tends to get a casual answer; one example in your prompt can shape every item in the reply. Later lessons use this on purpose.

Different models, same principles

Gemini, Claude, and GPT differ in style and strengths, and Ace's model selector in Ask Ace lets you choose between them. The principles in this course apply to all of them.

Try it

In Ask Ace, ask: "Give me three statistics about remote work, with sources." Then check one of the sources. Whatever you find, it's a lesson in why the output-format and input parts of a prompt matter: next time, give the model the facts and ask it to work with them.