Tokens, not words

Learn in five minutes

• A model reads tokens, which may be parts of words.
• A space-split counter is a toy and is labeled as one.
• Token counts affect cost and context. Confirm a vendor price before you quote it.
• The 2017 transformer paper is the usual citation for the architecture family.
• Different models tokenize the same sentence differently.

Do not compare two products by a word count. Compare token counts from each tokenizer, and say which tokenizer you used.

This section HTML

Visual

Diagram SVGDiagram PNGDiagram HTML

Flow SVGFlow PNGFlow HTML

One sentence, two counts

1. Take the sentence 'do not split blindly'.
2. Count spaces plus one and label it a toy.
3. If you have a real tokenizer, record its name and its count beside the toy.

This section HTML

Quiz

This section HTML

Ecosystem

Token

The piece of text a given model reads or emits.

Sisters: Tokenizers, Transformers, Decoding, Model cards.

The same next-token idea sits under chat, voice, and tool calls.

Pin a model id when you leave the toy explanations.

Config

• Write the goal in one sentence.
• Name the input and the allowed output.
• Save the trace next to the result.

Troubleshooting

• The answer looks fluent and is wrong. Check whether a source passage actually contains the claim.
• The loop never stops. Set a step cap and stop when the tool result repeats.

This section HTML