Token
A unit of input or output used by a model's tokenizer. A token may be a word, part of a word, punctuation, or another encoded piece of information.
For example
The word "unbelievable" might be split into several tokens. There is no fixed rule that one word always equals one token.
By Dave AI Editorial · Prompts & context