tokenization (noun)
The process by which a body of text is broken down into tokens before an AI model can read or act on it. Tokenization happens automatically and invisibly, but understanding that it happens explains why AI systems sometimes handle language in ways that feel slightly literal or unpredictable. The model is working with fragments, not the whole picture at once.
Example: "Odd spacing or formatting in a document can affect how it's tokenized, and in turn how well the model understands it."
Related: token, token budget