A tokenizer maps a rare technical word to several subword tokens. Which statement correctly describes this behavior?

Published
  1. Every word always occupies exactly one token.
  2. One visible word can consume multiple tokens, affecting context and generation budgets.
  3. Subword tokens must each be a complete dictionary word.
  4. Tokenization automatically verifies the factual accuracy of the text.

Comments — 0