This website uses cookies to ensure you get the best experience.
Learn more.
A tokenizer maps a rare technical word to several subword tokens. Which statement correctly describes this behavior?
Published
- Every word always occupies exactly one token.
- One visible word can consume multiple tokens, affecting context and generation budgets.
- Subword tokens must each be a complete dictionary word.
- Tokenization automatically verifies the factual accuracy of the text.
Comments — 0
Sign in to comment