Tokenization
Why cost, context limits and odd model mistakes all come back to tokens.
- Let’s build the GPT tokenizerAndrej Karpathy (video)Builds byte-pair encoding from scratch, about 2 hours.
- Tokenizers chapterHugging FaceBPE, WordPiece and Unigram, with code.
- TiktokenizerPlaygroundPaste text and see exactly how a model splits it.