Tag: tokenization
All the articles with the tag "tokenization".
-
The token tax of Norwegian Bokmål in LLMs
Published:• 2 min readA small experiment comparing English and Norwegian Bokmål token counts for the same official government text.
All the articles with the tag "tokenization".
A small experiment comparing English and Norwegian Bokmål token counts for the same official government text.