Tokenzer Design has a significant impact on the work of the language model, but evaluates the quality of the Tokizer to stay challenging. While the pressure of the text has come up as a normal metric characteristics, the latest workout questions its credibility as quality index. We are investigating whether to evaluate the Tokenzer in small models (350m) that are reliable for their impact on their large scales (2.7b parameters). By evaluating the tongers to formulated models, we find that it is the Kynizer Choice. Based on the findings, we recommend some additional metrics linked to lower performance than the screen pressure. It includes these metrics in the test program that enables reliable Token comparisons.
- † work done while in apple
- ‡ The University of Copenhagen & Rockwool Foundation What Research
