@appvoid do you actually have a B200?

Byte-level has been useful in my tiny-model tests too: a 4.90M-param byte-level GPT on a fixed 5M-char TinyStories slice reached validation PPL 1.0406–1.0632 across five seeds (mean 1.0462), though the samples were still visibly degenerate. So the tokenizer can be surprisingly capable, but low PPL on a tiny/overfit slice needs a held-out or broader test before treating it as language quality.