Tokenizers

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Trainers

TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Python

Rust

Node

BpeTrainer

tokenizers.trainers.BpeTrainer

ParityBpeTrainer

tokenizers.trainers.ParityBpeTrainer

UnigramTrainer

tokenizers.trainers.UnigramTrainer

WordLevelTrainer

tokenizers.trainers.WordLevelTrainer

WordPieceTrainer

tokenizers.trainers.WordPieceTrainer

Update on GitHub

←Post-processors Decoders→