Tokenizers documentation
Pre-tokenizers
Pre-tokenizers
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Python
Rust
Node
BertPreTokenizer
tokenizers.pre_tokenizers.BertPreTokenizer ByteLevel
tokenizers.pre_tokenizers.ByteLevel CharDelimiterSplit
tokenizers.pre_tokenizers.CharDelimiterSplit Digits
tokenizers.pre_tokenizers.Digits Metaspace
tokenizers.pre_tokenizers.Metaspace PreTokenizer
tokenizers.pre_tokenizers.PreTokenizer Punctuation
tokenizers.pre_tokenizers.Punctuation Sequence
tokenizers.pre_tokenizers.Sequence Split
tokenizers.pre_tokenizers.Split UnicodeScripts
tokenizers.pre_tokenizers.UnicodeScripts Whitespace
tokenizers.pre_tokenizers.Whitespace WhitespaceSplit
tokenizers.pre_tokenizers.WhitespaceSplit