Tokenizers documentation
Post-processors
Getting started
API
Post-processors
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Every post-processor is a template. BertProcessing, RobertaProcessing, ByteLevel and a Sequence of post-processors were spellings of one, so a tokenizer.json using any of them is
rewritten into a TemplateProcessing when it is converted, and there is no separate class for
them.
Python
Rust
Node
PostProcessor
tokenizers.processors.PostProcessor TemplateProcessing
tokenizers.processors.TemplateProcessing