Tokenizers documentation

Post-processors

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Post-processors

TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Every post-processor is a template. BertProcessing, RobertaProcessing, ByteLevel and a Sequence of post-processors were spellings of one, so a tokenizer.json using any of them is rewritten into a TemplateProcessing when it is converted, and there is no separate class for them.

Python
Rust
Node

PostProcessor

tokenizers.processors.PostProcessor

TemplateProcessing

tokenizers.processors.TemplateProcessing
Update on GitHub