Tokenizers documentation
Decoders
Decoders
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Python
Rust
Node
DecodeStream
tokenizers.decoders.DecodeStream BPEDecoder
tokenizers.decoders.BPEDecoder ByteFallback
tokenizers.decoders.ByteFallback ByteLevel
tokenizers.decoders.ByteLevel CTC
tokenizers.decoders.CTC Fuse
tokenizers.decoders.Fuse Metaspace
tokenizers.decoders.Metaspace Replace
tokenizers.decoders.Replace Sequence
tokenizers.decoders.Sequence Strip
tokenizers.decoders.Strip WordPiece
tokenizers.decoders.WordPiece