Tokenizers documentation
Input Sequences
Input Sequences
Not supported by the rc0 bindings yet. rc0 loads a
tokenizer.jsonand encodes and decodes with it — building a tokenizer from its components, editing one, saving one and training are not exposed. They are coming soon, along with the other bindings. SeeREQUIRED_FOR_V1.mdfor the full list.
These types represent all the different kinds of sequence that can be used as input of a Tokenizer.
Globally, any sequence can be either a string or a list of strings, according to the operating
mode of the tokenizer: raw text vs pre-tokenized.
TextInputSequence
tokenizers.TextInputSequence A str that represents an input sequence
PreTokenizedInputSequence
tokenizers.PreTokenizedInputSequence A pre-tokenized input sequence. Can be one of:
- A
Listofstr - A
Tupleofstr
alias of Union[List[str], Tuple[str]].
InputSequence
tokenizers.InputSequence Represents all the possible types of input sequences for encoding. Can be:
- When
is_pretokenized=False: TextInputSequence - When
is_pretokenized=True: PreTokenizedInputSequence
alias of Union[str, List[str], Tuple[str]].