What is the difference between 50256-consistent-oneword and 50256-consistent?

[Calvinnncy97]

Hey guys,

Specifically, I would like to ask which flags are set to train these 2 tokenizers? I can't find any flags that force tokenizer to have only 1 word tokens.

Thank you.

[alasdairforsythe]

#24