Galunid 69a6735087 Update special token handling in conversion scripts for gpt2 derived tokenizers (#3746) ...

We still have the heads up in `README.md` regarding `bpe` tokenizers and this patch is needed for 

- a couple of tokenizer tests
- some more `special` and `non-special` added tokens handling (as far as I understand it)

* Update special token handling

* Add mpt

2023-10-23 21:46:00 +02:00

8.5 KiB

Executable File

Raw Permalink Blame History

View Raw