llama.cpp/tests/test-tokenizer-1-bpe.cpp at db4b8ac37a42db4dfec5fe002e70f89dad7702d6

mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-11-09 10:17:06 +00:00

Files

Galunid daab3d7f45 Add more tokenizer tests (#3742 )

* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test

2023-10-24 09:17:17 +02:00

4.3 KiB

Raw Blame History

View Raw

4.3 KiB Raw Blame History

4.3 KiB

Raw Blame History