goerch afc4ca2889 convert : update convert-new.py with tokenizer fixes (#2614) ...

* Merge tokenizer fixes into the gguf branch.

* Add test vocabularies

* Adapt convert-new.py (and fix a clang-cl compiler error on windows)

2023-08-14 20:20:04 +03:00

38 KiB

Executable File

Raw Blame History

View Raw