goerch 8d177eddeb llama : improve token type support (#2668) ...

* Merge tokenizer fixes into the gguf branch.

* Add test vocabularies

* Adapt convert-new.py (and fix a clang-cl compiler error on windows)

* Improved tokenizer test

But does it work on MacOS?

* Improve token type support

- Added @klosax code to convert.py
- Improved token type support in vocabulary

* Exclude platform dependent tests

* More sentencepiece compatibility by eliminating magic numbers

* Restored accidentally removed comment

2023-08-21 18:56:02 +03:00

..

.editorconfig

editorconfig : ignore models folder

2023-08-17 19:17:25 +03:00

ggml-vocab-llama.gguf

llama : improve token type support (#2668)

2023-08-21 18:56:02 +03:00