llama.cpp

mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-11-01 09:01:57 +00:00

Files

goerch 49c25cce19 tests : use new tokenizer type API (#2692 )

* Merge tokenizer fixes into the gguf branch.

* Add test vocabularies

* Adapt convert-new.py (and fix a clang-cl compiler error on windows)

* Improved tokenizer test

But does it work on MacOS?

* Improve token type support

- Added @klosax code to convert.py
- Improved token type support in vocabulary

* Exclude platform dependent tests

* More sentencepiece compatibility by eliminating magic numbers

* Restored accidentally removed comment

* Improve commentary

* Use token type API in test-tokenizer-1.cpp

2023-08-21 20:11:14 +03:00

CMakeLists.txt

llama : remove C++ API + reorganize common source in /common dir

2023-08-18 16:22:48 +03:00

test-double-float.cpp

tests : Fix compilation warnings (Linux/GCC) (#2451 )

2023-08-02 11:06:19 +03:00

test-grad0.cpp

tests : Fix compilation warnings (Linux/GCC) (#2451 )

2023-08-02 11:06:19 +03:00

test-grammar-parser.cpp

llama : remove C++ API + reorganize common source in /common dir