llama.cpp/tests/test-tokenizer-random.py at 5b6da187508f49a9fa9d95fa22ae804a0780d256

mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-10-31 08:51:55 +00:00

Files

jaime-m-p 3b38d48609 Per token attributes (#7685 )

* Add per token attributes enum
* Using phi-3 for testing 'rstrip'
* Using jina-v2 for testing 'lstrip'
* Brute force test for 'lstrip' and 'rstrip'
* Implement 'rstrip' and 'lstrip'
* Update phi-3 GGUF file (obsolete since 917dc8c)
* Replace llama_token_type with llama_token_attribs

2024-06-04 09:17:17 +02:00

13 KiB

Raw Blame History

View Raw

13 KiB Raw Blame History

13 KiB

Raw Blame History