CS348Project/llama.cpp
5
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-11-03 09:22:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ac15de789547e5a6e93df552e787379b3a23ef26
llama.cpp/convert-pth-to-ggml.py
Ronsor 956dfda8ad Use tokenizer.vocab_size() instead of hardcoding 32000 in convert-pth-to-ggml.py (#142) 
There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.
2023-03-15 21:37:50 +02:00

5.3 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink