CS348Project/llama.cpp
5
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-11-01 09:01:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
3d82dbcbce2c677fc35fbf99574ccd107d95a1f8
llama.cpp/convert_hf_to_gguf.py
Bartowski 732b5fbf5e convert : avoid calls to tokenizer.added_tokens_decoder (#12473) 
tokenizer.added_tokens_decoder returns a fresh dict every time relatively slowly (~0.04s on average) which results in massive slowdowns when we have a huge number of added tokens
2025-03-20 08:36:37 +02:00

243 KiB
Executable File
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink