CS348Project/llama.cpp
5
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-10-30 08:42:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dd0dc366dab10e8df28d3924e7f313b5c695e908
llama.cpp/examples/perplexity/perplexity.cpp
Kawrakow 463173a6c0 llama : speedup tokenization (#2831) 
* Speedup tokenization

On current master it takes ~3.2 seconds to tokenize
Wikitext. With this change it becomes ~525 ms.

* Fixit: it was missing the piece after the last found occurence

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-08-27 16:50:33 +03:00

24 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink