staviq 1a159553f9 tokenizer : special token handling (#3538) ...

* Rewrite special token handling from #1931

* shorten param name, add st verification by type

* use offsets instead of copy by substr

* formatting, remove copying iterator on delete

* llama : normalize code-style

* swift fix

* print pfx/sfx if verb, main: split pfx input sfx

* dont add space when using special tokens

* minor : comment + spacing

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

2023-10-17 18:11:01 +03:00

7.3 KiB

Raw Blame History

View Raw