Christian Fillion 2d219b389e vocab : ignore invalid UTF-8 input in the BPE tokenizer (#11729) ...

Silently insert U+FFFD(s) (Unicode replacement character) instead until the
next valid codepoint can be found.

This fixes `llama_tokenize` throwing an exception across the C API boundary
or libllama's module boundary (the caller's runtime might be incompatible!)

Returing a proper error code might be desirable, however the signature
of `llama_tokenize` doesn't allow it as all return values already have
existing meaning.

2025-02-07 15:55:47 +02:00

31 KiB

Raw Blame History

View Raw