Douglas Hanley 177628bfd8 llama : improve BERT tokenization (#5740) ...

* implement nfd for stripping accents in wpm tokenizer

* sort nfd map; reuse iterator

* use builtin tolower

* add locale include

* Simplify to_lower cases

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

---------

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

2024-02-28 10:51:11 +02:00

95 KiB

Raw Blame History

View Raw