DAN™ 889bdd7686 command-r : add BPE pre-tokenization (#7063) ...

* Add BPE pre-tokenization for Command-R/R+.

* Bump transformers convert requirement.

* command-r : add individual digits regex

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

2024-05-05 08:19:30 +03:00

11 KiB

Executable File

Raw Blame History

View Raw