Billel Mokeddem 7ae33a616f llama : add Falcon3 support (#10883) ...

* Add Falcon3 model support

* Add fix for adding bos to added special tokens

* Add comment explaining the logic behind the if statement

* Add a log message to better track the when the following line of code is triggered

* Update log to only print when input and output characters are different

* Fix handling pre-normalized tokens

* Refactoring

2024-12-23 00:09:58 +02:00

16 KiB

Executable File

Raw Blame History

View Raw