Michael Coppola
|
940362224d
|
llama : add support for Tekken pre-tokenizer (#8579)
* llama : Added support for Tekken pre-tokenizer (#8577)
Removed uneeded `vocab.tokenizer_clean_spaces` assignment
* llama : fix order of pre-tokenizers
* * Tekken pre-tokenizer no longer uses clean_up_tokenization_spaces
* Updated chkhsh for Tekken tokenizer
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
2024-07-20 16:43:51 +03:00 |
|
Georgi Gerganov
|
e235b267a2
|
py : switch to snake_case (#8305)
* py : switch to snake_case
ggml-ci
* cont
ggml-ci
* cont
ggml-ci
* cont : fix link
* gguf-py : use snake_case in scripts entrypoint export
* py : rename requirements for convert_legacy_llama.py
Needed for scripts/check-requirements.sh
---------
Co-authored-by: Francis Couture-Harpin <git@compilade.net>
|
2024-07-05 07:53:33 +03:00 |
|
ditsuke
|
01a5f06550
|
chore: Remove rebase artifacts
|
2024-07-04 15:39:13 +00:00 |
|
ditsuke
|
b0a46993df
|
build(python): Package scripts with pip-0517 compliance
|
2024-07-04 15:39:13 +00:00 |
|