Georgi Gerganov
|
e0429d38e4
|
convert-new.py : output gguf (#2635)
* convert-new.py : output gguf (WIP)
* convert-new.py : add gguf key-value pairs
* llama : add hparams.ctx_train + no longer print ftype
* convert-new.py : minor fixes
* convert-new.py : vocab-only option should work now
* llama : fix tokenizer to use llama_char_to_byte
* tests : add new ggml-vocab-llama.gguf
* convert-new.py : tensor name mapping
* convert-new.py : add map for skipping tensor serialization
* convert-new.py : convert script now works
* gguf.py : pick some of the refactoring from #2644
* convert-new.py : minor fixes
|
2023-08-17 17:19:52 +03:00 |
|
Georgi Gerganov
|
c8ee87f141
|
gguf.py : merge all files in gguf.py
|
2023-08-16 19:55:49 +03:00 |
|
klosax
|
b6056c3db8
|
gguf.py : add tensor data layout
|
2023-08-15 19:53:44 +02:00 |
|
klosax
|
cedb4870c6
|
gguf.py : add token types
|
2023-08-14 22:08:40 +02:00 |
|
Georgi Gerganov
|
62490f1380
|
gguf : use UNIX line ending
|
2023-08-14 13:04:35 +03:00 |
|
klosax
|
5c5a95ba2d
|
gguf.py : dont add empty strings
|
2023-08-14 11:22:06 +02:00 |
|
M. Yusuf Sarıgöz
|
91d4bfd536
|
convert : write more metadata for LLaMA
|
2023-08-13 13:29:46 +03:00 |
|
klosax
|
d2ce9cfe8d
|
gguf.py : n_layer --> n_block
|
2023-08-13 00:01:20 +02:00 |
|
klosax
|
5d81a715d4
|
gguf.py : no need to convert tensors twice
|
2023-08-12 21:45:45 +02:00 |
|
klosax
|
4357e692ac
|
gguf.py : use custom alignment if present
|
2023-08-07 13:51:26 +02:00 |
|
klosax
|
278ada9572
|
gguf.py : bytesarray for gpt2bpe tokenizer
|
2023-08-04 04:07:57 +02:00 |
|
klosax
|
49380a23a3
|
gguf.py : accumulate kv and tensor info data + special tokens
|
2023-08-01 23:37:48 +02:00 |
|
M. Yusuf Sarıgöz
|
7aa0a0e7f7
|
gguf : support custom alignment value
|
2023-07-31 09:59:36 +03:00 |
|
klosax
|
ccd81a751b
|
gguf.py : add layer norm eps and merges
|
2023-07-30 14:48:14 +02:00 |
|
M. Yusuf Sarıgöz
|
32e037ffbe
|
gguf : fix set is not subscriptable
|
2023-07-30 01:01:13 +03:00 |
|
klosax
|
9577821487
|
gguf.py : support any type
|
2023-07-29 21:29:07 +02:00 |
|
M. Yusuf Sarıgöz
|
0f5e57f01d
|
gguf : handle already encoded string
|
2023-07-29 19:56:06 +03:00 |
|
M. Yusuf Sarıgöz
|
cc3dd7f042
|
gguf : write tokenizer data
|
2023-07-29 13:30:22 +03:00 |
|
M. Yusuf Sarıgöz
|
8a76dd8a85
|
gguf : write tensors one by one
|
2023-07-29 13:17:28 +03:00 |
|
M. Yusuf Sarıgöz
|
c861e234f4
|
gguf : write tensors one by one
|
2023-07-29 12:49:01 +03:00 |
|
M. Yusuf Sarıgöz
|
0c219fb5b5
|
gguf : fix writing gguf arrays
|
2023-07-29 12:42:54 +03:00 |
|
M. Yusuf Sarıgöz
|
93f7f7aef7
|
gguf : write tensors one by one and code reuse
|
2023-07-29 12:34:35 +03:00 |
|
M. Yusuf Sarıgöz
|
ea5f9ad2ca
|
gguf : fix writing gguf arrays
|
2023-07-29 12:25:43 +03:00 |
|
M. Yusuf Sarıgöz
|
06f423a8e1
|
gguf : write sample tensors to read
|
2023-07-29 10:26:26 +03:00 |
|
M. Yusuf Sarıgöz
|
1495735aac
|
gguf : fix writing tensors
|
2023-07-29 00:26:22 +03:00 |
|
M. Yusuf Sarıgöz
|
11ef380c2a
|
GGUF : write tensor (#2426)
* WIP: Write tensor
* GGUF : Support writing tensors in Python
* refactor : rm unused import and upd todos
* fix : fix errors upd writing example
* rm example.gguf
* gitignore *.gguf
* undo formatting
|
2023-07-28 11:34:16 +03:00 |
|
Georgi Gerganov
|
158be8f7f4
|
gguf.py : some code style changes
|
2023-07-27 15:37:06 +03:00 |
|
M. Yusuf Sarıgöz
|
c85d3178b3
|
refactor : reduce code duplication and better API (#2415)
|
2023-07-27 10:29:29 +03:00 |
|
M. Yusuf Sarıgöz
|
bae6b125f6
|
wip : implement GGUF (#2397)
* Add LLAMA_DEFAULT_RMS_EPS so we can change the default (#2384)
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* WIP: python class to write GGUF, incomplete C apı for reading
---------
Co-authored-by: Kawrakow <48489457+ikawrakow@users.noreply.github.com>
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
|
2023-07-26 18:21:13 +03:00 |
|