CS348Project/llama.cpp
5
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-11-10 10:27:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,960 Commits 498 Branches 4,746 Tags
032dce5a6a279d5730904cd411197e6ce5b51c53
Commit Graph

1095 Commits

Author SHA1 Message Date
Aaron Teo
032dce5a6a ggml-zdnn: fix sequencing of transforms 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:46:17 +08:00
Aaron Teo
cf0e190c40 ggml-zdnn: add more safeguards in matmul 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:44:39 +08:00
Aaron Teo
f239bbb02d ggml-zdnn: move weights transform into mulmat 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:38:44 +08:00
Aaron Teo
092fa3a328 ggml-zdnn: activate bias transform in matmul 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:27:35 +08:00
Aaron Teo
f7e8d6f2b2 ggml-zdnn: add logger to check if mat mul ops go through set_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:17:12 +08:00
Aaron Teo
6d71749c26 ggml-zdnn: add more debug info for extra buffer transform 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:10:07 +08:00
Aaron Teo
4b2f1cb1b8 ggml-zdnn: add bias data transform 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 16:05:53 +08:00
Aaron Teo
f800c80281 ggml-zdnn: add bias ztensor and data free 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 15:59:52 +08:00
Aaron Teo
bee7dd3020 ggml-zdnn: tighten memory usage, change string allocation 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 15:55:42 +08:00
Aaron Teo
aef93b3908 ggml-zdnn: add bias init_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-28 15:41:56 +08:00
Aaron Teo
f263f5d9ae ggml-zdnn: fix missing data transform call 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 18:30:10 +08:00
Aaron Teo
1c75ed63e5 ggml-zdnn: fix compiler error missing type 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 18:22:34 +08:00
Aaron Teo
a1d8568c14 ggml-zdnn: impl matmul 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 18:13:07 +08:00
Aaron Teo
59e9805ab0 ggml-zdnn: code clean up 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 16:26:29 +08:00
Aaron Teo
c1653ab639 ggml-zdnn: fix incorrect ztensor shape, reduce memory padding 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 16:22:06 +08:00
Aaron Teo
828519659b ggml-zdnn: update supports_op matmul matrix 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 16:11:37 +08:00
Aaron Teo
18658b8607 ggml-zdnn: impl init_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 12:02:20 +08:00
Aaron Teo
da2e0e70ba ggml-zdnn: switch buffers back and set to arbitrary number 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 02:31:22 +08:00
Aaron Teo
63fbc45ed6 ggml-zdnn: switch to std vector instead of array 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 01:09:01 +08:00
Aaron Teo
b7f4b6fde3 ggml-zdnn: rework init_tensor to create new buffers 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 01:03:53 +08:00
Aaron Teo
ee0ed78d54 ggml-zdnn: add check for view tensors to prevent init_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:56:32 +08:00
Aaron Teo
13c64448bd ggml-zdnn: assign tensor->extra to buffer 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:48:32 +08:00
Aaron Teo
13c05872f2 ggml-zdnn: implement at least 1 op to test 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:44:05 +08:00
Aaron Teo
9e84742e72 ggml-zdnn: test ztensor finding in init_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:40:22 +08:00
Aaron Teo
af9f4f0039 ggml-zdnn: fix compiler warnings and bugfixes 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:25:41 +08:00
Aaron Teo
ae2f656d7e ggml-zdnn: bugfix new impl 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:18:53 +08:00
Aaron Teo
7c6395f826 ggml-zdnn: rewrite the backend implementation 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-24 00:14:45 +08:00
Aaron Teo
04ddb2ac95 ggml-zdnn: update op out_prod to use tensor->extra 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-23 19:51:37 +08:00
Aaron Teo
77a753297b ggml-zdnn: support op out_prod 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-23 19:28:51 +08:00
Aaron Teo
11d58d29de ggml-zdnn: add comments to prevent accidentally deleting lines 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-23 14:54:44 +08:00
Aaron Teo
529bdb9fbd ggml-zdnn: last working matmul version 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-22 00:29:47 +08:00
Aaron Teo
60b9874dea ggml-zdnn: update set_tensor logging to check only for matmul 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-21 21:11:39 +08:00
Aaron Teo
b9756b6dd4 ggml-zdnn: add more loggers 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-21 21:09:21 +08:00
Aaron Teo
1989fc9bf4 ggml-zdnn: add set_tensor 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-21 20:37:53 +08:00
Aaron Teo
36d76c30fb ggml-zdnn: run compute and store into tensor->extra 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-21 20:30:54 +08:00
Aaron Teo
02cfcfb270 ggml-zdnn: add output buffer check 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-21 20:19:20 +08:00
Aaron Teo
fd4914b060 ggml-zdnn: tensor->extra logging check 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: add layout name mapping, ztensor information

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: separate logging into its own line

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: add shape comparison

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: add ggml_tensor shape log

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: fix incorrect shape logging

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-18 21:00:30 +08:00
Aaron Teo
e084821a3f ggml-zdnn: inital backend impl 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: temp change z17 to arch15

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

ggml-zdnn: fix build bugs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2025-07-18 20:04:20 +08:00
Reese Levine
21c021745d ggml: Add initial WebGPU backend (#14521) 
* Minimal setup of webgpu backend with dawn. Just prints out the adapter and segfaults

* Initialize webgpu device

* Making progress on setting up the backend

* Finish more boilerplate/utility functions

* Organize file and work on alloc buffer

* Add webgpu_context to prepare for actually running some shaders

* Work on memset and add shader loading

* Work on memset polyfill

* Implement set_tensor as webgpu WriteBuffer, remove host_buffer stubs since webgpu doesn't support it

* Implement get_tensor and buffer_clear

* Finish rest of setup

* Start work on compute graph

* Basic mat mul working

* Work on emscripten build

* Basic WebGPU backend instructions

* Use EMSCRIPTEN flag

* Work on passing ci, implement 4d tensor multiplication

* Pass thread safety test

* Implement permuting for mul_mat and cpy

* minor cleanups

* Address feedback

* Remove division by type size in cpy op

* Fix formatting and add github action workflows for vulkan and metal (m-series) webgpu backends

* Fix name

* Fix macos dawn prefix path
 2025-07-16 18:18:51 +03:00
Georgi Gerganov
225e7a1438 llama : add high-throughput mode (#14363) 
* kv-cache : prepare K/V buffers for separation

ggml-ci

* batched-bench : fix oob write

ggml-ci

* llama : add "virtual sequences"

ggml-ci

* llama : use "stream" vs "virtual sequence"

ggml-ci

* graph : fix stream splitting when KV cache is not used

ggml-ci

* kv-cache : add multi-stream save/load support

ggml-ci

* llama : add "--attn-streams" flag

ggml-ci

* kv-cache : fix handling when find_slot fails

ggml-ci

* kv-cache : restore find_slot impl

ggml-ci

* kv-cache : add comments

* kv-cache : add bounds checks for sequence id

ggml-ci

* cont : add n_seq_max to batch allocr

ggml-ci

* kv-cache : perform stream copies lazily after llama_synchronize

ggml-ci

* kv-cache : avoid throwing exceptions across the C boundary

ggml-ci

* CUDA: 4D FlashAttention support (#14628)

* CUDA: 4D FlashAttention support

* CUDA: fix WMMA FA kernel

* llama : rename attn_streams -> kv_unified

ggml-ci

* common : rename kv_split -> kv_unified

ggml-ci

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2025-07-16 16:35:42 +03:00
Georgi Gerganov
64978340b0 ggml : add asserts (#14720) 
* ggml : add asserts

ggml-ci

* cont : fix constant type

Co-authored-by: Diego Devesa <slarengh@gmail.com>

---------

Co-authored-by: Diego Devesa <slarengh@gmail.com>
 2025-07-16 14:43:32 +03:00
Jeff Bolz
ba1ceb3456 vulkan: fix noncontig check for mat_mul_id splitting (#14683) 
* vulkan: fix noncontig check for mat_mul_id splitting

Remove supports_op check for > 4096 (splitting fixes this)

* vulkan: fix batched matmul dequant for Q*_K
 2025-07-15 21:51:09 +02:00
Jeff Bolz
10a0351a97 vulkan: add RTE variants for glu/add/sub/mul/div (#14653) 2025-07-15 21:32:11 +02:00
R0CKSTAR
cbc68be51d cuda: fix build warnings in set-rows.cu (unused variable) (#14687) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2025-07-15 15:28:53 +08:00
Anton Mitkov
bdca38376f sycl: Hotfix for non dnnl codepath (#14677) 2025-07-14 18:12:42 +01:00
shalinib-ibm
55c509daf5 ggml : refactor llamafile_sgemm PPC code (#14673) 
Remove un-necessary templates from class definition and packing functions
Reduce deeply nested conditionals, if-else switching in mnapck function
Replace repetitive code with inline functions in Packing functions

2 ~ 7% improvement in Q8 Model
15 ~ 50% improvement in Q4 Model

Signed-off-by: Shalini Salomi Bodapati <Shalini.Salomi.Bodapati@ibm.com>
 2025-07-14 16:16:42 +03:00
Akarshan Biswas
0f4c6ec0f1 SYCL: use 1D kernel for set_rows (#14618) 
* SYCL: Use 1D kernel for set_rows

* Remove dangling comment

* Refactor and use ceil_div
 2025-07-14 10:37:55 +01:00
Anton Mitkov
65a3ebb0aa sycl: Batched mulmat rework for oneDNN dispatch (#14617) 2025-07-14 10:37:35 +01:00
Sigbjørn Skjæret
923e3ea2e3 cuda : add set rows for bf16 (#14664) 2025-07-13 15:01:24 +02:00
Yavor Ivanov
e743cddb60 cuda : add ELU support (#14657) 2025-07-13 11:33:16 +02:00