CS348Project/llama.cpp
5
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-10-30 08:42:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0cd6bd3483fa66124b76a8a8ac794d9ee18c70c1
llama.cpp/ggml-cuda/template-instances
History
Johannes Gäßler 9b596417af CUDA: quantized KV support for FA vec (#7527) 
* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code
2024-06-01 08:44:14 +02:00
..
fattn-vec-f16-instance-hs64-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs64-f16-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs64-f16-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs64-f16-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs64-f16-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs64-f16-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-f16-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q4_1-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q5_1-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs128-q8_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f16-instance-hs256-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs64-f16-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-f16-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q4_1-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q5_1-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-q4_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-q4_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-q5_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-q5_1.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs128-q8_0-q8_0.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-vec-f32-instance-hs256-f16-f16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-wmma-f16-instance-kqfloat-cpb16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-wmma-f16-instance-kqfloat-cpb32.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-wmma-f16-instance-kqhalf-cpb8.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-wmma-f16-instance-kqhalf-cpb16.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
fattn-wmma-f16-instance-kqhalf-cpb32.cu
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00
generate_cu_files.py
CUDA: quantized KV support for FA vec (#7527)
2024-06-01 08:44:14 +02:00