Georgi Gerganov 2f9ec7e271 cuda : improve text-generation and batched decoding performance (#3776) ...

* cuda : prints wip

* cuda : new cublas gemm branch for multi-batch quantized src0

* cuda : add F32 sgemm branch

* cuda : fine-tune >= VOLTA params + use MMQ only for small batches

* cuda : remove duplicated cuBLAS GEMM code

* cuda : add CUDA_USE_TENSOR_CORES and GGML_CUDA_FORCE_MMQ macros

* build : add compile option to force use of MMQ kernels

2023-10-27 17:01:23 +03:00

299 KiB

Raw Permalink Blame History

View Raw