mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2025-11-03 09:22:01 +00:00
370359e5ba
* WIP: start implementing LLaVA * rm scratch buf for now, will revert after cleanup * LLaVA image encoder is working. will combine with llama * Add llava inference code, but it's buggy. debugging * LLaVA is working e2e, needs to optimize memory allocation + cleanup * Use ggml_allocr + rm unnecessary code * fix: crlf -> lf * fix: new line at EoF * fix: trailing whitespace * Add readme * Update readme * Some cleanup * Are you happy editorconfig? * rm unused batch image preprocessing * rm unused import * fix: rm designated initializers * introduce pad-to-square mode for non-square images * are you happy editorconfig? * gitignore /llava * Handle cases where image file does not exist * add llava target to Makefile * add support for 13b model variant * Maybe seed is unlucky? * Check if apples are compared to apples * are you happy editorconfig? * Use temperature = 0.1 by default * command line: use gpt_params_parse() * minor * handle default n_predict * fix typo * llava : code formatting, rename files, fix compile warnings * do not use Wno-cast-qual for MSVC --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
58 lines
1.7 KiB
Markdown
58 lines
1.7 KiB
Markdown
# LLaVA
|
|
|
|
Currently this implementation supports [llava-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) variants.
|
|
|
|
The pre-converted [7b](https://huggingface.co/mys/ggml_llava-v1.5-7b)
|
|
and [13b](https://huggingface.co/mys/ggml_llava-v1.5-13b)
|
|
models are available.
|
|
|
|
After API is confirmed, more models will be supported / uploaded.
|
|
|
|
## Usage
|
|
Build with cmake or run `make llava` to build it.
|
|
|
|
After building, run: `./llava` to see the usage. For example:
|
|
|
|
```sh
|
|
./llava -m llava-v1.5-7b/ggml-model-q5_k.gguf --mmproj llava-v1.5-7b/mmproj-model-f16.gguf --image path/to/an/image.jpg
|
|
```
|
|
|
|
**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.
|
|
|
|
## Model conversion
|
|
|
|
- Clone `llava-v15-7b`` and `clip-vit-large-patch14-336`` locally:
|
|
|
|
```sh
|
|
git clone https://huggingface.co/liuhaotian/llava-v1.5-7b
|
|
|
|
git clone https://huggingface.co/openai/clip-vit-large-patch14-336
|
|
```
|
|
|
|
2. Use `llava-surgery.py` to split the LLaVA model to LLaMA and multimodel projector constituents:
|
|
|
|
```sh
|
|
python ./examples/llava/llava-surgery.py -m ../llava-v1.5-7b
|
|
```
|
|
|
|
3. Use `convert-image-encoder-to-gguf.py` to convert the LLaVA image encoder to GGUF:
|
|
|
|
```sh
|
|
python ./examples/llava/convert-image-encoder-to-gguf -m ../clip-vit-large-patch14-336 --llava-projector ../llava-v1.5-7b/llava.projector --output-dir ../llava-v1.5-7b
|
|
```
|
|
|
|
4. Use `convert.py` to convert the LLaMA part of LLaVA to GGUF:
|
|
|
|
```sh
|
|
python ./convert.py ../llava-v1.5-7b
|
|
```
|
|
|
|
Now both the LLaMA part and the image encoder is in the `llava-v1.5-7b` directory.
|
|
|
|
## TODO
|
|
|
|
- [ ] Support server mode.
|
|
- [ ] Support non-CPU backend for the image encoding part.
|
|
- [ ] Support different sampling methods.
|
|
- [ ] Support more model variants.
|