goerch b08e75baea Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170) ...

* Fix für #2721

* Reenable tokenizer test for LLaMa

* Add `console.cpp` dependency

* Fix dependency to `common`

* Fixing wrong fix.

* Make console usage platform specific

Work on compiler warnings.

* Adapting makefile

* Remove trailing whitespace

* Adapting the other parts of the makefile

* Fix typo.

* Fixing the last deviations from sentencepiece indicated by test-tokenizer-1

* Simplify logic

* Add missing change...

* Fix ugly compiler warning

* llama_tokenize should accept strings containing NUL now

* Adding huichen's test case

2023-09-16 13:41:33 +02:00

..

CMakeLists.txt

gguf : new file format with flexible meta data (beta) (#2398)

2023-08-21 23:07:43 +03:00

common.cpp

Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170)

2023-09-16 13:41:33 +02:00

common.h

examples : add compiler version and target to build info (#2998)

2023-09-15 16:59:49 -04:00

console.cpp

check C++ code with -Wmissing-declarations (#3184)

2023-09-15 15:38:27 -04:00

console.h

gguf : new file format with flexible meta data (beta) (#2398)

2023-08-21 23:07:43 +03:00

grammar-parser.cpp

check C++ code with -Wmissing-declarations (#3184)

2023-09-15 15:38:27 -04:00

grammar-parser.h

gguf : new file format with flexible meta data (beta) (#2398)

2023-08-21 23:07:43 +03:00

log.h

examples : replace fprintf to stdout with printf (#3017)

2023-09-05 15:10:27 -04:00