Mathijs Henquet 78203641fe server : Add option to return token pieces in /tokenize endpoint (#9108) ...

* server : added with_pieces functionality to /tokenize endpoint

* server : Add tokenize with pieces tests to server.feature

* Handle case if tokenizer splits along utf8 continuation bytes

* Add example of token splitting

* Remove trailing ws

* Fix trailing ws

* Maybe fix ci

* maybe this fix windows ci?

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

2024-09-12 22:30:11 +02:00

23 KiB

Raw Blame History

View Raw