Fabio R. Sluzala 353ec251a4 We could use std::unordered_map over std::map (#305) ...

* Improve performance by changing std::map to std::unordered_map and std::map<id, token> id_to_token; to std::vector<token> id_to_token;

* fix last commit on gpt_vocab_init add vocab.id_to_token.resize(vocab.token_to_id.size());

* Removed include <map>

* Nest struct token score inside gpt_vocab

* renamed token to tok

2023-03-21 19:21:50 +02:00

22 KiB

Raw Blame History

View Raw