There isn't really a use-case for fully-shuffled batches
* test-model-random : use F32 as the KV cache type
Temporary until F16 is fixed on ARM when using FP16_VECTOR_ARITHMETIC
This generates random models and then tests different concurrencies
of batches to check if the output is consistent.
This can detect when e.g. the recurrent cache has been broken,
or anything else which would affect the consistency of the output
when inferencing multiple distinct sequences.
More architectures will be added, but for now this starts with Mamba.
Eventually, consistency of pooled embeddings will also be tested.
The goal is to reduce accidental regressions
by making it easy to quickly test a lot of edge cases
on the supported architectures,
without having to download any model.
