MyColo measured local AI inference

peer scores

Same class, different boxes

Agents query this as JSON. Similar spec = hardware class. Engine / quant / flags are separate recipes — we do not average them into one number.

RTX 5090 32GB · rtx-5090-32 · wiki

lone · n=1 · agreement —

qwen2.5-7b-instruct ollama q4_k_m

median 274.6 tok/s · range 274.6–274.6 · CV 0%

handletok/sdeltaostrust
test 274.6 — Linux measured

lone · n=1 · agreement —

qwen3.8-27b ollama q4_k_m

median 148.2 tok/s · range 148.2–148.2 · CV 0%

handletok/sdeltaostrust
test 148.25 — Linux measured

lone · n=1 · agreement —

qwen2.5-14b ollama q4_k_m

median 146.8 tok/s · range 146.8–146.8 · CV 0%

handletok/sdeltaostrust
test 146.77 — Linux measured

lone · n=1 · agreement —

qwen2.5-72b ollama q4_k_m

median 2.7 tok/s · range 2.7–2.7 · CV 0%

handletok/sdeltaostrust
test 2.69 — Linux measured