Skip to content

Model evaluation report

  • Host: z690-ex-glacial
  • GPU: NVIDIA GeForce RTX 3090
  • Ollama: ollama version is 0.23.2
  • Run date: 2026-05-10 15:22:11 UTC
  • Runs: 75 (15 models × task/mode pairs); errors: 1

Production picks

No model met both thresholds (min_acc=0.8, max_e2e=500ms across all tasks).

Open-set discovery viability

Open-set discovery viability. Threshold: mean cosine similarity ≥ 0.7.

No model met the threshold. Best score: 0.612.

Recommendation: do NOT ship mempalace mine --mode discover. Closed-set classification stays required.

Instruct vs reasoning (qwen3:4b)

Not enough qwen3:4b paired results to compare.

Per-task rankings

Calibration (sentence-type, exact match)

Rank Model accuracy e2e p50 ms TPS p50 VRAM resident MB
1 qwen3:4b-instruct-2507-q4_K_M 0.950 85.8 282.6 7481
2 qwen3:4b-instruct-2507-q8_0 0.950 85.2 218.1 9271
3 llama3.2:3b-instruct-q4_K_M 0.950 92.4 322.1 7126
4 phi3.5:3.8b-mini-instruct-q4_K_M 0.950 30.4 352.8 16564
5 qwen3:1.7b-q4_K_M 0.950 75.3 472.9 5024
6 qwen3:4b-instruct-2507-fp16 0.950 94.2 153.8 13215
7 qwen2.5:3b-instruct-q4_K_M 0.900 100.5 166.7 4082
8 gemma3:4b-it-q4_K_M 0.900 214.5 106.5 4745
9 gemma3:4b-it-qat 0.900 116.6 249.2 6146
10 qwen2.5:1.5b-instruct-q4_K_M 0.900 98.1 192.2 2668
11 gemma3:1b-it-q4_K_M 0.800 174.1 218.7 1416
12 llama3.2:1b-instruct-q4_K_M 0.550 107.9 182.7 3914
13 qwen2.5:0.5b-instruct-q4_K_M 0.350 93.9 323.8 1721
14 qwen3:0.6b-q4_K_M 0.250 77.0 418.9 4220
15 gemma3:270m-it-q8_0 0.050 108.6 254.7 742

Room classification — closed-set (exact match)

Rank Model accuracy e2e p50 ms TPS p50 VRAM resident MB
1 qwen3:4b-instruct-2507-fp16 0.650 129.6 104.7 13215
2 qwen3:4b-instruct-2507-q8_0 0.640 118.2 152.7 9271
3 qwen3:4b-instruct-2507-q4_K_M 0.610 109.1 206.4 7481
4 gemma3:4b-it-qat 0.540 197.0 159.1 6146
5 gemma3:4b-it-q4_K_M 0.520 222.2 169.3 4745
6 phi3.5:3.8b-mini-instruct-q4_K_M 0.470 66.3 237.0 16564
7 qwen2.5:3b-instruct-q4_K_M 0.390 110.2 254.9 4082
8 qwen2.5:1.5b-instruct-q4_K_M 0.390 101.0 339.7 2668
9 qwen3:1.7b-q4_K_M 0.280 95.1 322.1 5024
10 llama3.2:3b-instruct-q4_K_M 0.270 119.3 269.5 7126
11 qwen3:0.6b-q4_K_M 0.250 84.0 429.0 4220
12 gemma3:1b-it-q4_K_M 0.220 209.8 155.3 1416
13 llama3.2:1b-instruct-q4_K_M 0.220 107.2 349.2 3914
14 qwen2.5:0.5b-instruct-q4_K_M 0.220 91.3 522.9 1721
15 gemma3:270m-it-q8_0 0.122 153.7 304.7 742

Room classification — open-set (cosine similarity)

Rank Model accuracy exact_match_count high_similarity_count low_similarity_count mean_similarity e2e p50 ms TPS p50 VRAM resident MB
1 gemma3:4b-it-q4_K_M 0.612 2.00 14.00 32.00 0.61 213.1 171.5 4745
2 gemma3:4b-it-qat 0.612 3.00 15.00 31.00 0.61 155.9 163.9 6146
3 qwen3:4b-instruct-2507-fp16 0.595 0.00 9.00 33.00 0.59 152.6 98.8 13215
4 qwen3:4b-instruct-2507-q8_0 0.593 0.00 8.00 33.00 0.59 129.4 146.8 9271
5 qwen2.5:1.5b-instruct-q4_K_M 0.592 0.00 12.00 35.00 0.59 122.4 249.7 2668
6 phi3.5:3.8b-mini-instruct-q4_K_M 0.592 0.00 5.00 25.00 0.59 92.7 229.4 16564
7 qwen2.5:3b-instruct-q4_K_M 0.588 0.00 8.00 34.00 0.59 122.0 241.5 4082
8 qwen3:4b-instruct-2507-q4_K_M 0.586 0.00 10.00 35.00 0.59 116.2 199.9 7481
9 qwen2.5:0.5b-instruct-q4_K_M 0.575 1.00 16.00 42.00 0.57 111.4 285.9 1721
10 gemma3:1b-it-q4_K_M 0.548 1.00 5.00 57.00 0.55 195.3 171.1 1416
11 llama3.2:3b-instruct-q4_K_M 0.547 1.00 8.00 46.00 0.55 125.5 245.4 7126
12 qwen3:1.7b-q4_K_M 0.531 1.00 6.00 58.00 0.53 98.5 306.9 5024
13 llama3.2:1b-instruct-q4_K_M 0.486 0.00 6.00 71.00 0.49 127.1 219.0 3914
14 qwen3:0.6b-q4_K_M 0.455 0.00 3.00 79.00 0.46 108.8 177.6 4220
15 gemma3:270m-it-q8_0 0.381 0.00 0.00 96.00 0.38 199.8 193.8 742

Entity extraction (mean F1)

Rank Model accuracy mean_f1 mean_precision mean_recall valid_json_rate e2e p50 ms TPS p50 VRAM resident MB
1 qwen3:4b-instruct-2507-q4_K_M 0.778 0.78 0.78 0.80 1.00 624.0 172.3 7481
2 qwen3:4b-instruct-2507-fp16 0.772 0.77 0.79 0.78 1.00 1063.1 86.3 13215
3 qwen3:4b-instruct-2507-q8_0 0.766 0.77 0.78 0.78 1.00 777.4 128.3 9271
4 gemma3:4b-it-qat 0.761 0.76 0.76 0.79 1.00 834.5 150.4 6146
5 gemma3:4b-it-q4_K_M 0.743 0.74 0.74 0.77 1.00 902.4 163.1 4745
6 qwen2.5:3b-instruct-q4_K_M 0.679 0.68 0.69 0.70 1.00 2139.3 105.9 4082
7 llama3.2:3b-instruct-q4_K_M 0.672 0.67 0.74 0.63 1.00 1786.6 105.6 7126
8 phi3.5:3.8b-mini-instruct-q4_K_M 0.641 0.64 0.68 0.64 1.00 816.5 213.0 16564
9 gemma3:1b-it-q4_K_M 0.608 0.61 0.55 0.71 1.00 707.4 277.9 1416
10 qwen3:0.6b-q4_K_M 0.480 0.48 0.57 0.47 1.00 535.4 176.3 4220
11 qwen2.5:0.5b-instruct-q4_K_M 0.460 0.46 0.46 0.50 1.00 2267.0 91.4 1721
12 qwen2.5:1.5b-instruct-q4_K_M 0.367 0.37 0.39 0.35 1.00 1428.6 112.0 2668
13 llama3.2:1b-instruct-q4_K_M 0.361 0.36 0.90 0.24 1.00 528.2 121.8 3914
14 qwen3:1.7b-q4_K_M 0.313 0.31 0.67 0.23 1.00 198.6 299.6 5024

Memory extraction (mean coverage)

Rank Model accuracy mean_coverage mean_hallucination_rate mean_type_accuracy valid_json_rate e2e p50 ms TPS p50 VRAM resident MB
1 qwen3:4b-instruct-2507-q4_K_M 0.950 0.95 0.36 0.89 1.00 537.5 173.0 7481
2 qwen3:4b-instruct-2507-q8_0 0.938 0.94 0.28 0.89 1.00 638.7 129.7 9271
3 qwen3:4b-instruct-2507-fp16 0.938 0.94 0.32 0.88 1.00 921.2 86.7 13215
4 gemma3:4b-it-q4_K_M 0.912 0.91 0.19 0.90 1.00 633.4 165.1 4745
5 phi3.5:3.8b-mini-instruct-q4_K_M 0.912 0.91 0.17 0.95 1.00 765.3 211.4 16564
6 qwen2.5:1.5b-instruct-q4_K_M 0.912 0.91 0.31 0.89 1.00 1625.0 103.8 2668
7 gemma3:4b-it-qat 0.887 0.89 0.19 0.96 1.00 601.4 152.5 6146
8 qwen2.5:3b-instruct-q4_K_M 0.838 0.84 0.00 0.94 1.00 1123.2 105.0 4082
9 qwen3:1.7b-q4_K_M 0.838 0.84 0.02 0.82 1.00 279.7 306.5 5024
10 llama3.2:3b-instruct-q4_K_M 0.812 0.81 0.05 0.91 1.00 821.1 107.9 7126
11 llama3.2:1b-instruct-q4_K_M 0.762 0.76 0.22 0.55 1.00 1164.6 114.1 3914
12 qwen3:0.6b-q4_K_M 0.738 0.74 0.07 0.42 1.00 327.1 191.5 4220
13 gemma3:1b-it-q4_K_M 0.731 0.73 0.22 0.67 0.97 469.3 272.7 1416
14 qwen2.5:0.5b-instruct-q4_K_M 0.113 0.11 0.03 0.01 0.12 757.0 99.3 1721
15 gemma3:270m-it-q8_0 0.025 0.03 0.97 0.00 1.00 358.3 170.6 742

Speed (calibration, smallest task — most stable timing)

Rank Model e2e p50 ms e2e p95 ms TTFT p50 ms TPS p50 TPS p95
1 phi3.5:3.8b-mini-instruct-q4_K_M 30.4 39.7 19.4 352.8 369.9
2 qwen3:1.7b-q4_K_M 75.3 79.9 67.5 472.9 502.6
3 qwen3:0.6b-q4_K_M 77.0 84.0 68.4 418.9 448.1
4 qwen3:4b-instruct-2507-q8_0 85.2 90.2 71.2 218.1 239.9
5 qwen3:4b-instruct-2507-q4_K_M 85.8 92.5 73.0 282.6 312.0
6 llama3.2:3b-instruct-q4_K_M 92.4 5050.2 81.2 322.1 390.9
7 qwen2.5:0.5b-instruct-q4_K_M 93.9 107.8 80.1 323.8 391.6
8 qwen3:4b-instruct-2507-fp16 94.2 102.1 72.6 153.8 165.7
9 qwen2.5:1.5b-instruct-q4_K_M 98.1 105.6 78.6 192.2 222.6
10 qwen2.5:3b-instruct-q4_K_M 100.5 111.7 81.4 166.7 198.0
11 llama3.2:1b-instruct-q4_K_M 107.9 139.9 90.6 182.7 238.6
12 gemma3:270m-it-q8_0 108.6 144.6 78.5 254.7 382.9
13 gemma3:4b-it-qat 116.6 132.5 83.4 249.2 266.2
14 gemma3:1b-it-q4_K_M 174.1 183.5 161.2 218.7 319.8
15 gemma3:4b-it-q4_K_M 214.5 233.0 186.9 106.5 123.6

VRAM

Model Resident MB Peak MB Delta MB
gemma3:270m-it-q8_0 742 5546 4804
gemma3:1b-it-q4_K_M 1416 9613 8197
qwen2.5:0.5b-instruct-q4_K_M 1721 9126 7405
qwen2.5:1.5b-instruct-q4_K_M 2668 10271 7603
llama3.2:1b-instruct-q4_K_M 3914 12213 8299
qwen2.5:3b-instruct-q4_K_M 4082 16513 12431
qwen3:0.6b-q4_K_M 4220 22707 18487
gemma3:4b-it-q4_K_M 4745 17377 12632
qwen3:1.7b-q4_K_M 5024 23417 18393
gemma3:4b-it-qat 6146 13430 7284
llama3.2:3b-instruct-q4_K_M 7126 11225 4099
qwen3:4b-instruct-2507-q4_K_M 7481 22961 15480
qwen3:4b-instruct-2507-q8_0 9271 20336 11065
qwen3:4b-instruct-2507-fp16 13215 16942 3727
phi3.5:3.8b-mini-instruct-q4_K_M 16564 18595 2031

Errors

Model Task Mode Error
gemma3:270m-it-q8_0 entity_extraction default Warmup failed: Cannot reach http://localhost:11434/api/chat: timed out