/hub/runs/protectli-analyst-simulation-profile
Partial resultTest date: 2026-10-06Measured on: Minisforum MS-S1 MAXFirewall supplying the logs: Protectli V1610

Home SOC: Models Reading Simulation Logs

I converted the simulation logs into summaries, then asked each model to review them twice. Some simulations produced no alerts, so completion across all requests is shown separately from explanations on the summaries that contain alerts.

This page keeps the shared controls, retained measurements, failures, and comparison limits together. The linked files are there when you need the underlying table or method.

What you can compare here

21 source cases produced 9 different summaries: 8 with alerts and 1 empty summary shared by 13 sources. Each model made 42 requests, but only 16 were the alert-bearing tasks (8 summaries, twice). Those repeats and duplicate empty inputs do not represent 42 different incidents.

Related pages

Read the article Project repository

You can reuse Trevor-measured data from this page under the repository MIT terms. Retain the required notices when reusing the project documentation, and credit Trevor Unland when citing this run. Project media has separate terms in the repository.

Recorded results

Open the details under any row for the specific setup, outcome, artifact identity, and supporting file.

Qwen3 4B Instruct

Simulation logs; default sampling

Partial result

Measured on Minisforum MS-S1 MAX · CPU

Finished reports
42 of 42
21 source cases, each asked twice. 13 sources produced the same empty summary, so this total includes duplicate inputs. (answers)
All expected explanations shown
16 of 16
8 different summaries contain alerts, each asked twice. An answer counts only when every expected explanation is visible in the rendered report. (answers)
Passed full answer checklist
6 of 16
The same 16 alert-bearing attempts. The unedited answer must meet explanation, list-length, repetition and review-priority checks as well as finish. (answers)

Completion across all requests includes the empty summaries. Read the explanation and checklist counts for the 16 attempts with alerts. Identical repeated answers reused their source-linked agent review after exact identity checks.

Setup and original evidence

Minisforum MS-S1 MAX · Ollama 0.35.1 · CPU · Native simulation-source profile

Context window: 4,096 tokens for input and output together. CPU-thread setting: 2; this is a runtime setting, not the host's core count.

Installed artifact digest: 0edcdef34593eac1aa2be9c7d06c432dcf81945adca5eca2f27662c18f168ba0

Results and methodActual test host

Qwen3.5 4B

Simulation logs; default sampling

Partial result

Measured on Minisforum MS-S1 MAX · CPU

Finished reports
42 of 42
21 source cases, each asked twice. 13 sources produced the same empty summary, so this total includes duplicate inputs. (answers)
All expected explanations shown
4 of 16
8 different summaries contain alerts, each asked twice. An answer counts only when every expected explanation is visible in the rendered report. (answers)
Passed full answer checklist
4 of 16
The same 16 alert-bearing attempts. The unedited answer must meet explanation, list-length, repetition and review-priority checks as well as finish. (answers)

Completion across all requests includes the empty summaries. Read the explanation and checklist counts for the 16 attempts with alerts. Identical repeated answers reused their source-linked agent review after exact identity checks.

Setup and original evidence

Minisforum MS-S1 MAX · Ollama 0.35.1 · CPU · Native simulation-source profile

Context window: 4,096 tokens for input and output together. CPU-thread setting: 2; this is a runtime setting, not the host's core count.

Installed artifact digest: d8b0f5e9760cd1682034f292d7ef72ec46f432149be0df7574bf2d6e92e38c04

Results and methodActual test host

Qwen3.5 9B

Simulation logs; default sampling

Partial result

Measured on Minisforum MS-S1 MAX · CPU

Finished reports
42 of 42
21 source cases, each asked twice. 13 sources produced the same empty summary, so this total includes duplicate inputs. (answers)
All expected explanations shown
16 of 16
8 different summaries contain alerts, each asked twice. An answer counts only when every expected explanation is visible in the rendered report. (answers)
Passed full answer checklist
14 of 16
The same 16 alert-bearing attempts. The unedited answer must meet explanation, list-length, repetition and review-priority checks as well as finish. (answers)

Completion across all requests includes the empty summaries. Read the explanation and checklist counts for the 16 attempts with alerts. Identical repeated answers reused their source-linked agent review after exact identity checks.

Setup and original evidence

Minisforum MS-S1 MAX · Ollama 0.35.1 · CPU · Native simulation-source profile

Context window: 4,096 tokens for input and output together. CPU-thread setting: 2; this is a runtime setting, not the host's core count.

Installed artifact digest: 56671c2ab9385f9cfcb404638e32cd62d88e3501d44822208363c010179a3c90

Results and methodActual test host

Gemma 4 12B QAT

Simulation logs; default sampling

Partial result

Measured on Minisforum MS-S1 MAX · CPU

Finished reports
40 of 42
21 source cases, each asked twice. 13 sources produced the same empty summary, so this total includes duplicate inputs. (answers)
All expected explanations shown
14 of 16
8 different summaries contain alerts, each asked twice. An answer counts only when every expected explanation is visible in the rendered report. (answers)
Passed full answer checklist
14 of 16
The same 16 alert-bearing attempts. The unedited answer must meet explanation, list-length, repetition and review-priority checks as well as finish. (answers)

Completion across all requests includes the empty summaries. Read the explanation and checklist counts for the 16 attempts with alerts. Identical repeated answers reused their source-linked agent review after exact identity checks.

Setup and original evidence

Minisforum MS-S1 MAX · Ollama 0.35.1 · CPU · Native simulation-source profile

Context window: 4,096 tokens for input and output together. CPU-thread setting: 2; this is a runtime setting, not the host's core count.

Installed artifact digest: 38044be4f923e5a55264ed7df4eaac2676651a905f735197c504045140c02bd3

Results and methodActual test host

Supporting evidence

Results and method

These are the few files that support this page directly. The repository holds the wider project history.

Simulation-source results Distinct inputs, duplicate empty briefs, repeated assessments and complete outcomes.Simulation method Virtual lab, offline engine replay and physical inline test boundaries.

Controls

  • Ollama 0.35.1 in the isolated SOC runtime; two CPU threads and no GPU layers
  • Frozen briefs, schemas and expected contexts; no weight training or per-model tuning
  • Context 4096; output cap 700; temperature 0; seed 17; thinking disabled; keepalive zero
  • One request at a time; resource guard; source-bound validation; original failures preserved
  • 21 sources × 2 repeats × 4 models = 168 outcomes
  • 16 nonempty attempts per model; 120-second deadline; source-bound renderer checks
  • Two CPU threads were a resource limit for the scheduled service. No thread-count sweep was run; larger-model timeouts do not establish that the models could not work with more compute.
  • The 4096-token context is the input-and-output window. The separate 700-token setting limits the answer. A timeout, output-cap stop and input truncation are different failures.

What this run established

  • $166/168 requests delivered; Gemma timed out twice on the HTTP UID task.
  • $The original 4B and 9B displayed expected context on 16/16 nonempty attempts.
  • $Qwen3.5 4B delivered all requests but displayed complete specific context on 4/16 nonempty attempts.
  • $Shorter simulation briefs did not erase the dense historical-window failures.

The Lab Notes

Get an email when I publish new hardware reviews, benchmarks, and security research. Unsubscribe anytime.

Email goes to Buttondown. Privacy details.

© 2026 Trevor Unland. All rights reserved.

PrivacyTermsRSS Feed

$ echo "Built with React + TypeScript"