Home SOC: Investigation Failures and Input Fix

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Answers passing evidence and decision checks

0 of 18

6 constructed measured cases × 3 conditions (answers)

Finished 9B reviews

0 of 24

18 measured plus 6 practice; all capped (answers)

Some answers copied real groups and then claimed there were no alerts; empty cases invented groups. Practice answers stayed outside the measured denominator.

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Recorded passes before the input fix

16 of 18

Exposed regression set; two expanded inputs were truncated (answers)

Largest original expanded input

5464 tokens

Truncated to 2050 under context 4096

Two inputs were cut from 5,107 and 5,464 tokens to 2,050. The grouped condition saw different actual evidence from the expanded conditions, so 16/18 cannot support a grouping advantage.

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Answers passing evidence and decision checks

6 of 6

The same 6 constructed cases, asked once in this condition with complete model input. These cases were already used during development. (answers)

6/6 passed in each of the three conditions. The model answered these known cases correctly after the input fix; assistance and grouping did not score higher than evidence alone. No humans took this test.

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Answers passing evidence and decision checks

6 of 6

The same 6 constructed cases, asked once in this condition with complete model input. These cases were already used during development. (answers)

6/6 passed in each of the three conditions. The model answered these known cases correctly after the input fix; assistance and grouping did not score higher than evidence alone. No humans took this test.

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Answers passing evidence and decision checks

6 of 6

The same 6 constructed cases, asked once in this condition with complete model input. These cases were already used during development. (answers)

6/6 passed in each of the three conditions. The model answered these known cases correctly after the input fix; assistance and grouping did not score higher than evidence alone. No humans took this test.

Measured on Minisforum MS-S1 MAX · Recorded backend in request receipts

Wrong-answer checks passed

1 of 6

6 deliberately wrong answers. Only 1 review met every required defect check; the larger model was unreliable as a judge. (answers)

Reviews matching every answer-key category

18 of 24

24 reviews, including practice. 18 matched every independently defined answer-key category; this is separate from the 6 wrong-answer controls. (answers)

The larger reviewer missed defects and raised false alarms. Its agreement never overrides the source score.