AI failure analysis for hardware

Dim on cold mornings.
The parts are fine.

A lamp glows dim on one side, only on cold mornings. The issues list blames the LED lot. Closeout read 5,313 raw log rows, asked for one test, scrapped nothing, and proved the cause. Then it wrote the fix.

As shipped · cold start0.18uniformity error · FAIL
Same unit · fixed firmware0.02uniformity error · PASS
MIL-014Dome uniformity fail at S850 of 479 failedMaterialFirmware
LIVE C Readouts come from compiled firmware run when this page was built. Fictional beacon, synthetic units.
01The problem

Three teams.
Three theories.

Yield drops. An issue opens. The blame moves down the line.

  1. Assemblyblames
  2. Materialblames
  3. Test Stationblames
  4. Firmware
a

Every teardown scraps a unit.

Failures are few. Open one without a question and it's gone for nothing.

b

The evidence is scattered.

Logs in one system. FA reports in another. The change log in an inbox. Nobody reads them together.

c

Fixes ship before the cause is proven.

A lot gets swapped. The weather warms up. Yield recovers. Nobody knows if the fix worked.

02The dataREPLAY · RECORDED 2026-11-09

Every row.
No summary.

480 units. 8 stations. 3 days. No summary and no hints. The model reads every raw row in one call. Code computes every rate.

5,313raw log rows
0.9 MBof station data
~232Ktokens read in one pass
2open issues on the list
40 of 5,313 rowsbuild BCN-EVT2-B07
unitU0006ts2026-10-27T06:25:00stationS1-shellopOP-3a1eventassembleresultokambient_c21.2unitU0006ts2026-10-27T06:29:00stationS2-pcbaopOP-3a1eventplaceresultokdriver_reelDRV-A-R3unitU0006ts2026-10-27T06:32:00stationS3-screwsopOP-e58eventscrewdriveresultoktorque_ncm27.7unitU0006ts2026-10-27T06:35:00stationS4-speaker-hapticopOP-2d9eventplaceresultokunitU0006ts2026-10-27T06:37:00stationS5-batteryopOP-3a1eventfitresultokunitU0006ts2026-10-27T06:41:00stationS6-dome-bondopOP-3a1eventbondresultokfixtureF2dome_offset_mm0.073led_lotL-2604unitU0006ts2026-10-27T06:44:00stationS7-flash-calopOP-3a1eventflash_calresultokfw1.4.2calwrittenboot_ms50unitU0006ts2026-10-27T06:46:00stationS8-functional-packopOP-2d9eventtesttestT-OPT-01cameraCAM-2uniformity0.036limit0.05resultpassunit_temp_c22.2unitU0006ts2026-10-27T06:47:00stationS8-functional-packopOP-2d9eventtesttestT-TCH-01nestN2baseline_drift8limit30resultpassunitU0006ts2026-10-27T06:48:00stationS8-functional-packopOP-7c2eventtesttestT-AUD-01snr_db24.5limit18resultpassunitU0006ts2026-10-27T06:49:00stationS8-functional-packopOP-7c2eventtesttestT-PWR-01rail_mv3285resultpassunitU0016ts2026-10-27T07:15:00stationS1-shellopOP-2d9eventassembleresultokambient_c19.7unitU0016ts2026-10-27T07:19:00stationS2-pcbaopOP-7c2eventplaceresultokdriver_reelDRV-A-R3unitU0016ts2026-10-27T07:22:00stationS3-screwsopOP-7c2eventscrewdriveresultoktorque_ncm28.9unitU0016ts2026-10-27T07:25:00stationS4-speaker-hapticopOP-3a1eventplaceresultokunitU0016ts2026-10-27T07:27:00stationS5-batteryopOP-7c2eventfitresultokunitU0016ts2026-10-27T07:31:00stationS6-dome-bondopOP-e58eventbondresultokfixtureF2dome_offset_mm0.056led_lotL-2604unitU0016ts2026-10-27T07:34:00stationS7-flash-calopOP-91feventflash_calresultokfw1.4.2calwrittenboot_ms50unitU0016ts2026-10-27T07:36:00stationS8-functional-packopOP-7c2eventtesttestT-OPT-01cameraCAM-2uniformity0.211limit0.05resultfailunit_temp_c20.7unitU0016ts2026-10-27T07:37:00stationS8-functional-packopOP-b04eventtesttestT-TCH-01nestN1baseline_drift14limit30resultpassunitU0016ts2026-10-27T07:38:00stationS8-functional-packopOP-b04eventtesttestT-AUD-01snr_db23.1limit18resultpassunitU0016ts2026-10-27T07:39:00stationS8-functional-packopOP-e58eventtesttestT-PWR-01rail_mv3307resultpassunitU0016ts2026-10-27T08:24:00stationS8-functional-packopOP-91feventretesttestT-OPT-01cameraCAM-2uniformity0.209limit0.05resultfailunit_temp_c22unitU0037ts2026-10-27T09:04:00stationS2-pcbaopOP-7c2eventplaceresultokdriver_reelDRV-A-R2unitU0037ts2026-10-27T09:07:00stationS3-screwsopOP-7c2eventscrewdriveresultoktorque_ncm28.3unitU0037ts2026-10-27T09:10:00stationS4-speaker-hapticopOP-3a1eventplaceresultokunitU0037ts2026-10-27T09:12:00stationS5-batteryopOP-91feventfitresultokunitU0037ts2026-10-27T09:16:00stationS6-dome-bondopOP-91feventbondresultokfixtureF2dome_offset_mm0.044led_lotL-2604unitU0037ts2026-10-27T09:19:00stationS7-flash-calopOP-91feventflash_calresultokfw1.4.2calwrittenboot_ms50unitU0037ts2026-10-27T09:21:00stationS8-functional-packopOP-e58eventtesttestT-OPT-01cameraCAM-2uniformity0.236limit0.05resultfailunit_temp_c20.4unitU0037ts2026-10-27T09:22:00stationS8-functional-packopOP-91feventtesttestT-TCH-01nestN2baseline_drift12limit30resultpassunitU0037ts2026-10-27T09:23:00stationS8-functional-packopOP-7c2eventtesttestT-AUD-01snr_db25.8limit18resultpassunitU0037ts2026-10-27T09:24:00stationS8-functional-packopOP-2d9eventtesttestT-PWR-01rail_mv3306resultpassunitU0037ts2026-10-27T10:09:00stationS8-functional-packopOP-3a1eventretesttestT-OPT-01cameraCAM-2uniformity0.204limit0.05resultfailunit_temp_c22unitU0046ts2026-10-27T09:45:00stationS1-shellopOP-e58eventassembleresultokambient_c18.9unitU0046ts2026-10-27T09:49:00stationS2-pcbaopOP-91feventplaceresultokdriver_reelDRV-A-R1unitU0046ts2026-10-27T09:52:00stationS3-screwsopOP-7c2eventscrewdriveresultoktorque_ncm28.2unitU0046ts2026-10-27T09:55:00stationS4-speaker-hapticopOP-e58eventplaceresultokunitU0046ts2026-10-27T09:57:00stationS5-batteryopOP-e58eventfitresultokunitU0046ts2026-10-27T10:01:00stationS6-dome-bondopOP-b04eventbondresultokfixtureF2dome_offset_mm0.032led_lotL-2604

T-OPT-01 failure rate, split by code

driver vendor
DRV-A4%11/289
DRV-B21%39/190
led lot
L-26047%15/209
L-261113%35/270
s6 fixture
F14%9/239
F217%41/240
shift
AM17%41/240
PM4%9/239
camera
CAM-18%19/244
CAM-213%31/235
start temp band
<12C63%38/60
>=18C3%12/419
34 of 50

first-pass failures passed on retest. A warm retest hides the bug, so the yield report looks better than the line.

  1. material

    LED lot L-2604 issued to S7 kitting.

  2. material

    LED lot L-2611 received. IQC sample of 32 passed.

  3. ecn

    ECN-0417: second-source LED driver DRV-B approved; mixed with DRV-A from today.

  4. facility

    HVAC fault on line 2. Morning ambient below spec until repair.

  5. maintenance

    S6 fixture F2 preventive maintenance overdue (scheduled 2026-10-24).

  6. firmware

    S7 flashing fw 1.4.2 (unchanged since EVT1).

03The loopREPLAY · RECORDED 2026-11-09

Scores suggest.
Evidence decides.

Muse Wonder proposes the causes. SubQ Snap 1.1 scores them and picks the cheapest test that splits them. The finding goes back in. Gemini 4 Argon tries to break the answer blind. Repeat until one cause holds.

  1. ProposeMuse Wonder
  2. ScoreSubQ Snap 1.1
  3. Pick a probeSubQ Snap 1.1
  4. FA labPhysical finding
  5. Judge the findingSubQ Snap 1.1
  6. ChallengeGemini 4 Argon
Scored by SubQ Snap 1.1 on the whole build
Faulty new driver chip44
Lamps tested too cold96
Worn assembly tool41
Bad batch of LEDs2
Test station misreads99
Firmware doesn't suit the new chip0
Firmware timing bug0

Next test · scraps nothing · 3 h

Chill a failed lamp, watch the chip start up

Booted at 8 C with a logic analyser on the LED driver. Driver ready at 188 ms. Firmware calibration write at 50 ms: NACK. Uniformity 0.199.

Every round, in full

OwnerMaterialtriage · issues list (first blame)Test Stationround 0 · subq-snap-1.1Firmwareround 2 · subq-snap-1.1
Candidate causesScore
SupplierSupplier action
ProcessProcess change
AssemblyFixture fix
MaterialMaterial action
Test StationTest limit
FirmwareFirmware fix

Bars are scores from the scorer, not probabilities. The tick at 0.85 is the confirm threshold. Tap a cause for its evidence.

Probes on the tableCost
  • TeardownTear down U0141, the worst dome (uniformity 0.244)1 scrapped6 h
  • X-rayX-ray the dome bond on U01410 scrapped2 h
  • TeardownTear down U0016, failed again on retest1 scrapped6 h
  • X-rayX-ray the dome bond on U0016, failed again on retest0 scrapped2 h
  • Cold retestChill U0161 to 8 C and retest with a logic analyser on the LED driver0 scrapped3 h
  • Warm retestRetest U0161 at 22 C0 scrapped1 h
  • Boot logPull the UART boot log from U0161 at room temperature0 scrapped2 h
  • TeardownTear down U0161, passed on retest1 scrapped6 h
  • Bench testBench-test 30 LEDs from lot L-2611 for intensity and colour0 scrapped4 h
  • Fixture swapBuild 20 morning units on fixture F1 instead of F20 scrapped5 h
  • Camera checkMeasure the golden dome on CAM-1 and CAM-20 scrapped1 h

Why this oneTests H6; scraps 0 unit, 3 h.

FA-001Finding from the FA lab

Booted at 8 C with a logic analyser on the LED driver. Driver ready at 188 ms. Firmware calibration write at 50 ms: NACK. Uniformity 0.199.

temp c
8
driver ready ms
188
cal write at ms
50
cal write
NACK
uniformity
0.199
H2 supportsH5 supportsH6 supportsH7 supports
  • call-002scoreSubquadratic · subq-snap-1.11.1 s~231K tokens in
  • call-003probe pickSubquadratic · subq-snap-1.11.1 s~232K tokens in
  • call-004finding judgeSubquadratic · subq-snap-1.10.3 s~1K tokens in
1/6probes used
0/2units scrapped
12model calls
213.3smodel time
  1. open
  2. triage
  3. investigating
  4. cause confirmed
  5. ca in progress
  6. ca verified
  7. monitoring
04The fixREPLAY · RECORDED 2026-11-09

A fix is code.
Code can be checked.

Once H7 holds, Muse Wonder writes the firmware patch. Compiled C judges it. A failed check sends it back.

The patch model saw

  • confirmed root-cause claim
  • supporting FA findings
  • shipped firmware.c (comments removed)
  • hal.h API

It never saw

  • the fix
  • the sealed truth
  • the gate's test cases
  • the hypothesis model's reasoning
  • the held-out builds
--- firmware.c (shipped)+++ firmware.c (attempt 2)@@ -3,10 +3,28 @@ #define CAL_TARGET 42  int fw_calibrate(void) {-    hal_delay_ms(50); /* fixed boot-time write slot */-    return hal_led_write_cal(CAL_TARGET); /* 1 accepted, 0 dropped */+    int start_ms = hal_time_ms();++    for (;;) {+        if (hal_led_read_cal() == CAL_TARGET) {+            return 1;+        }++        if (hal_led_driver_ready()) {+            if (hal_led_write_cal(CAL_TARGET) &&+                hal_led_read_cal() == CAL_TARGET) {+                return 1;+            }+        }++        if (hal_time_ms() - start_ms >= 1000) {+            return 0;+        }++        hal_delay_ms(1);+    } }  int fw_cal_applied(void) {-    return hal_led_read_cal() >= 0 ? 1 : 0;+    return hal_led_read_cal() == CAL_TARGET ? 1 : 0; } 

Gate · compiled C, run by code

  • PASSwarm unit passesuniformity 0.02, pass, cal_applied=1, 20 ms
  • PASScold second-source unit passes (ready 180 ms)uniformity 0.02, pass, cal_applied=1, 180 ms
  • PASSslower cold unit passes (ready 260 ms)uniformity 0.02, pass, cal_applied=1, 260 ms
  • PASSknown-bad dome still failsuniformity 0.22, fail, cal_applied=1, 20 ms
  • PASSdead driver: gives up within 1000 ms and reports failurereturned after 1000 ms virtual, rc=0, cal_applied=0
  • PASSdeterministic rerunidentical output

Accepted. Every check passed.

Attempt 2 passed every check. The gate rejected 1 before it. Code made that call, not a model.

The bug, in time and in parts

0 ms50 ms100 ms150 ms200 ms250 mscold driver ready · 188 mswarm · 20 msAs shippedwrites at a fixed 50 mswrite · NACK · lostWith the fixwaits, writes, reads backwrite · ACK · read backDIMEVEN
Same cold unit, same driver. Only the firmware changed.

Drag to rotate

Cold fault: the driver needs 180 ms, firmware writes at 50 ms. Calibration drops. One sector stays dim. The parts test fine.

StationPartSpec (fictional)
S1-shellTouch capBC-110 · PC, soft-touch
S6-dome-bondDomeBC-160 · frosted PC, 1.2 mm
S6-dome-bondDiffuser ringBC-161 · opal PMMA
S7-flash-calLED ringBC-170 · 12 RGB LEDs + driver
S3-screwsScrews ×4BC-130 · M2×5 PT
S2-pcbaMain PCBABC-120 · SoC, IMU, mics
S4-speaker-hapticSpeaker + hapticBC-140 · 20 mm, ERM
S5-batteryBatteryBC-150 · Li-po 1200 mAh
S1-shellBase housingBC-111 · PC/ABS

Exploded beacon and timing are drawn from the fictional product. All synthetic.

05The shortcutLIVE C + ENGINE

Widen the limit.
Watch a bad dome pass.

Widen the T-OPT-01 limit from 0.05 to 0.24 and yield jumps. 15 units with a real defect ship too. The gate blocks any change that frees a known-bad unit.

0.02Warm control0.18Cold start, as shipped0.22Known-bad domelimit 0.05widened 0.24
Blocked. At 0.24 the known-bad dome reads 0.22 and passes. The limit stays at 0.05.
06The proofLIVE C

The next build
is the judge.

Run both firmware builds yourself. Every result is compiled C on the server. No model decides a pass.

320held-out units across 20 fresh builds
113 → 0cold units failing, before and after
15/15known-bad domes still caught
0warm units broken by the fix

113cold lamps failing

cold lamp fails truly broken, must stay red passesOne square per lamp. Compiled C, run when this page was built.

THE VIRTUAL TEST BENCH

Cold Dome

S8 · FUNCTIONAL TEST
As shippedPREVIEW
READYUniformity error —

Run the test to measure this unit.

After correctionWAITING
NOT SHOWNUniformity error —

A passing result must come from running the correction.

Start with a measured failure.

07The rules

No agent can
talk an issue shut.

Built with

SubquadraticMeta Model APIGoogle DeepMind