SPECTER FLEET · LIVE REMOTE

Canonical report · updated every hour · dual SpecterS1Watch / SpecterS2Watch

mDNS + Specter viability — 60‑minute fleet soak

Run start 20260720T203930Z · interval 3600s · planned samples 24 · proxy http://127.0.0.1:8788 · auto-refresh 120s · out shadow-dual-soak-24h-pages-20260720

8
S1 good
8
S2 good
8
Fleet worst
Fleet average · sample 12/24
8.0/10
Worst lane: s1
S1 die 86.0°C · S2 die 83.2°C
Specter grade 92 · {'s1': 'S1 86° · 43% · WARN', 's2': 'S2 84° · 44% · WARN'}
MQTT
connected
Pulse
healthy
Issues
['s1:thermal_warn', 's2:thermal_warn']
Latest hourly diff
S1 overall 7→8 (↑1); S1.specter 5→6; S1.thermal 2→4; S1 die 94.7°C→86.0°C (Δ-8.7); Specter issues +['s1:thermal_warn']; Specter issues −['s1:thermal_critical']; Fleet worst 7→8

Hourly history + diffs

Each row is one 60‑minute sample. Diffs compare grade dimensions, die temp (≥1°C), repair focus, and Specter issues vs the prior hour.

#UTCS1 grade / tempS2 grade / tempFleetDiff vs previous hour
12026-07-20T03:41:12Z8/10 good
87.2°C
8/10 good
81.6°C
8/10Baseline sample (no prior hour).
22026-07-20T04:41:17Z7/10 good
94.7°C
8/10 good
81.6°C
7/10
  • S1 overall 8→7 (↓1)
  • S1.specter 6→5
  • S1.thermal 4→2
  • S1 die 87.2°C→94.7°C (Δ+7.5)
  • Specter issues +['s1:thermal_critical']
  • Specter issues −['s1:thermal_warn']
  • Fleet worst 8→7
32026-07-20T05:41:24Z7/10 good
95.1°C
9/10 great
77.2°C
7/10
  • S1.thermal 2→1
  • S1 thermal primary: VIDEO_FIFO write timeouts (encode path pressure) → Severe SoC heat (≥95°C)
  • S2 overall 8→9 (↑1)
  • S2.specter 6→9
  • S2.thermal 6→9
  • S2 die 81.6°C→77.2°C (Δ-4.4)
  • S2 cleared repairs: specter, thermal
  • Specter issues −['s2:thermal_warn']
42026-07-20T11:54:34Z8/10 good
84.1°C
8/10 good
84.8°C
8/10
  • S1 overall 7→8 (↑1)
  • S1.specter 5→6
  • S1.thermal 1→6
  • S1 die 95.1°C→84.1°C (Δ-11.0)
  • S1 thermal primary: Severe SoC heat (≥95°C) → VIDEO_FIFO write timeouts (encode path pressure)
  • S2 overall 9→8 (↓1)
  • S2.specter 9→6
  • S2.storage 10→8
  • S2.thermal 9→6
  • S2 die 77.2°C→84.8°C (Δ+7.6)
  • S2 new repairs: specter, thermal
  • Specter issues +['s1:mqtt_stale', 's1:thermal_warn', 's2:mqtt_stale', 's2:thermal_warn']
  • Specter issues −['s1:thermal_critical']
  • Fleet worst 7→8
52026-07-20T14:24:22Z7/10 good
95.5°C
7/10 good
86.0°C
7/10
  • S1 overall 8→7 (↓1)
  • S1.specter 6→5
  • S1.storage 10→8
  • S1.thermal 6→1
  • S1 die 84.1°C→95.5°C (Δ+11.4)
  • S1 thermal primary: VIDEO_FIFO write timeouts (encode path pressure) → Severe SoC heat (≥95°C)
  • S2 overall 8→7 (↓1)
  • S2.thermal 6→4
  • S2 die 84.8°C→86.0°C (Δ+1.2)
  • Specter issues +['s1:thermal_critical']
  • Specter issues −['s1:mqtt_stale', 's1:thermal_warn', 's2:mqtt_stale']
  • Fleet worst 8→7
62026-07-20T16:26:16Z7/10 good
94.7°C
8/10 good
83.6°C
7/10
  • S1.storage 8→10
  • S1.thermal 1→2
  • S1 thermal primary: Severe SoC heat (≥95°C) → VIDEO_FIFO write timeouts (encode path pressure)
  • S2 overall 7→8 (↑1)
  • S2.storage 8→10
  • S2.thermal 4→6
  • S2 die 86.0°C→83.6°C (Δ-2.4)
72026-07-20T17:31:23Z7/10 good
94.7°C
8/10 good
84.8°C
7/10
  • S2.reachability 10→8
  • S2 die 83.6°C→84.8°C (Δ+1.2)
82026-07-20T18:31:30Z8/10 good
87.2°C
8/10 good
84.8°C
8/10
  • S1 overall 7→8 (↑1)
  • S1.specter 5→6
  • S1.thermal 2→4
  • S1 die 94.7°C→87.2°C (Δ-7.5)
  • S2.reachability 8→10
  • Specter issues +['s1:thermal_warn']
  • Specter issues −['s1:thermal_critical']
  • Fleet worst 7→8
92026-07-20T20:37:20Z7/10 good
94.7°C
9/10 great
75.6°C
7/10
  • S1 overall 8→7 (↓1)
  • S1.specter 6→5
  • S1.thermal 4→2
  • S1 die 87.2°C→94.7°C (Δ+7.5)
  • S2 overall 8→9 (↑1)
  • S2.thermal 6→9
  • S2 die 84.8°C→75.6°C (Δ-9.2)
  • S2 cleared repairs: thermal
  • Specter issues +['s1:thermal_critical']
  • Specter issues −['s1:thermal_warn']
  • Fleet worst 8→7
12026-07-20T20:39:14Z7/10 good
94.7°C
8/10 good
82.8°C
7/10
  • S2 overall 9→8 (↓1)
  • S2.thermal 9→6
  • S2 die 75.6°C→82.8°C (Δ+7.2)
  • S2 new repairs: thermal
12026-07-20T20:39:35Z7/10 good
94.7°C
8/10 good
82.8°C
7/10No material grade/thermal/issue diffs vs previous hour.
22026-07-20T21:39:49Z8/10 good
86.0°C
8/10 good
83.2°C
8/10
  • S1 overall 7→8 (↑1)
  • S1.specter 5→6
  • S1.thermal 2→4
  • S1 die 94.7°C→86.0°C (Δ-8.7)
  • Specter issues +['s1:thermal_warn']
  • Specter issues −['s1:thermal_critical']
  • Fleet worst 7→8

Latest grading checklist (1–10)

S1 · SpecterS1Watch · 8/10

DimensionScore 1–10Evidence
reachability10/10MQTT age 11.3s + SSH OK
thermal4/10die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784583545_169.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 39.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s1:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=86°C skin=62.7°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
SoC warn band (78–90°C)0.45die=86°C
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3526 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (4/10)
die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s1:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · SpecterS2Watch · 8/10

DimensionScore 1–10Evidence
reachability10/10MQTT age 8.4s + SSH OK
thermal6/10die 83.2°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784583519_664.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 44.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=83.2°C skin=50.8°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=83.2°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3526 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 83.2°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #2 · 2026-07-20T21:39:49Z

Specter chips: {'s1': 'S1 86° · 43% · WARN', 's2': 'S2 84° · 44% · WARN'} · issues: ['s1:thermal_warn', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 7→8 (↑1); S1.specter 5→6; S1.thermal 2→4; S1 die 94.7°C→86.0°C (Δ-8.7); Specter issues +['s1:thermal_warn']; Specter issues −['s1:thermal_critical']; Fleet worst 7→8

S1 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 11.3s + SSH OK
thermal4/10die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784583545_169.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 39.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s1:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=86°C skin=62.7°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
SoC warn band (78–90°C)0.45die=86°C
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3526 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (4/10)
die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s1:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 8.4s + SSH OK
thermal6/10die 83.2°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784583519_664.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 44.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=83.2°C skin=50.8°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=83.2°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3526 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 83.2°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #1 · 2026-07-20T20:39:35Z

Specter chips: {'s1': 'S1 95° · 47% · CRITICAL', 's2': 'S2 83° · 47% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: No material grade/thermal/issue diffs vs previous hour.

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 4.3s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579949_486.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 44.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=62.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=62.6°C Δ=32.1°C — package retains heat under continuous NPU/encode
SoC critical band (90–95°C)0.85die=94.7°C
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈15.19h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 22.8s + SSH OK
thermal6/10die 82.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579917_539.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 47.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=82.8°C skin=50.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Elevated 1-min load average; SoC warn band (78–90°C).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Elevated 1-min load average0.55load1=2.54 load5=2.44 — host/container CPU busy
SoC warn band (78–90°C)0.45die=82.8°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=4 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35144 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 82.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Elevated 1-min load average; SoC warn band (78–90°C).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #1 · 2026-07-20T20:39:14Z

Specter chips: {'s1': 'S1 95° · 47% · CRITICAL', 's2': 'S2 83° · 47% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S2 overall 9→8 (↓1); S2.thermal 9→6; S2 die 75.6°C→82.8°C (Δ+7.2); S2 new repairs: thermal

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 12.6s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579889_328.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 44.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=62.7°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=62.7°C Δ=32.0°C — package retains heat under continuous NPU/encode
SoC critical band (90–95°C)0.85die=94.7°C
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈15.19h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 0.3s + SSH OK
thermal6/10die 82.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579917_539.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 47.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=82.8°C skin=50.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Elevated 1-min load average; SoC warn band (78–90°C).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Elevated 1-min load average0.55load1=2.95 load5=2.5 — host/container CPU busy
SoC warn band (78–90°C)0.45die=82.8°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35144 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 82.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Elevated 1-min load average; SoC warn band (78–90°C).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #9 · 2026-07-20T20:37:20Z

Specter chips: {'s1': 'S1 95° · 47% · CRITICAL', 's2': 'S2 79° · 47% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 8→7 (↓1); S1.specter 6→5; S1.thermal 4→2; S1 die 87.2°C→94.7°C (Δ+7.5); S2 overall 8→9 (↑1); S2.thermal 6→9; S2 die 84.8°C→75.6°C (Δ-9.2); S2 cleared repairs: thermal; Specter issues +['s1:thermal_critical']; Specter issues −['s1:thermal_warn']; Fleet worst 8→7

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 20.1s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579757_216.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 44.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=62.4°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Recorder coarse_mode engaged (thermal protection path).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9517 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=62.4°C Δ=32.3°C — package retains heat under continuous NPU/encode
SoC critical band (90–95°C)0.85die=94.7°C
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.73 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈15.16h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Recorder coarse_mode engaged (thermal protection path).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 9/10 great

DimensionScore 1–10Evidence
reachability10/10MQTT age 8.3s + SSH OK
thermal9/10die 75.6°C OK
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784579792_628.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 47.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=75.6°C skin=50.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Elevated 1-min load average; Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Elevated 1-min load average0.55load1=2.97 load5=2.45 — host/container CPU busy
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35144 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #8 · 2026-07-20T18:31:30Z

Specter chips: {'s1': 'S1 87° · 40% · WARN', 's2': 'S2 85° · 40% · WARN'} · issues: ['s1:thermal_warn', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 7→8 (↑1); S1.specter 5→6; S1.thermal 2→4; S1 die 94.7°C→87.2°C (Δ-7.5); S2.reachability 8→10; Specter issues +['s1:thermal_warn']; Specter issues −['s1:thermal_critical']; Fleet worst 7→8

S1 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 18.1s + SSH OK
thermal4/10die 87.2°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784572219_536.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 37.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s1:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=87.2°C skin=64.0°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9517 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.73 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
SoC warn band (78–90°C)0.45die=87.2°C
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35187 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (4/10)
die 87.2°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s1:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 13.5s + SSH OK
thermal6/10die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784572254_407.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 40.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=84.8°C skin=52.1°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); SoC warn band (78–90°C).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9517 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.73 CRITICAL temp lines in recent log — sustained overheat, not transient spike
SoC warn band (78–90°C)0.45die=84.8°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35125 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); SoC warn band (78–90°C).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #7 · 2026-07-20T17:31:23Z

Specter chips: {'s1': 'S1 95° · 38% · CRITICAL', 's2': 'S2 86° · 37% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S2.reachability 10→8; S2 die 83.6°C→84.8°C (Δ+1.2)

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 13.7s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784568629_369.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 34.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=63.5°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Recorder coarse_mode engaged (thermal protection path).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9517 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=63.5°C Δ=31.2°C — package retains heat under continuous NPU/encode
SoC critical band (90–95°C)0.85die=94.7°C
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.73 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈12.06h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Recorder coarse_mode engaged (thermal protection path).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability8/10MQTT age 30.6s + SSH OK
thermal6/10die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 19 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784568650_669.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 37.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=84.8°C skin=52.1°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 19 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9519 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=84.8°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35113 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 19 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #6 · 2026-07-20T16:26:16Z

Specter chips: {'s1': 'S1 95° · 35% · CRITICAL', 's2': 'S2 84° · 34% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1.storage 8→10; S1.thermal 1→2; S1 thermal primary: Severe SoC heat (≥95°C) → VIDEO_FIFO write timeouts (encode path pressure); S2 overall 7→8 (↑1); S2.storage 8→10; S2.thermal 4→6; S2 die 86.0°C→83.6°C (Δ-2.4)

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 18.0s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784564706_101.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 32.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=62.4°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=62.4°C Δ=32.3°C — package retains heat under continuous NPU/encode
SoC critical band (90–95°C)0.85die=94.7°C
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈10.97h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 16 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); SoC critical band (90–95°C); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 24.3s + SSH OK
thermal6/10die 83.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784564736_760.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 34.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=83.6°C skin=51.0°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=83.6°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35101 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 83.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #5 · 2026-07-20T14:24:22Z

Specter chips: {'s1': 'S1 96° · 76% · CRITICAL', 's2': 'S2 85° · 78% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 8→7 (↓1); S1.specter 6→5; S1.storage 10→8; S1.thermal 6→1; S1 die 84.1°C→95.5°C (Δ+11.4); S1 thermal primary: VIDEO_FIFO write timeouts (encode path pressure) → Severe SoC heat (≥95°C); S2 overall 8→7 (↓1); S2.thermal 6→4; S2 die 84.8°C→86.0°C (Δ+1.2); Specter issues +['s1:thermal_critical']; Specter issues −['s1:mqtt_stale', 's1:thermal_warn', 's2:mqtt_stale']; Fleet worst 8→7

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 0.7s + SSH OK
thermal1/10die 95.5°C severe CRITICAL | CAUSE: Primary cause: Severe SoC heat (≥95°C) — die=95.5°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRIT
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784557425_437.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage8/10used 75.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: Severe SoC heat (≥95°C) · class=compute_encode_npu_heat_soak · die=95.5°C skin=65.3°C

Primary cause: Severe SoC heat (≥95°C) — die=95.5°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
Severe SoC heat (≥95°C)1.0die=95.5°C
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=95.5°C skin=65.3°C Δ=30.2°C — package retains heat under continuous NPU/encode
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Long uptime under load (heat accumulation)0.5uptime≈8.94h with die≥90°C — heat soak after multi-hour continuous run

Resolution / repair plans (score ≤ 6)

thermal (1/10)
die 95.5°C severe CRITICAL | CAUSE: Primary cause: Severe SoC heat (≥95°C) — die=95.5°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 21.9s + SSH OK
thermal4/10die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 18 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784557396_638.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage8/10used 78.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=86°C skin=52.8°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 18 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9518 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.62 CRITICAL temp lines in recent log — sustained overheat, not transient spike
SoC warn band (78–90°C)0.45die=86°C
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3578 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (4/10)
die 86.0°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 18 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #4 · 2026-07-20T11:54:34Z

Specter chips: {'s1': 'S1 84° · 70% · WARN', 's2': 'S2 84° · 71% · WARN'} · issues: ['s1:thermal_warn', 's1:mqtt_stale', 's2:thermal_warn', 's2:mqtt_stale'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 7→8 (↑1); S1.specter 5→6; S1.thermal 1→6; S1 die 95.1°C→84.1°C (Δ-11.0); S1 thermal primary: Severe SoC heat (≥95°C) → VIDEO_FIFO write timeouts (encode path pressure); S2 overall 9→8 (↓1); S2.specter 9→6; S2.storage 10→8; S2.thermal 9→6; S2 die 77.2°C→84.8°C (Δ+7.6); S2 new repairs: specter, thermal; Specter issues +['s1:mqtt_stale', 's1:thermal_warn', 's2:mqtt_stale', 's2:thermal_warn']; Specter issues −['s1:thermal_critical']; Fleet worst 7→8

S1 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 3.2s + SSH OK
thermal6/10die 84.1°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784548425_998.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 68.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s1:thermal_warn', 's1:mqtt_stale']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=84.1°C skin=64.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); Elevated 1-min load average.

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9517 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.73 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Elevated 1-min load average0.55load1=2.52 load5=2.16 — host/container CPU busy
SoC warn band (78–90°C)0.45die=84.1°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.35112 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 84.1°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 17 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Repeated CRITICAL SoC log spam (sustained, not one-shot); Elevated 1-min load average.
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s1:thermal_warn', 's1:mqtt_stale']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 1.9s + SSH OK
thermal6/10die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784548430_217.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage8/10used 71.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn', 's2:mqtt_stale']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=84.8°C skin=51.8°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=84.8°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3548 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 84.8°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn', 's2:mqtt_stale']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #3 · 2026-07-20T05:41:24Z

Specter chips: {'s1': 'S1 95° · 52% · CRITICAL', 's2': 'S2 77° · 51% · OK'} · issues: ['s1:thermal_critical'] · mqtt=True pulse=healthy

Hourly diff: S1.thermal 2→1; S1 thermal primary: VIDEO_FIFO write timeouts (encode path pressure) → Severe SoC heat (≥95°C); S2 overall 8→9 (↑1); S2.specter 6→9; S2.thermal 6→9; S2 die 81.6°C→77.2°C (Δ-4.4); S2 cleared repairs: specter, thermal; Specter issues −['s2:thermal_warn']

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 16.4s + SSH OK
thermal1/10die 95.1°C severe CRITICAL | CAUSE: Primary cause: Severe SoC heat (≥95°C) — die=95.1°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRIT
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784526012_764.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 49.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: Severe SoC heat (≥95°C) · class=compute_encode_npu_heat_soak · die=95.1°C skin=59.2°C

Primary cause: Severe SoC heat (≥95°C) — die=95.1°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot).

CauseWeightDetail
Severe SoC heat (≥95°C)1.0die=95.1°C
VIDEO_FIFO write timeouts (encode path pressure)0.9516 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=95.1°C skin=59.2°C Δ=35.9°C — package retains heat under continuous NPU/encode
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.84 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3560 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure

Resolution / repair plans (score ≤ 6)

thermal (1/10)
die 95.1°C severe CRITICAL | CAUSE: Primary cause: Severe SoC heat (≥95°C) — die=95.1°C. Class=compute_encode_npu_heat_soak. Also: VIDEO_FIFO write timeouts (encode path pressure); Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 9/10 great

DimensionScore 1–10Evidence
reachability10/10MQTT age 17.4s + SSH OK
thermal9/10die 77.2°C OK
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784526011_207.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 51.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter9/10no specter issues for lane

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=77.2°C skin=46.7°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance); Many pre_event_thermal dumps (disk I/O + prior heat events).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3523 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

No dimensions ≤ 6 — no urgent repairs this hour.

Hour #2 · 2026-07-20T04:41:17Z

Specter chips: {'s1': 'S1 95° · 49% · CRITICAL', 's2': 'S2 82° · 48% · WARN'} · issues: ['s1:thermal_critical', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: S1 overall 8→7 (↓1); S1.specter 6→5; S1.thermal 4→2; S1 die 87.2°C→94.7°C (Δ+7.5); Specter issues +['s1:thermal_critical']; Specter issues −['s1:thermal_warn']; Fleet worst 8→7

S1 · 7/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 18.4s + SSH OK
thermal2/10die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, mo
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784522417_189.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 46.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter5/10specter thermal critical flag ['s1:thermal_critical']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=94.7°C skin=61.8°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot); SoC critical band (90–95°C).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9515 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Internal heat soak (high die, moderate skin)0.9die=94.7°C skin=61.8°C Δ=32.9°C — package retains heat under continuous NPU/encode
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.95 CRITICAL temp lines in recent log — sustained overheat, not transient spike
SoC critical band (90–95°C)0.85die=94.7°C
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=4 — steady encode load
RF-DETR / NPU inference path active alongside encode0.55rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3549 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure

Resolution / repair plans (score ≤ 6)

thermal (2/10)
die 94.7°C CRITICAL | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Internal heat soak (high die, moderate skin); Repeated CRITICAL SoC log spam (sustained, not one-shot); SoC critical band (90–95°C).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. NPU+encode heat stack: keep RF-DETR but avoid extra SSH/model jobs until cooler.
  4. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  5. Heat soak: soft reboot (HITL) after brief idle cool; not PoE-first.
  6. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  7. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
specter (5/10)
specter thermal critical flag ['s1:thermal_critical']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 20.6s + SSH OK
thermal6/10die 81.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784522451_254.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 48.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=81.6°C skin=49.1°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=81.6°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3523 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 81.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

Hour #1 · 2026-07-20T03:41:12Z

Specter chips: {'s1': 'S1 87° · 45% · WARN', 's2': 'S2 82° · 45% · WARN'} · issues: ['s1:thermal_warn', 's2:thermal_warn'] · mqtt=True pulse=healthy

Hourly diff: Baseline sample (no prior hour).

S1 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 14.7s + SSH OK
thermal4/10die 87.2°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784518827_004.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 42.0%
services8/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s1:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=87.2°C skin=62.2°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9515 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Repeated CRITICAL SoC log spam (sustained, not one-shot)0.95 CRITICAL temp lines in recent log — sustained overheat, not transient spike
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
Continuous justified H.265 encode (~30fps surveillance)0.65justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
SoC warn band (78–90°C)0.45die=87.2°C
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3537 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (4/10)
die 87.2°C high WARN | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 15 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Repeated CRITICAL SoC log spam (sustained, not one-shot); Recorder coarse_mode engaged (thermal protection path); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s1:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

S2 · 8/10 good

DimensionScore 1–10Evidence
reachability10/10MQTT age 4.1s + SSH OK
thermal6/10die 81.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (t
recording7/10chunk present (writing flag thin): ngs/2026/07/20/chunk_1784518836_275.h265
rfdetr7/10RF-DETR path live precomp=None n_dets=0 persons=None
storage10/10used 45.0%
services9/10core+agent seen (no-oak-line)
host_mcp10/10health+recording host_mcp OK
specter6/10specter warn ['s2:thermal_warn']

Thermal cause investigation

Primary: VIDEO_FIFO write timeouts (encode path pressure) · class=encode_path_pressure · die=81.6°C skin=49.6°C

Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).

CauseWeightDetail
VIDEO_FIFO write timeouts (encode path pressure)0.9520 FIFO stall lines in recent ghost_core log — frame drops under thermal/load
Recorder coarse_mode engaged (thermal protection path)0.7h265_status coarse_mode=1 — pipeline already throttling under heat policy
SoC warn band (78–90°C)0.45die=81.6°C
Continuous justified H.265 encode (~30fps surveillance)0.35justified_ok=True est_fps=30.0 chunks_5m=5 — steady encode load
Many pre_event_thermal dumps (disk I/O + prior heat events)0.3523 pre_event_thermal_*.h265 files counted — prune if storage/I/O pressure
RF-DETR / NPU inference path active alongside encode0.3rfdetr telemetry present n_dets/persons≈0 — DSP/NPU + encode stacks heat

Resolution / repair plans (score ≤ 6)

thermal (6/10)
die 81.6°C WARN band | CAUSE: Primary cause: VIDEO_FIFO write timeouts (encode path pressure) — 20 FIFO stall lines in recent ghost_core log — frame drops under thermal/load. Class=encode_path_pressure. Also: Recorder coarse_mode engaged (thermal protection path); SoC warn band (78–90°C); Continuous justified H.265 encode (~30fps surveillance).
  1. Review ghost_core VIDEO_FIFO stalls; reduce concurrent host pulls; confirm justified encode rate.
  2. coarse_mode already on — expect reduced quality until die < ~85°C; do not force full-rate.
  3. Continuous 30fps H.265 is a baseline heat source on OAK-4 — plan cool windows for maintenance reboots.
  4. Prune pre_event_thermal_*.h265 dumps to cut disk I/O heat contribution.
  5. Run log hygiene (shadow-log-hygiene.sh) — reduces log I/O heat slightly.
  6. Confirm coarse_mode / VIDEO_FIFO in ghost_core logs (expected under heat).
  7. Avoid stacked heavy pulls; schedule soft reboot in a cool window (HITL).
specter (6/10)
specter warn ['s2:thermal_warn']
  1. Fix SSH targets so Specter probe is not probe_error.
  2. POST /fleet/specter/sync and re-read chips/issues.
  3. Confirm no false offline when SSH true (icmp_unreachable is soft).

JSONL: /Users/deanbarrett/gp-firmware/cost-sentinel/data/shadow-dual-soak-24h-pages-20260720/samples.jsonl · summary: /Users/deanbarrett/gp-firmware/cost-sentinel/data/shadow-dual-soak-24h-pages-20260720/SUMMARY.md · viability JSON: /Users/deanbarrett/gp-firmware/cost-sentinel/data/mdns-specter-viability-20260718T163941Z.hourly.json · scale 1=critical 10=excellent · thermal investigation uses VIDEO_FIFO / coarse_mode / H.265 / RF-DETR / log spam weights