ABCDNet · robustness experiment · holdout cohort · 2026-08-12

Cleaning robustness on the holdout subjects

The clean-vs-uncleaned comparison restricted to subject IDs 003, 006, 007, 013, and 020 — the numeric IDs held out of ABCDNet's training split. Matching both control and patient prefixes gives 8 subjects (TCRc007, TCRc013, TCRc020, TCRp003, TCRp006, TCRp007, TCRp013, TCRp020): 46 paired recordings, 874 paired channel-spectra.

What the numbers say

The holdout cohort behaves like the full cohort, slightly noisier. Per-spectrum agreement is 75.3% (full cohort: 77.1%) with Cohen's κ = 0.659, and probability vectors move a bit more (mean TV distance 0.221 vs. 0.187), driven by the same D↔Z traffic: of the 874 paired channel-spectra, D→Z (72) and Z→D (17) dominate the off-diagonal.

Averaging over the 19 channels of each recording, agreement is 80.4% (κ = 0.721), with 9 of 46 recordings flipping — the wide CI (59.0–93.8%) reflects that flips concentrate in a few subjects: TCRp020 contributes four of the nine (all toward A), and TCRc013 two. Median probability mass again moves from D (Δp +0.040) toward Z (Δp −0.057) on uncleaned data, Wilcoxon p ≈ 10⁻⁶ and 10⁻⁵.

At the subject level, 6 of 8 subjects keep their label. Both flips are near-tied calls: TCRp003 goes C (0.344) → Z (0.353), and TCRp020 — the least stable subject in the whole dataset, with only 38.2% per-spectrum agreement — goes B (0.372) → A (0.425). Mean subject-level TV distance is 0.113 (max 0.279).

Per-spectrum confusion matrix — cleaned (rows) vs. uncleaned (columns)

Mean |Δp| per class: A 0.079, B 0.044, C 0.054, D 0.115, Z 0.149. Per-spectrum Wilcoxon signed-rank is significant for all classes except B (median shifts ≤ 0.016) — the systematic but small D→Z drift under uncleaned data.

Per-recording results — probabilities averaged over the 19 channels

Agreement 80.4%, κ = 0.721, mean TV distance 0.140 (median 0.109, max 0.431). The nine flipped recordings:

Continuous A–D scale — per recording, channels averaged, Z ignored

Treating the classes as an ordered scale (A=1, B=2, C=3, D=4), each recording is scored as the expected value over its channel-averaged probabilities with Z dropped and the remaining A–D mass renormalized. Clean and uncleaned scores track closely: Pearson r = 0.952 (Spearman ρ = 0.901), mean |Δscore| = 0.196 (median 0.066, 90th pct 0.558, max 1.541) on the 1–4 scale. The shift is directional — cleaned scores run higher by a mean of +0.179 (Wilcoxon p = 3.2×10⁻⁵), i.e. artifact removal pushes recordings up-scale toward D. Restricted to the 36 recordings labeled non-Z in both versions: mean |Δscore| 0.167, r = 0.978.

After dropping Z, the retained A–D mass averages 0.76 (clean) / 0.70 (uncleaned); scores for Z-dominated recordings rest on little mass — the largest mover, TCRp003_L_BaselineRest (3.96 → 2.42), is labeled Z in both versions. Filter with the abcd_mass_* columns in per_recording.csv.

Per-subject classifications

Probabilities averaged over all of a subject's paired spectra, then argmax. Flipped rows are marked.

Raw outputs for this cohort: experiments/results_holdout/ — per_spectrum.csv, per_recording.csv, per_subject.csv, summary.md. Generated with experiments/clean_vs_uncleaned.py --subjects 006,003,007,013,020 --out experiments/results_holdout.