Problem

DrivenData's DaT Parkinson's Challenge asked for one number per 3D DaT-SPECT brain scan: the probability that the scan is abnormal. Entries were scored by log loss. It measures how far the predicted probabilities were from the truth; lower is better. So the model had to be right about how sure it was, not only about which scans look abnormal.

The data was 1,362 scans from ten French hospitals, labelled by 11 nuclear medicine specialists. Scans came in 21 acquisition families with different sizes and resolutions. In the widest, the head filled about 1% of the image[src].

Approach

Model architecture: two views of each scan, stacked into one calibrated probability
1 · Data
DaT-SPECT brain scans
Labelled normal or abnormal
  • n = 1,362 (747 abnormal, 615 normal)
  • 10 French hospitals
  • 21 acquisition families
  • FOV 176–630 mm per axis
→ 3D volume, varying shape and spacing
2 · Preprocess
Standardise
Same geometry and intensity scale for every scan
  • Reorient to RAS
  • Resample to 2.0 mm isotropic
  • Centre-of-mass crop 96 × 96 × 96
  • Reference-region intensity normalisation
→ 1 × 96³ tensor
3 · Two models in parallel
3D ResNet ensemble
10 CNNs: 2 training runs × 5 stratified folds
  • GroupNorm, widths 16/32/64/128, dropout 0.3
  • AdamW lr 3e-4, cosine, 30 epochs, early stopping
  • Run 2 augmentation: elastic, PSF blur, signed gamma, count noise, mixup
  • No left–right flip: asymmetry is signal
  • TTA: 9 laterality-preserving views, logit mean
AUROC 0.9393ECE 0.0187log loss 0.3021
Striatal features → LightGBM
157 hand-built SBR-style striatal features
  • Logit correlation with the CNN ensemble ρ = 0.821
AUROC 0.9184ECE 0.0329log loss 0.3659
→ 2 logits per scan
4 · Combine
Logistic stack
3 parameters, scored by cross-validation
  • CNN +0.694, trees +0.364, intercept +0.048
  • Carries the calibration: no separate temperature
  • Clip to [1e-6, 1 − 1e-6]
→ p(abnormal)
5 · Output
Probability the scan is abnormal
Final model (candidate B), out of fold
AUROC 0.9455
All metrics are 5-fold stratified out-of-fold (OOF) on n = 1,362. AUROC: how well the model ranks abnormal above normal scans (1 = perfect). ECE: expected calibration error, the average gap between predicted probability and observed rate (0 = perfect).
  • Crop around the brain, not the image. A fixed centre crop cut off the striatum, the region that matters, in wide scans. Each scan is cropped around its centre of mass instead.
  • Never flip left and right. Abnormality is often one-sided, and mirroring a scan would blur that signal.
  • Two different model families. The tree model on hand-built striatal measurements is weaker alone. But its outputs correlate only 0.821 with the CNN's, so it is wrong on different scans. It carries 34% of the weight in the final stack, and adding it cut log loss by 0.0184 at 3.3 sigma[src].
  • Let the stack do the calibration. The stack's CNN weight sits below 1, which works like temperature scaling. So there is no separate calibration step to fit and maintain.
  • Choose calibrators out of sample. Early on, isotonic regression was picked because it had been scored on its own training predictions. Under cross-validation it did worse than no calibration: 0.3520 against 0.3271 log loss[src].

Result

Baseline model: strong ranking on most acquisition families, weakest on the small 128 @ 1.5 mm family
All scans, baseline 0.9302
AUROC (higher is better)
0.800.850.900.951.00
256 @ 2.5 mmn = 460 · ECE 0.0509
0.9176
128 @ 3.9 mmn = 325 · ECE 0.0544
0.9277
128 @ 2.3 mmn = 207 · ECE 0.0397
0.9396
128 @ 2.4 mmn = 149 · ECE 0.0407
0.9560
Small families, pooledn = 125 · ECE 0.0676
0.9097
var @ 2.5 mmn = 66 · ECE 0.0930
0.9463
128 @ 1.5 mmn = 30 · ECE 0.1552
0.8373
Run 1 baseline, stratified OOF, so every family was seen in training. Families are named as recorded: 21 in total, the 6 with n ≥ 30 shown and the rest pooled. Per-family results for the final model are not recorded.

The final model ranks scans well: out-of-fold AUROC 0.9455[src]. AUROC is the chance that a random abnormal scan scores above a random normal one; 1 is perfect. The CNN ensemble's probabilities are close to honest. ECE, the average gap between predicted probability and the rate actually observed, is 0.0187 for the CNN ensemble. For the baseline, calibrating each family with a temperature fitted only on the other families cost just +0.0006 log loss overall, so the calibration carries over to acquisitions it was not fitted on[src].

The weak spots are the small families. The 30-scan 1.5 mm family and the pooled small families have the lowest AUROC and the highest ECE, so a probability on an uncommon acquisition deserves less trust. The per-family breakdown exists only for the baseline, and every family was seen in training[src]. On the leaderboard my entry finished #159 of 1009 (top 16%), with private scores worse than public by a similar margin for every build[src].

What I learned

  • Diversity beats quality in a stack. The trees and the second CNN run were each weaker alone, yet they gave the largest gains, because their errors fell in different places.
  • Pooled calibration hides the groups that matter. The baseline looked well calibrated on all scans together, yet its ECE on the 30-scan 1.5 mm family was 0.1552[src]. I only broke results down by family for the baseline, so I cannot say whether the final model closes that gap.

Tools

Python, PyTorch, LightGBM, scikit-learn, NumPy, pandas, SciPy, pytest, Docker, Claude Code. The assistant never saw competition data. It wrote scripts and received only aggregate summaries.

Sources

Each [src] marker cites one of these project records:

  • DaT Parkinson's Challenge (DrivenData): 3D DaT-SPECT normal vs abnormal