Age prediction with site-effect removal

Link to the challenge

The challenge is available on RAMP.studio. The code of the challenge is available on GitHub RAMP.kits.

Contents

1. Challenge description
2. Ranking of the best models
3. Metrics
4. Baselines

Challenge description

The challenge uses the openBHB dataset and aims to i) predict age from derived data from 3D T1 anatomical MRI while ii) removing site information from the learned representation. Thus, we aim to compare the capacity of proposed models to encode a relevant representation of the data (feature extraction and dimensionality reduction) that preserve the biological variability associated with age while removing the site-specific information. The algorithms submitted must output a low-dimension features vector (p < 10000). Derived data are composed of Quasi-Raw, VBM, and SBM.

Ranking of the best models

A public leaderboard with up-to-date results is available here

Warning: This public leaderboard contains the metrics computed on the public validation set. It may be biased by models trained on all public data. To access to this leaderboard, you must be registered to the challenge.

Last update: 28/07/2022

Metrics

Models submitted are evaluated with the standard linear evaluation protocol (see Figure below) to predict age and site from the data encoded by the submitted models.

Legend: model evaluation workflow of a new submission. When a new trained model is submitted to RAMP, a linear probe (regressor for age prediction and classifier for site classification) is trained on top of the public embedded data (i.e. public data encoded by the submitted model). Once trained, this linear probe predicts the downstream targets (age and site) on the private embedded data (age is predicted from both private internal and external tests while site is predicted from private internal test only). 3 metrics are then derived: Mean Absolute Error (MAE) for age prediction on internal and external test; Balanced Accuracy (BAcc) for site prediction on internal test. These 3 metrics are combined to derive the final challenge metric Lc.

Three metrics are computed on encoded data (a.k.a model representation):

• MAE{int} :arrow_down: Mean Absolute Error (MAE) for age prediction on internal test (images acquired from same scanners/sites as training images);
• MAE{ext} :arrow_down: MAE for age prediction on external test (cross-scanners/sites images);
• Bacc :arrow_down Balanced Accuracy for site prediction on internal test.

The overall challenge metrics is computed as follows:

Lc = BAcc(sites)^0.3 . MAE{ext}(age) + (1/Nsites)^0.3 MAE{int}(age)

As a result, the model representation should be invariant to noise induced during acquisition (low Bacc) but should retain biological variability associated to age (low MAE).

Baselines

Implementation: you can find the PyTorch implementation of the previous models in the GitHub repository: https://github.com/Duplums/brain_age_with_site_removal.

We performed baseline experiments,a nd trained CNNs with various architectures on whole-brain measures (VBM and Quasi-Raw). The objective function is a simple l1-loss on age prediction for all models. We tested ComBat data-based debiasing models, that correponds to classical harmonization of the training set (test sets are left unharmonized since age and site labels are not available). We reported the 3 metrics used in the challenge: MAE on the openBHB and privateBHB test sets and Bacc for site prediction. The latent space dimension varied across CNN architectures and it is always reported.

Legend: baselines obtained with 1) no de-biasing strategy (first 3 rows) and 2) ComBat residualization on training data (last 3 rows) with VBM and Quasi-Raw data for 3 representative CNN families.

We can notice that all models retain site information without any debiasing strategy. Overall, Quasi-Raw data are more biased than VBM, and CNN preserve this bias to some extent. The ResNet seems to be the best trade-off as it is robust to site and it generalizes well on the privateBHB test set. Interesingly, ComBat harmonization does remove most of site bias in CNN representation space (with site prediction Bacc almost matching the one obtained using only age as input). However, it also heavily degrades CNN performance on age prediction for all testing sets (in particular for DenseNet). ComBat is not fitted for Quasi-Raw data as it mainly relies on voxel-wise statistics, and raw data are not properly registrated voxel-wise across images. Consequently, we did not evaluated this approach on Quasi-Raw images.