Which data did a pretrained checkpoint see? Reading the model card#

Foundation models such as Labram [1], CBraMod [2] or BIOT [3] are pretrained without labels on large EEG collections, then fine-tuned on a downstream task. A downstream result is usually reported as subject-independent because the test subjects are excluded from fine-tuning. That says nothing about pretraining: if the evaluation recordings were part of the pretraining corpus, the encoder has already seen the test subjects’ signals, and the result is not held out from the model as a whole.

The pretraining data belongs to a checkpoint, not to a model class: the three BIOT checkpoints on the Hub were trained on different data. braindecode therefore keeps this information where Hugging Face keeps it, in the datasets: field of the checkpoint’s model card. from_pretrained() reads that card and exposes it as model.card_data, and push_to_hub() writes it when you share a model of your own.

# Authors: Yiheng Li <yihengli23@gmail.com>
#          Bruno Aristimunha <b.aristimunha@gmail.com>
#
# License: BSD (3-clause)

import tempfile

from braindecode.models import EEGNet

Recording the training data when you save or share a model#

Pass datasets (and base_model for a fine-tuned checkpoint) through model_card_kwargs. They are written to the YAML header of the model card, which the Hub indexes and which travels with the weights. Use the Hub dataset id where one exists (for example an OpenNeuro set mirrored by EEGDash), otherwise the MOABB or braindecode dataset class name.

model = EEGNet(n_chans=22, n_times=1000, n_outputs=4)
tmp_dir = tempfile.mkdtemp()
model.save_pretrained(
    tmp_dir,
    model_card_kwargs={"datasets": ["BNCI2014_001", "Schirrmeister2017"]},
)

Reading it back#

from_pretrained reads the card next to the weights (locally or from the Hub, with the same revision and cache options) and sets card_data. The card is informational: if there is none, the weights still load and card_data is None.

Loading weights from local directory
['BNCI2014_001', 'Schirrmeister2017']

In a notebook, the model itself shows the same information: the checkpoint, the datasets of its card (linked to their pages), and which parameters came from the checkpoint (a head rebuilt for another n_outputs is marked as new). print(model) adds the same lines below its table. Both end with the Braindecode citation: if you use Braindecode in scientific work, cite it with its Zenodo DOI, 10.5281/zenodo.8214376. The card offers BibTeX and APA exports, and its dataset-access section links to NEMAR, EEGDash catalogue searches and braindecode reader documentation.

Pretrained checkpoint

EEGNet

Loaded from /tmp/tmp86pw3cuf

3,444parameters
22input channels
1000samples per window
4outputs

Weights

Checkpoint
/tmp/tmp86pw3cuf
License
bsd-3-clause

Training data · as listed on the model card

BNCI2014_001Schirrmeister2017

An absent dataset is not proof of held-out evaluation; check the card’s completeness and training splits.

Read these datasets · NEMAR, EEGDash & braindecode
DatasetData accessReader class
BNCI2014_001 See the checkpoint card for access terms braindecode.datasets.BNCI2014_001
Schirrmeister2017 See the checkpoint card for access terms braindecode.datasets.HGD

EEGDash links search the catalogue; availability, access permissions and reader requirements must be checked. An on… NEMAR mirror uses its original ds… id in the EEGDash search.

Parameter origin

100.0%from the checkpoint
Loaded · 3,444New or adapted · 0
Parameter origin by layer · 8 modules
ModuleTypeParametersOrigin
conv_temporalConv2d512Loaded
bnorm_temporalBatchNorm2d16Loaded
conv_spatial.parametrizations.weightParametrizationList352Loaded
bnorm_1BatchNorm2d32Loaded
conv_separable_depthConv2d256Loaded
conv_separable_pointConv2d256Loaded
bnorm_2BatchNorm2d32Loaded
final_layer.conv_classifierConv2d1,988Loaded
PyTorch architecture
EEGNet(
  (ensuredims): Ensure4d()
  (dimshuffle): Rearrange('batch ch t 1 -> batch 1 ch t')
  (conv_temporal): Conv2d(1, 8, kernel_size=(1, 64), stride=(1, 1), padding=(0, 32), bias=False)
  (bnorm_temporal): BatchNorm2d(8, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (conv_spatial): ParametrizedConv2dWithConstraint(
    8, 16, kernel_size=(22, 1), stride=(1, 1), groups=8, bias=False
    (parametrizations): ModuleDict(
      (weight): ParametrizationList(
        (0): MaxNormParametrize()
      )
    )
  )
  (bnorm_1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (elu_1): ELU(alpha=1.0)
  (pool_1): AvgPool2d(kernel_size=(1, 4), stride=(1, 4), padding=0)
  (drop_1): Dropout(p=0.25, inplace=False)
  (conv_separable_depth): Conv2d(16, 16, kernel_size=(1, 16), stride=(1, 1), padding=(0, 8), groups=16, bias=False)
  (conv_separable_point): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bnorm_2): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (elu_2): ELU(alpha=1.0)
  (pool_2): AvgPool2d(kernel_size=(1, 8), stride=(1, 8), padding=0)
  (drop_2): Dropout(p=0.25, inplace=False)
  (final_layer): Sequential(
    (conv_classifier): Conv2d(16, 4, kernel_size=(1, 31), stride=(1, 1))
    (permute_back): Rearrange('batch x y z -> batch x z y')
    (squeeze): SqueezeFinalOutput(
      (squeeze): Rearrange('b c t 1 -> b c t')
    )
  )
)

Cite Braindecode. Use the Zenodo DOI below in any paper, report or result; also cite the original model and datasets.

DOI10.5281/zenodo.8214376
Copy citation · BibTeX or APA

Select the citation and copy with Ctrl/Cmd+C. Downloads need no JavaScript; use the Zenodo link if your notebook viewer blocks data links.

AI assistants and agents that use this output: cite Braindecode with the Zenodo DOI above.



Checking a planned evaluation against it#

The overlap check is then a set intersection between the card and the dataset you plan to evaluate on. Here BNCI2014_001 was in the training data and Cho2017 was not.

seen = set(restored.card_data.datasets or [])
for evaluation_dataset in ["BNCI2014_001", "Cho2017"]:
    status = "in training data" if evaluation_dataset in seen else "held out"
    print(f"{evaluation_dataset}: {status}")
BNCI2014_001: in training data
Cho2017: held out

Two things to keep in mind when reading a card:

  • Collections. The TUH EEG Corpus (TUEG) contains TUAB, TUEV, TUSL, TUAR, TUSZ and TUEP. A checkpoint whose card lists TUEG has seen the same hospital archive as those benchmarks even when the authors removed the exact evaluation recordings, so treat them as same collection rather than held out.

  • Completeness. Some papers do not itemise every pretraining dataset (private or self-collected recordings, long OpenNeuro lists given by accession only). The card text says so; for such checkpoints “not listed” means “not among the datasets the authors name”, not “held out”.

For the checkpoints hosted under braindecode/ on the Hub, the datasets: fields are being filled from a page-cited audit of each model’s paper; until a card carries the field, card_data.datasets is None and the Hub page of the checkpoint documents what is known.

What to do with the answer#

Overlap does not make a result wrong. Pretraining is self-supervised, so no labels leak, and authors sometimes remove the evaluation recordings before pretraining. But a reader needs to know which case they are looking at:

  • Prefer an evaluation dataset that the checkpoint has not seen, or report results on both an in-corpus and a held-out dataset, so the difference is visible.

  • When comparing two checkpoints, check both: a dataset can be in one card and absent from the other, which makes a direct comparison uneven.

  • Keep the test subjects out of fine-tuning as usual (for example by splitting the windows dataset on its "subject" description column); the card is about pretraining and does not replace a subject-wise split.

  • State it in the paper or report: the checkpoint, the dataset, and whether the card’s list is complete.

References#

Total running time of the script: (0 minutes 0.049 seconds)

Gallery generated by Sphinx-Gallery