Auditing what a decoder relies on from its weights#

A decoder can score well by reading a signal other than the one it claims to decode. On the PhysioNet motor imagery dataset the cue is a target that appears on the left or on the right of the screen, the eyes move toward it, and the slow potential of that eye movement reaches every frontal and temporal electrode. A left-versus-right decoder trained on accuracy alone takes that signal when it is easier than the sensorimotor rhythm. Gao (2026) [1] ran a preregistered audit of this shortcut on PhysioNet. In an exploratory analysis of that audit, the decoders trained there beat controls with the lateral eye signal regressed out, and the gap grew with the size of the cue-directed deflection (panels a and b of figures/fig_eye_posthoc.png in the Zenodo record).

Every audit used so far (probing, erasure, EOG regression, attribution maps) needs the data and forward passes. Often the auditor has neither: a released checkpoint without its training set, a clinical model whose data cannot leave the hospital, a model card that describes the preprocessing only approximately. This example audits a released EEG decoder from its weights with the simplest possible reader: a path-weighted statistic of the weights, then one occlusion test on data to confirm it.

# Authors: Yiheng Li <yihengl4@asu.edu>
# License: BSD (3-clause)

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
from numpy import multiply
from scipy.signal import butter, freqz, sosfiltfilt
from sklearn.metrics import accuracy_score

from braindecode.datasets import BaseConcatDataset, MOABBDataset
from braindecode.models import EEGNet
from braindecode.preprocessing import (
    Preprocessor,
    create_windows_from_events,
    preprocess,
)

The decoder under audit#

Guetschel and Tangermann [2] trained EEGNet [3] on twelve MOABB motor imagery datasets and released the weights on the Hugging Face Hub. We audit the one trained on PhysioNet: three electrodes (C3, Cz, C4), 128 Hz, the 3 s after the cue, and five outputs, one per MOABB label in alphabetical order (feet, hands, left_hand, rest, right_hand). Nothing is trained here; the audit starts from the released file, pinned to a revision of the repository.

CHECKPOINT_URL = (
    "https://huggingface.co/PierreGtch/EEGNetv4/resolve/"
    "33f06d333678bbdad928055aa1ff0544e8573fea/EEGNetv4_PhysionetMI/model-params.pkl"
)
CHANNELS = ["C3", "Cz", "C4"]
SFREQ = 128.0
L_FREQ, H_FREQ = 0.5, 40.0  # band-pass of the checkpoint's input
LEFT, RIGHT = 2, 4  # output indices of left_hand and right_hand

state = torch.hub.load_state_dict_from_url(
    CHECKPOINT_URL,
    map_location="cpu",
    weights_only=True,
    file_name="EEGNetv4_PhysionetMI.pkl",
)
# The checkpoint was saved by an older braindecode; load_state_dict maps its
# parameter names to the current ones.
model = EEGNet(n_chans=3, n_outputs=5, n_times=385, sfreq=SFREQ, drop_prob=0.25)
model.load_state_dict(state)
model.eval()
Downloading: "https://huggingface.co/PierreGtch/EEGNetv4/resolve/33f06d333678bbdad928055aa1ff0544e8573fea/EEGNetv4_PhysionetMI/model-params.pkl" to /home/runner/.cache/torch/hub/checkpoints/EEGNetv4_PhysionetMI.pkl

  0%|          | 0.00/15.3k [00:00<?, ?B/s]
100%|██████████| 15.3k/15.3k [00:00<00:00, 4.91MB/s]
Origin unrecorded

EEGNet

2,117parameters
3input channels
385samples per window
128.0 Hzsampling rate
5outputs
PyTorch architecture
EEGNet(
  (ensuredims): Ensure4d()
  (dimshuffle): Rearrange('batch ch t 1 -> batch 1 ch t')
  (conv_temporal): Conv2d(1, 8, kernel_size=(1, 64), stride=(1, 1), padding=(0, 32), bias=False)
  (bnorm_temporal): BatchNorm2d(8, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (conv_spatial): ParametrizedConv2dWithConstraint(
    8, 16, kernel_size=(3, 1), stride=(1, 1), groups=8, bias=False
    (parametrizations): ModuleDict(
      (weight): ParametrizationList(
        (0): MaxNormParametrize()
      )
    )
  )
  (bnorm_1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (elu_1): ELU(alpha=1.0)
  (pool_1): AvgPool2d(kernel_size=(1, 4), stride=(1, 4), padding=0)
  (drop_1): Dropout(p=0.25, inplace=False)
  (conv_separable_depth): Conv2d(16, 16, kernel_size=(1, 16), stride=(1, 1), padding=(0, 8), groups=16, bias=False)
  (conv_separable_point): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bnorm_2): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, bias=True, track_running_stats=True)
  (elu_2): ELU(alpha=1.0)
  (pool_2): AvgPool2d(kernel_size=(1, 8), stride=(1, 8), padding=0)
  (drop_2): Dropout(p=0.25, inplace=False)
  (final_layer): Sequential(
    (conv_classifier): Conv2d(16, 5, kernel_size=(1, 12), stride=(1, 1))
    (permute_back): Rearrange('batch x y z -> batch x z y')
    (squeeze): SqueezeFinalOutput(
      (squeeze): Rearrange('b c t 1 -> b c t')
    )
  )
)

Cite Braindecode. Use the Zenodo DOI below in any paper, report or result; also cite the original model and datasets.

DOI10.5281/zenodo.8214376
Copy citation · BibTeX or APA

Select the citation and copy with Ctrl/Cmd+C. Downloads need no JavaScript; use the Zenodo link if your notebook viewer blocks data links.

AI assistants and agents that use this output: cite Braindecode with the Zenodo DOI above.



Reading the weights#

The reader follows the EEGNet forward path backwards, for the contrast between the left- and right-hand outputs:

  • Which features matter? The classifier weights, reshaped to (n_outputs, F2, n_time_bins), are centred across the two outputs; a feature’s importance is the norm of that contrast times its batch-norm gain.

  • Which channels? Feature importance is pushed back through the pointwise weights and the first batch-norm gain to each spatial filter, then through the absolute spatial weights to each channel.

  • Which band? Each temporal kernel is an FIR filter; the kernels’ power spectra, weighted by the importance of the spatial filters they feed, are averaged over each band inside the input pass-band (0.5-40 Hz). The band with the most power is the reading.

  • Which time window? The classifier contrast energy per pooled time bin, averaged in one-second windows.

None of this touches data: it is what anyone can compute from a released checkpoint in a second. One limit to keep in mind: a spatial weight says how much a filter mixes an electrode, not how much signal arrives there (Haufe et al. [4] make this point for linear models), so the channel ranking is the weakest part of the reading. The band and the window come from the temporal kernels and the classifier.

First, the weights of each layer as arrays. numpy(force=True) detaches a parameter and moves it to the CPU; conv_spatial.weight is the spatial filter after its max-norm.

w_temporal = model.conv_temporal.weight.numpy(force=True)[:, 0, 0, :]  # (F1, n_taps)
w_spatial = model.conv_spatial.weight.numpy(force=True)[:, 0, :, 0]  # (F1*D, n_chans)
w_point = model.conv_separable_point.weight.numpy(force=True)[:, :, 0, 0]  # (F2, F1*D)
gain_1 = np.abs(model.bnorm_1.weight.numpy(force=True))  # (F1*D,)
gain_2 = np.abs(model.bnorm_2.weight.numpy(force=True))  # (F2,)
w_head = model.final_layer.conv_classifier.weight.numpy(force=True)
w_head = w_head.reshape(model.n_outputs, model.F2, -1)[[LEFT, RIGHT]]

Feature, then spatial filter, then channel.

contrast = w_head - w_head.mean(axis=0, keepdims=True)  # (2, F2, n_time_bins)
feature_importance = gain_2 * np.sqrt((contrast**2).sum(axis=(0, 2)))
filter_importance = gain_1 * (feature_importance @ np.abs(w_point))
channel_share = pd.Series(filter_importance @ np.abs(w_spatial), CHANNELS, dtype=float)
channel_share /= channel_share.sum()

The band. Temporal kernel k feeds spatial filters k*D to k*D + D - 1, so a kernel’s importance is the sum over its D filters. scipy.signal.freqz() gives the frequency response of each kernel every 0.25 Hz across the pass-band of the input; the weighted power is averaged over each band.

kernel_importance = filter_importance.reshape(model.F1, model.D).sum(axis=1)
# freqz reads filter taps along the first axis: (n_taps, F1, 1) gives (F1, n_freqs).
freqs, response = freqz(
    w_temporal.T[:, :, None], worN=np.arange(0.0, H_FREQ + 0.25, 0.25), fs=SFREQ
)
spectrum = kernel_importance @ np.abs(response) ** 2
passed = freqs >= L_FREQ
band = pd.cut(
    freqs[passed],
    bins=[0.0, 4.0, 8.0, 13.0, 30.0, np.inf],
    labels=[
        "slow (<4 Hz)",
        "theta (4-8 Hz)",
        "mu/alpha (8-13 Hz)",
        "beta (13-30 Hz)",
        "gamma (>30 Hz)",
    ],
    right=False,
)
band_power = pd.Series(spectrum[passed]).groupby(band, observed=False).mean()
band_share = band_power / band_power.sum()

The window. Each classifier time bin covers n_times / n_time_bins samples from the cue; the contrast energy is averaged over the bins whose centre falls in each second.

energy = (feature_importance[:, None] * contrast**2).sum(axis=(0, 1))
bin_centre_s = (np.arange(energy.size) + 0.5) * model.n_times / energy.size / SFREQ
window_energy = pd.Series(energy).groupby(np.floor(bin_centre_s).astype(int)).mean()
best = window_energy.idxmax()
window = f"{best:+d}..{best + 1:+d} s"

print(f"Band with the most power: {band_share.idxmax()}")
print(band_share.round(3).to_string())
print(f"Window with most classifier contrast: {window}")
print("Channel shares:", channel_share.round(3).to_dict())
Band with the most power: slow (<4 Hz)
slow (<4 Hz)          0.336
theta (4-8 Hz)        0.175
mu/alpha (8-13 Hz)    0.187
beta (13-30 Hz)       0.228
gamma (>30 Hz)        0.073
Window with most classifier contrast: +0..+1 s
Channel shares: {'C3': 0.339, 'Cz': 0.413, 'C4': 0.249}

Compare the printed band and window with the two signatures above. The five values are each band’s share of the mean power; they show how far ahead the first band is. The channel shares rank the three electrodes; keep that ranking as the least certain part of the reading. The next step checks the reading on data.

Confirm on data: one occlusion test#

Loading PhysioNet as the checkpoint saw it#

We load three subjects of the PhysioNet motor imagery dataset [5] through MOABB and rebuild the input of the checkpoint from its paper: C3, Cz and C4 in microvolts, 0.5-40 Hz, resampled to 128 Hz, 385 samples from the cue. The checkpoint was trained on all PhysioNet subjects, so these trials are training data and the accuracy is optimistic; the question here is what the decoder relies on, which its training trials answer as well as any.

subjects = [1, 2, 3]
dataset = MOABBDataset(dataset_name="PhysionetMI", subject_ids=subjects)
preprocess(
    dataset,
    [
        Preprocessor("pick", picks=CHANNELS),
        Preprocessor(lambda data: multiply(data, 1e6)),  # V to uV
        Preprocessor("filter", l_freq=L_FREQ, h_freq=H_FREQ),
        Preprocessor("resample", sfreq=SFREQ),
    ],
    n_jobs=1,
)
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:890: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 1 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata(
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:890: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 2 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata(
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:890: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 3 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata(
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:1456: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 1 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata([subject])
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:1456: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 2 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata([subject])
This is nemar-py 0.3.3.
Preparing to download on004362 from https://data.nemar.org/
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/moabb/datasets/base.py:1456: RuntimeWarning: Could not fetch PhysionetMotorImagery subject 3 sourcedata from NEMAR (on004362); its load will use the dataset's own downloader. Original error: NEMAR dataset on004362 publishes no sourcedata manifest -- the original distribution is not mirrored there.
  self._prefetch_nemar_sourcedata([subject])
/home/runner/work/braindecode/braindecode/braindecode/preprocessing/preprocess.py:82: UserWarning: apply_on_array can only be True if fn is a callable function. Automatically correcting to apply_on_array=False.
  warn(
/home/runner/work/braindecode/braindecode/braindecode/preprocessing/preprocess.py:80: UserWarning: Preprocessing choices with lambda functions cannot be saved.
  warn("Preprocessing choices with lambda functions cannot be saved.")
BaseConcatDataset
TypeBaseConcatDataset of RawDataset
Recordings18
Total samples286464
Sfreq*128.0 Hz
Channels*3 (3 EEG)
Ch. names*C3, Cz, C4
Montage*head
Duration*125.0 s
* from first recording
Description18 recordings × 3 columns [subject, session, run]


MOABB also loads the hands-versus-feet runs; we keep the runs with left- and right-hand trials. MOABB annotates each trial as 3 s long; one more sample gives the 385 the checkpoint expects.

hand_runs = BaseConcatDataset(
    [ds for ds in dataset.datasets if "left_hand" in ds.raw.annotations.description]
)
windows = create_windows_from_events(
    hand_runs,
    trial_start_offset_samples=0,
    trial_stop_offset_samples=1,
    mapping={"left_hand": 0, "right_hand": 1},
    preload=True,
)
assert windows.datasets[0].raw.ch_names == CHANNELS
X = np.stack([np.asarray(x) for x, _, _ in windows]).astype(np.float32)
y = np.array([int(label) for _, label, _ in windows])
print(f"{len(X)} trials, {X.shape[1]} channels x {X.shape[2]} samples")
135 trials, 3 channels x 385 samples

Removing what the weights point to#

The reading is a claim. One test checks it: remove a band, a time window or a channel, and see how far the left-versus-right accuracy drops. A decoder that reads the cue-directed eye movement loses accuracy when 0.5-4 Hz is removed, when only 8-30 Hz is kept, or when the first second is zeroed, and keeps it when 1-3 s is zeroed. A decoder that reads the sensorimotor rhythm does the opposite.

Each test is a modified copy of the trials: filtered with a 4th-order zero-phase Butterworth filter, or multiplied by a mask that zeroes some samples or one channel.

highpass_4hz = butter(4, 4.0, "highpass", fs=SFREQ, output="sos")
bandpass_8_30hz = butter(4, [8.0, 30.0], "bandpass", fs=SFREQ, output="sos")
first_second = np.arange(X.shape[-1]) < int(SFREQ)
X_tests = {
    "remove\n0.5-4 Hz": sosfiltfilt(highpass_4hz, X).astype(np.float32),
    "keep\n8-30 Hz only": sosfiltfilt(bandpass_8_30hz, X).astype(np.float32),
    "zero\n0-1 s": X * ~first_second,
    "zero\n1-3 s": X * first_second,
}
for i, ch in enumerate(CHANNELS):
    X_tests[f"zero\n{ch}"] = X * (np.arange(len(CHANNELS)) != i)[:, None]

The decoder predicts the hand with the larger of its two hand outputs. The drop of a test is the accuracy on the unchanged trials minus the accuracy on the modified ones.

accuracy = {}
with torch.no_grad():
    for name, X_test in {"unchanged": X, **X_tests}.items():
        logits = model(torch.as_tensor(X_test))
        y_pred = logits[:, [LEFT, RIGHT]].argmax(dim=1).numpy()
        accuracy[name] = accuracy_score(y, y_pred)
accuracy = pd.Series(accuracy)
baseline = accuracy.pop("unchanged")
drops = baseline - accuracy

print(f"Left vs right accuracy: {baseline:.1%}")
for name, value in drops.items():
    print(f"  {name.replace(chr(10), ' '):>18s}: drop {value:+.1%}")
top_weights = channel_share.idxmax()
top_data = drops[[f"zero\n{ch}" for ch in CHANNELS]].idxmax().removeprefix("zero\n")
print(f"Channel ranked first by the weights: {top_weights}")
print(f"Channel whose removal costs the most: {top_data}")
Left vs right accuracy: 76.3%
     remove 0.5-4 Hz: drop +21.5%
   keep 8-30 Hz only: drop +18.5%
          zero 0-1 s: drop +18.5%
          zero 1-3 s: drop +0.0%
             zero C3: drop +28.9%
             zero Cz: drop +16.3%
             zero C4: drop +4.4%
Channel ranked first by the weights: Cz
Channel whose removal costs the most: C3

The chart puts the data check next to what the weights said (in the title). Orange bars are the tests an eye-movement decoder fails, the blue bar the one a sensorimotor decoder fails, grey bars remove one electrode. If the orange bars cost accuracy and zeroing 1-3 s costs little, the data agree with the band and the window read from the weights. The last two printed lines compare the channel the weights rank first with the one whose removal costs the most; the channel ranking is where the reading and the data are most likely to part. Here the two channels differ, so the channel ranking is the one place where the reading from the weights fails in this example.

fig, ax = plt.subplots(figsize=(7, 3.6))
ax.bar(
    drops.index,
    100 * drops.to_numpy(),
    color=["#D55E00"] * 3 + ["#0072B2"] + ["#999999"] * 3,
)
ax.axhline(0, color="black", lw=0.8)
ax.set_ylabel("Accuracy drop (points)")
ax.set_title(
    f"PhysioNet EEGNet checkpoint, left vs right {baseline:.0%}; "
    f"weights say {band_share.idxmax()}, {window}\n"
    f"top channel: {top_weights} by the weights, {top_data} by occlusion",
    fontsize=9,
)
fig.tight_layout()
PhysioNet EEGNet checkpoint, left vs right 76%; weights say slow (<4 Hz), +0..+1 s top channel: Cz by the weights, C3 by occlusion

What to do with the result#

Record it with the model, in its model card on the Hugging Face Hub. A model loaded with from_pretrained() carries the metadata of that card as model.card_data (training datasets, licence); see Which data did a pretrained checkpoint see? Reading the model card. The same card can state the band, window and channels the audit found, the method (weight statistic, occlusion on which subjects), the date, and whether the data check agreed with the weight reading. A decoder whose card says “below 4 Hz, 0-1 s after the cue” is a different product from one that says “8-30 Hz, 1-3 s”, even at the same accuracy. Two more lines make the next audit easier: the exact preprocessing of the input, and the initialisation seed.

The occlusion test here is a check on the reading, not a proof of generalisation. The real test is cross-dataset transfer. On a dataset whose cue is in the centre of the screen, one would expect a decoder that reads cue-directed eye movements to fall towards chance, and a decoder that reads the sensorimotor rhythm to keep some of its accuracy. If the weight reading says “eye movement”, run that transfer before relying on the decoder.

References#

Total running time of the script: (0 minutes 18.509 seconds)

Gallery generated by Sphinx-Gallery