braindecode.models.FNIRST#
- class braindecode.models.FNIRST(n_outputs=None, n_chans=None, chs_info=None, n_times=None, input_window_seconds=None, sfreq=None, *, embed_dim=64, depth=6, num_heads=8, mlp_dim=64, pool='cls', head_dim=64, n_conv_filters=8, spatial_kernel_size=5, temporal_kernel_size=30, temporal_stride=4, channel_order=None, drop_prob=0.0, embed_drop_prob=0.0, activation=<class 'torch.nn.modules.activation.GELU'>)[source]#
fNIRS-T from Wang et al. (2022).
Convolution Attention/Transformer
Figure of the authors’ repository (
fig/model.png, wzhlearning/fNIRS-Transformer at6ebe987f, MIT): the \(H \times W \times C\) input (channels, samples, HbO/HbR) feeds a “Spatial-level” and a “Channel-level” branch, each a “Transformer Encoder” with a CLS token. Bottom: fNIRS-T takes filtered, baseline-corrected segments; fNIRS-PreT replaces those steps by a “Preprocessing Module” (FNIRSPreT). The paper (IEEE JBHI 2022) is not open access; its caption is not reproduced.#From the abstract [1]: “Inspired by the success of Transformer based on self-attention mechanism in the fields of natural language processing and computer vision, we propose an fNIRS classification network based on Transformer, named fNIRS-T. We explore the spatial-level and channel-level representation of fNIRS signals to improve data utilization and network representation capacity.” On the three-class finger- and foot-tapping task (30 subjects), “fNIRS-T, CNN, and LSTM obtain 75.49%, 72.89%, and 61.94% on test sets”.
In the authors’ code the two levels read the same signals at two spatial scales: a convolution turns the time course of each fNIRS channel (channel level) or of a group of consecutive channels (spatial level) into one token, and a transformer lets these tokens attend to each other; time never becomes a token axis. Only the abstract, the README, the figure and the code (
model.py,KFold_Train.py,LOSO_Train.py) are public, and the description below comes from them.Architecture Overview
In the figure’s symbols, \(H\) =
n_chans // 2fNIRS channels, \(W\) =n_times, \(C\) = 2 chromophores, \(H_S = H - k_s + 1\) spatial tokens, \(H_C = H\) channel tokens and \(D\) =embed_dim:input: H HbO rows and H HbR rows (batch, 2 H, W) to_chromophore_planes (batch, 2, H, W) Spatial level (to_patch_embedding, transformer_patch): Conv_S, Rearrange, Linear, LayerNorm (batch, H_S, D) prepend CLS, add positions, encoder (batch, H_S + 1, D) Channel level (to_channel_embedding, transformer_channel): Conv_C, Rearrange, Linear, LayerNorm (batch, H_C, D) prepend CLS, add positions, encoder (batch, H_C + 1, D) CLS token of each level, Concat (batch, 2 D) final_norm, final_layer (batch, n_outputs)
Macro Components
to_chromophore_planesInput. The authors’ input is “[B, 2, fNIRS channels, sampling points]”, where ‘“2” means two chromophores: HbO and HbR’ (model.py, README). The rows ofxare split into these two planes, from the stacked or MNE’s interleaved order.to_patch_embeddingSpatial level (“fNIRS Spatial Embedding”). \(\mathrm{Conv}_S\) maps the two planes to \(C_S\) =n_conv_filtersmaps with a \(k_s \times k_t\) kernel and stride \(1 \times s_t\) (the authors: 8 maps, \(5 \times 30\), stride \(1 \times 4\)), so each token covers \(k_s\) consecutive channels in the dataset’s order. The \(C_S \times W_S\) responses of a token are flattened, projected to \(D\) and layer-normalized.to_channel_embeddingChannel level (“fNIRS Channel Embedding”). The same with a \(1 \times k_t\) kernel (\(\mathrm{Conv}_C\)): one token per fNIRS channel.cls_token_*,pos_embedding_*,transformer_*Transformer Encoder. Each level prepends a learned class token, adds a learned positional embedding (one row per token, the token’s only identity), applies dropout and runsdepthpre-norm blocks of multi-head self-attention and a GELU feed-forward layer of widthmlp_dim. The queries, keys and values come from bias-free linear maps, and each of thenum_headsheads of width \(d_h\) =head_dimcomputes, as in the authors’Attention,\[\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left( \frac{Q K^{\top}}{\sqrt{d_h}} \right) V .\]final_norm,final_layerHead. The class token of each level (or, withpool="mean", the mean of its tokens) is taken, the two vectors are concatenated (“Concat”), layer-normalized and classified by a linear layer (the authors’mlp_head).
Temporal, Spatial, and Spectral Encoding
Temporal: the convolutions slide a \(k_t\)-sample kernel with stride \(s_t\) along time, and the resulting values become features of a token. The README’s segments are 140 samples (dataset A), 200 (B) and 256 (C). Kernels count samples, so
sfreqis unused.Spatial: channels or groups of \(k_s\) consecutive channels, identified by their position in the token sequence; no montage or optode geometry. The channel order must be the one used in training.
Chromophores: HbO and HbR are the two input planes, mixed by both convolutions from the first layer.
Spectral: none in the model; fNIRS-T expects band-pass filtered, baseline-corrected segments, and
FNIRSPreTlearns that step.
Pretrained Weights
None: the authors release code, not weights.
Differences from the reference implementation
xis(batch, n_chans, n_times),n_chans = 2 H, HbO rows then HbR rows, or MNE’s interleaved order read fromchs_info. The figure draws an “MLP Head” per level before “Concat”; the authors’ code, which this module follows, concatenates first and applies one head, and the output is logits. The positional tables have one row per token (the authors allocate 101 rows and slice), so an authors’state_dictloads after a conversion rather than amapping(to_qkvsplit intoqueries/keys/values, tables cut,mlp_head.{0,1}tofinal_norm/final_layer); the port then matches the authors’ model to float32 rounding (max-abs below 1e-6 on their dataset A, B and C configurations). The attention always projects its output (the authors skip it whennum_heads=1andhead_dim=embed_dim).maskis an addition of the port;ch_maskis refused, as the spatial kernel and the positional tables span the whole montage.- Parameters:
n_outputs (
int|None) – Number of outputs of the model. This is the number of classes in the case of classification.chs_info (
list[dict] |None) – Information about each individual EEG channel. This should be filled withinfo["chs"]. Refer tomne.Infofor more details.n_times (
int|None) – Number of time samples of the input window.input_window_seconds (
float|None) – Length of the input window in seconds.sfreq (
float|None) – Sampling frequency of the EEG recordings.embed_dim (
int) – \(D\), the authors’dim, “last dimension of output tensor after linear transformation”; 64 for their datasets A and B, 128 for C.depth (
int) – Authors’depth, “number of Transformer blocks” per level.num_heads (
int) – Authors’heads, “number of the multi-head self-attention”.mlp_dim (
int) – Authors’mlp_dim, “dimension of the MLP layer”.pool (
Literal['cls','mean']) – Read-out of each level (authors’pool):"cls", its class token;"mean", the mean over its tokens (class token included).head_dim (
int) – \(d_h\), the authors’dim_head: width of each attention head.n_conv_filters (
int) – \(C_S = C_C\), feature maps of both convolutions (authors: 8).spatial_kernel_size (
int) – \(k_s\), height of \(\mathrm{Conv}_S\) in HbO/HbR pairs (authors: 5); at most the number of pairs,n_chans // 2.temporal_kernel_size (
int) – \(k_t\), temporal kernel of both convolutions, in samples (authors: 30);n_timesmust be at least this.temporal_stride (
int) – \(s_t\), temporal stride of both convolutions, in samples (authors: 4).channel_order (
Optional[Literal['stacked','interleaved']]) –[HbO_1..HbO_H, HbR_1..HbR_H]or MNE’s[HbO_1, HbR_1, ...];Nonereads it fromchs_info.drop_prob (
float) – Authors’dropoutin the attention and feed-forward sub-layers.embed_drop_prob (
float) – Authors’emb_dropout, “dropout for patch embeddings”.activation (
type[Module]) – Activation of the feed-forward sub-layers (authors: GELU).
- Raises:
ValueError – If some input signal-related parameters are not specified: and can not be inferred.
Notes
The authors’ code uses three datasets: A (mental arithmetic vs rest, 8 subjects, 52 channels, 140 samples), B (Shin et al. 2017, same task, 29 subjects, 36 channels, 200 samples) and C (Bak et al. 2019, finger and foot tapping, 3 classes, 30 subjects, 20 channels, 256 samples). The fNIRS-T inputs are band-passed (0.01-0.1 Hz on B and C), baseline-corrected and z-scored per window. Training is 5 x 5-fold cross-validation over pooled trials or leave-one-subject-out, 120 epochs, batch 128, AdamW with cosine annealing (
T_max=30) and label smoothing 0.1 in both loops, and the best test-fold accuracy is kept. On B and C the loss is “flooded” (loss = (loss - b).abs() + b) with a level \(b\) the README decays “at the specified epoch to obtain continuous regularization”: 0.45, 0.40 and 0.35 from the zero-based epochs 0, 30 and 50.Replication on dataset C with the authors’
KFold_Train.py(mean of the 25 runs of each seed, averaged over seeds 0-2; macro-F1 at the kept epoch):Model
Paper (abstract)
Port (acc / F1)
Authors’ code
fNIRS-T
75.49 %
75.03 % / 0.7498
74.73 % / 0.7469
fNIRS-PreT
“competitive”
72.50 % / 0.7244
72.50 % / 0.7245
Added in version 2.1.
References
Hugging Face Hub integration
When the optional
huggingface_hubpackage is installed, all models automatically gain the ability to be pushed to and loaded from the Hugging Face Hub. Install with:pip install braindecode[hub]
Pushing a model to the Hub:
from braindecode.models import FNIRST # Train your model model = FNIRST(n_chans=22, n_outputs=4, n_times=1000) # ... training code ... # Push to the Hub; ``datasets`` and ``base_model`` go to the card model.push_to_hub( repo_id="username/my-fnirst-model", commit_message="Initial model upload", model_card_kwargs={"datasets": ["BNCI2014_001"]}, )
Loading a model from the Hub:
from braindecode.models import FNIRST # Load pretrained model model = FNIRST.from_pretrained("username/my-fnirst-model") # Data the weights were trained on, from the Hub model card print(model.card_data.datasets if model.card_data else None) # Load with a different number of outputs (head is rebuilt automatically) model = FNIRST.from_pretrained("username/my-fnirst-model", n_outputs=4)
Extracting features and replacing the head:
import torch x = torch.randn(1, model.n_chans, model.n_times) # Extract encoder features (consistent dict across all models) out = model(x, return_features=True) features = out["features"] # Replace the classification head model.reset_head(n_outputs=10)
Saving and restoring full configuration:
import json config = model.get_config() # all __init__ params with open("config.json", "w") as f: json.dump(config, f) model2 = FNIRST.from_config(config) # reconstruct (no weights)
All model parameters (both EEG-specific and model-specific such as dropout rates, activation functions, number of filters) are automatically saved to the Hub and restored when loading.
See Loading and Adapting Pretrained Foundation Models for a complete tutorial.
Methods
- forward(x, return_features=False, ch_mask=None, mask=None)[source]#
Forward pass of the fNIRS-T (and fNIRS-PreT) model.
- Parameters:
x (
Tensor) – Input tensor with shape(batch_size, n_chans, n_times).return_features (
bool) – If True, return the normalized pooled features and the class tokens of the two levels.ch_mask (
Tensor|None) – Not supported: the spatial kernel and the positional tables span every channel.mask (
Tensor|None) – Boolean tensor with the shape ofx; True samples are set to zero before the model reads them.
- Returns:
Logits with shape
(batch_size, n_outputs), or withreturn_features=Truethe dict{"features", "cls_token"}, both of shape(batch_size, 2 * embed_dim).- Return type:
- reset_head(n_outputs)[source]#
Replace the classification head for a new number of outputs.
This is called automatically by
from_pretrained()when the user passes ann_outputsthat differs from the saved config. Override in subclasses that need a model-specific head structure. Implementations keep changed constructor arguments in sync withself._update_init_kwargs, so that a saved model can be loaded back. Implementations requiring positive outputs can also useself._set_n_outputsto validate and record the new value.Examples
>>> from braindecode.models import BENDR >>> model = BENDR(n_chans=22, n_times=1000, n_outputs=4) >>> model.reset_head(10) >>> model.n_outputs 10
Added in version 1.4.