braindecode.models.FNIRST#

class braindecode.models.FNIRST(n_outputs=None, n_chans=None, chs_info=None, n_times=None, input_window_seconds=None, sfreq=None, *, embed_dim=64, depth=6, num_heads=8, mlp_dim=64, pool='cls', head_dim=64, n_conv_filters=8, spatial_kernel_size=5, temporal_kernel_size=30, temporal_stride=4, channel_order=None, drop_prob=0.0, embed_drop_prob=0.0, activation=<class 'torch.nn.modules.activation.GELU'>)[source]#

fNIRS-T from Wang et al. (2022).

Convolution Attention/Transformer

fNIRS-T and fNIRS-PreT architecture

Figure of the authors’ repository (fig/model.png, wzhlearning/fNIRS-Transformer at 6ebe987f, MIT): the \(H \times W \times C\) input (channels, samples, HbO/HbR) feeds a “Spatial-level” and a “Channel-level” branch, each a “Transformer Encoder” with a CLS token. Bottom: fNIRS-T takes filtered, baseline-corrected segments; fNIRS-PreT replaces those steps by a “Preprocessing Module” (FNIRSPreT). The paper (IEEE JBHI 2022) is not open access; its caption is not reproduced.#

From the abstract [1]: “Inspired by the success of Transformer based on self-attention mechanism in the fields of natural language processing and computer vision, we propose an fNIRS classification network based on Transformer, named fNIRS-T. We explore the spatial-level and channel-level representation of fNIRS signals to improve data utilization and network representation capacity.” On the three-class finger- and foot-tapping task (30 subjects), “fNIRS-T, CNN, and LSTM obtain 75.49%, 72.89%, and 61.94% on test sets”.

In the authors’ code the two levels read the same signals at two spatial scales: a convolution turns the time course of each fNIRS channel (channel level) or of a group of consecutive channels (spatial level) into one token, and a transformer lets these tokens attend to each other; time never becomes a token axis. Only the abstract, the README, the figure and the code (model.py, KFold_Train.py, LOSO_Train.py) are public, and the description below comes from them.

Architecture Overview

In the figure’s symbols, \(H\) = n_chans // 2 fNIRS channels, \(W\) = n_times, \(C\) = 2 chromophores, \(H_S = H - k_s + 1\) spatial tokens, \(H_C = H\) channel tokens and \(D\) = embed_dim:

input: H HbO rows and H HbR rows             (batch, 2 H, W)
to_chromophore_planes                        (batch, 2, H, W)
Spatial level (to_patch_embedding, transformer_patch):
  Conv_S, Rearrange, Linear, LayerNorm       (batch, H_S, D)
  prepend CLS, add positions, encoder        (batch, H_S + 1, D)
Channel level (to_channel_embedding, transformer_channel):
  Conv_C, Rearrange, Linear, LayerNorm       (batch, H_C, D)
  prepend CLS, add positions, encoder        (batch, H_C + 1, D)
CLS token of each level, Concat              (batch, 2 D)
final_norm, final_layer                      (batch, n_outputs)

Macro Components

  • to_chromophore_planes Input. The authors’ input is “[B, 2, fNIRS channels, sampling points]”, where ‘“2” means two chromophores: HbO and HbR’ (model.py, README). The rows of x are split into these two planes, from the stacked or MNE’s interleaved order.

  • to_patch_embedding Spatial level (“fNIRS Spatial Embedding”). \(\mathrm{Conv}_S\) maps the two planes to \(C_S\) = n_conv_filters maps with a \(k_s \times k_t\) kernel and stride \(1 \times s_t\) (the authors: 8 maps, \(5 \times 30\), stride \(1 \times 4\)), so each token covers \(k_s\) consecutive channels in the dataset’s order. The \(C_S \times W_S\) responses of a token are flattened, projected to \(D\) and layer-normalized.

  • to_channel_embedding Channel level (“fNIRS Channel Embedding”). The same with a \(1 \times k_t\) kernel (\(\mathrm{Conv}_C\)): one token per fNIRS channel.

  • cls_token_*, pos_embedding_*, transformer_* Transformer Encoder. Each level prepends a learned class token, adds a learned positional embedding (one row per token, the token’s only identity), applies dropout and runs depth pre-norm blocks of multi-head self-attention and a GELU feed-forward layer of width mlp_dim. The queries, keys and values come from bias-free linear maps, and each of the num_heads heads of width \(d_h\) = head_dim computes, as in the authors’ Attention,

    \[\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left( \frac{Q K^{\top}}{\sqrt{d_h}} \right) V .\]
  • final_norm, final_layer Head. The class token of each level (or, with pool="mean", the mean of its tokens) is taken, the two vectors are concatenated (“Concat”), layer-normalized and classified by a linear layer (the authors’ mlp_head).

Temporal, Spatial, and Spectral Encoding

  • Temporal: the convolutions slide a \(k_t\)-sample kernel with stride \(s_t\) along time, and the resulting values become features of a token. The README’s segments are 140 samples (dataset A), 200 (B) and 256 (C). Kernels count samples, so sfreq is unused.

  • Spatial: channels or groups of \(k_s\) consecutive channels, identified by their position in the token sequence; no montage or optode geometry. The channel order must be the one used in training.

  • Chromophores: HbO and HbR are the two input planes, mixed by both convolutions from the first layer.

  • Spectral: none in the model; fNIRS-T expects band-pass filtered, baseline-corrected segments, and FNIRSPreT learns that step.

Pretrained Weights

None: the authors release code, not weights.

Differences from the reference implementation

x is (batch, n_chans, n_times), n_chans = 2 H, HbO rows then HbR rows, or MNE’s interleaved order read from chs_info. The figure draws an “MLP Head” per level before “Concat”; the authors’ code, which this module follows, concatenates first and applies one head, and the output is logits. The positional tables have one row per token (the authors allocate 101 rows and slice), so an authors’ state_dict loads after a conversion rather than a mapping (to_qkv split into queries/keys/values, tables cut, mlp_head.{0,1} to final_norm/final_layer); the port then matches the authors’ model to float32 rounding (max-abs below 1e-6 on their dataset A, B and C configurations). The attention always projects its output (the authors skip it when num_heads=1 and head_dim=embed_dim). mask is an addition of the port; ch_mask is refused, as the spatial kernel and the positional tables span the whole montage.

Parameters:
  • n_outputs (int | None) – Number of outputs of the model. This is the number of classes in the case of classification.

  • n_chans (int | None) – Number of EEG channels.

  • chs_info (list[dict] | None) – Information about each individual EEG channel. This should be filled with info["chs"]. Refer to mne.Info for more details.

  • n_times (int | None) – Number of time samples of the input window.

  • input_window_seconds (float | None) – Length of the input window in seconds.

  • sfreq (float | None) – Sampling frequency of the EEG recordings.

  • embed_dim (int) – \(D\), the authors’ dim, “last dimension of output tensor after linear transformation”; 64 for their datasets A and B, 128 for C.

  • depth (int) – Authors’ depth, “number of Transformer blocks” per level.

  • num_heads (int) – Authors’ heads, “number of the multi-head self-attention”.

  • mlp_dim (int) – Authors’ mlp_dim, “dimension of the MLP layer”.

  • pool (Literal['cls', 'mean']) – Read-out of each level (authors’ pool): "cls", its class token; "mean", the mean over its tokens (class token included).

  • head_dim (int) – \(d_h\), the authors’ dim_head: width of each attention head.

  • n_conv_filters (int) – \(C_S = C_C\), feature maps of both convolutions (authors: 8).

  • spatial_kernel_size (int) – \(k_s\), height of \(\mathrm{Conv}_S\) in HbO/HbR pairs (authors: 5); at most the number of pairs, n_chans // 2.

  • temporal_kernel_size (int) – \(k_t\), temporal kernel of both convolutions, in samples (authors: 30); n_times must be at least this.

  • temporal_stride (int) – \(s_t\), temporal stride of both convolutions, in samples (authors: 4).

  • channel_order (Optional[Literal['stacked', 'interleaved']]) – [HbO_1..HbO_H, HbR_1..HbR_H] or MNE’s [HbO_1, HbR_1, ...]; None reads it from chs_info.

  • drop_prob (float) – Authors’ dropout in the attention and feed-forward sub-layers.

  • embed_drop_prob (float) – Authors’ emb_dropout, “dropout for patch embeddings”.

  • activation (type[Module]) – Activation of the feed-forward sub-layers (authors: GELU).

Raises:

ValueError – If some input signal-related parameters are not specified: and can not be inferred.

Notes

The authors’ code uses three datasets: A (mental arithmetic vs rest, 8 subjects, 52 channels, 140 samples), B (Shin et al. 2017, same task, 29 subjects, 36 channels, 200 samples) and C (Bak et al. 2019, finger and foot tapping, 3 classes, 30 subjects, 20 channels, 256 samples). The fNIRS-T inputs are band-passed (0.01-0.1 Hz on B and C), baseline-corrected and z-scored per window. Training is 5 x 5-fold cross-validation over pooled trials or leave-one-subject-out, 120 epochs, batch 128, AdamW with cosine annealing (T_max=30) and label smoothing 0.1 in both loops, and the best test-fold accuracy is kept. On B and C the loss is “flooded” (loss = (loss - b).abs() + b) with a level \(b\) the README decays “at the specified epoch to obtain continuous regularization”: 0.45, 0.40 and 0.35 from the zero-based epochs 0, 30 and 50.

Replication on dataset C with the authors’ KFold_Train.py (mean of the 25 runs of each seed, averaged over seeds 0-2; macro-F1 at the kept epoch):

Model

Paper (abstract)

Port (acc / F1)

Authors’ code

fNIRS-T

75.49 %

75.03 % / 0.7498

74.73 % / 0.7469

fNIRS-PreT

“competitive”

72.50 % / 0.7244

72.50 % / 0.7245

Added in version 2.1.

License

References

Hugging Face Hub integration

When the optional huggingface_hub package is installed, all models automatically gain the ability to be pushed to and loaded from the Hugging Face Hub. Install with:

pip install braindecode[hub]

Pushing a model to the Hub:

from braindecode.models import FNIRST

# Train your model
model = FNIRST(n_chans=22, n_outputs=4, n_times=1000)
# ... training code ...

# Push to the Hub; ``datasets`` and ``base_model`` go to the card
model.push_to_hub(
    repo_id="username/my-fnirst-model",
    commit_message="Initial model upload",
    model_card_kwargs={"datasets": ["BNCI2014_001"]},
)

Loading a model from the Hub:

from braindecode.models import FNIRST

# Load pretrained model
model = FNIRST.from_pretrained("username/my-fnirst-model")
# Data the weights were trained on, from the Hub model card
print(model.card_data.datasets if model.card_data else None)

# Load with a different number of outputs (head is rebuilt automatically)
model = FNIRST.from_pretrained("username/my-fnirst-model", n_outputs=4)

Extracting features and replacing the head:

import torch

x = torch.randn(1, model.n_chans, model.n_times)
# Extract encoder features (consistent dict across all models)
out = model(x, return_features=True)
features = out["features"]

# Replace the classification head
model.reset_head(n_outputs=10)

Saving and restoring full configuration:

import json

config = model.get_config()            # all __init__ params
with open("config.json", "w") as f:
    json.dump(config, f)

model2 = FNIRST.from_config(config)    # reconstruct (no weights)

All model parameters (both EEG-specific and model-specific such as dropout rates, activation functions, number of filters) are automatically saved to the Hub and restored when loading.

See Loading and Adapting Pretrained Foundation Models for a complete tutorial.

Methods

forward(x, return_features=False, ch_mask=None, mask=None)[source]#

Forward pass of the fNIRS-T (and fNIRS-PreT) model.

Parameters:
  • x (Tensor) – Input tensor with shape (batch_size, n_chans, n_times).

  • return_features (bool) – If True, return the normalized pooled features and the class tokens of the two levels.

  • ch_mask (Tensor | None) – Not supported: the spatial kernel and the positional tables span every channel.

  • mask (Tensor | None) – Boolean tensor with the shape of x; True samples are set to zero before the model reads them.

Returns:

Logits with shape (batch_size, n_outputs), or with return_features=True the dict {"features", "cls_token"}, both of shape (batch_size, 2 * embed_dim).

Return type:

Tensor | dict[str, Tensor | None]

reset_head(n_outputs)[source]#

Replace the classification head for a new number of outputs.

This is called automatically by from_pretrained() when the user passes an n_outputs that differs from the saved config. Override in subclasses that need a model-specific head structure. Implementations keep changed constructor arguments in sync with self._update_init_kwargs, so that a saved model can be loaded back. Implementations requiring positive outputs can also use self._set_n_outputs to validate and record the new value.

Parameters:

n_outputs (int) – New number of output classes.

Return type:

None

Examples

>>> from braindecode.models import BENDR
>>> model = BENDR(n_chans=22, n_times=1000, n_outputs=4)
>>> model.reset_head(10)
>>> model.n_outputs
10

Added in version 1.4.