braindecode.models.FNIRS4D#

class braindecode.models.FNIRS4D(n_outputs=None, n_chans=None, chs_info=None, n_times=None, input_window_seconds=None, sfreq=None, *, n_layers=4, d_state=64, avg_size=10, drop_prob=0.5, prenorm=False, compute_hbt=True, channel_order=None, mean_branch='multiscale', activation=<class 'torch.nn.modules.activation.GELU'>, branch_activation=<class 'torch.nn.modules.activation.ELU'>)[source]#

fNIRS4D from Wang et al. (2026).

Convolution Recurrent

fNIRS4D architecture

fig/fig2.png of wzhlearning/fNIRS4D (eb13e5e, Apache-2.0), resized: preprocessing, “fNIRS4D for Signal Classification” with its “Selectable Branches” MAMHB (inter-subject) and MHB (intra-subject), and the three task paradigms. The paper (IEEE TIM 2026) is not open access; its caption is not reproduced.#

From the abstract [1]: “many studies focus solely on deep neural networks (DNNs) but neglect the importance of fNIRS domain knowledge. We refine domain knowledge into neuroscientific knowledge and expert prior. This article aims to integrate domain knowledge into DNNs, including activation patterns, hemodynamic responses, and feature engineering.”

The backbone is a state space model: “Diagonally structured state space sequence model (S4D), simple and efficient diagonal versions of SSMs, are introduced to the fNIRS domain. We incorporate domain knowledge into an S4D-based lightweight fNIRS classification model named fNIRS4D.” The S4D layer is that of Gu et al. [2]. The repository’s fig/fig1.png ties each kind of knowledge to a block: activation patterns to the “Mixed Hemoglobin” input, hemodynamic responses to Knowledge Embedding (KE), feature engineering to the selectable mean hemoglobin branches, with Knowledge Aggregation (KA) over all of them. Only the abstract, README, figures and code are public, and the text below comes from them.

Architecture Overview

With \(H\) = n_chans rows (d_model in the authors’ S4D), \(T\) = n_times samples, \(N\) = n_layers and the figure’s names:

input: HbO and HbR rows                      (batch, H, T)
Mixed Hemoglobin [HbO, HbT], to_fnirs_tensor (batch, 1, H, T)
Selectable branch (mean_hemoglobin_branch):
  MHB or MAMHB                               (batch, H)
Knowledge Embedding (fNIRS_knowledge_embedding):
  Conv_1 (1 x T), IN, Conv_1x1, Rearrange    (batch, H, H)
Knowledge Aggregation:
  N x (S4D, dropout, residual, LN)           (batch, H, H)
  AP_4: mean over the sequence               (batch, H)
concatenate with the branch, final_layer FC  (batch, n_outputs)

Macro Components

  • compute_hbt Mixed Hemoglobin. The rows become HbO and total hemoglobin, \(\mathrm{HbT} = \mathrm{HbO} + \mathrm{HbR}\), as in the authors’ dataloader.py; HbO/HbR pairs come from channel_order or chs_info.

  • fNIRS_knowledge_embedding Knowledge Embedding (KE). A convolution with \(H\) filters of \(1 \times T\), so each filter reads the whole window of one row (the kernel shape of FNIRSNet’s DHRConv), instance normalization and a \(1 \times 1\) convolution. The result is rearranged so that each row becomes a sequence of \(H\) filter responses, the sequence the S4D layers read.

  • s4d_layers, norms Knowledge Aggregation (KA). \(N\) S4D blocks, each S4D, dropout, a residual sum and a LayerNorm over the rows (post-norm, or before the layer with prenorm=True), then the mean over the sequence (\(\mathrm{AP}_4\)). Each S4D layer runs one diagonal state space model per row. Gu et al. start from the continuous system (their Eq. (1)), discretize it with the zero-order hold (Section 3.1),

    \[x'(t) = A x(t) + B u(t), \quad y(t) = C x(t), \qquad \bar{A} = e^{\Delta A}, \quad \bar{B} = (\Delta A)^{-1} (e^{\Delta A} - I) \, \Delta B,\]

    and apply it as a convolution with a Vandermonde kernel (Eqs. (6)-(7)):

    \[y = u * \bar{K}, \qquad \bar{K}_\ell = \sum_{n} C_n \bar{A}_n^{\ell} \bar{B}_n .\]

    \(A\) is diagonal and complex, initialized with S4D-Lin, \(A_n = -\tfrac{1}{2} + i \pi n\) (Eq. (9)); the step \(\Delta\) is learned per row from a log-uniform start in [0.001, 0.1]. As in the authors’ S4DKernel, \(B = 1\), the sum keeps d_state // 2 entries and the kernel is twice its real part (“same as adding conjugate pairs”, Listing 1 of Gu et al.). The layer adds a skip term \(D u\), then GELU, dropout, and a \(1 \times 1\) convolution with a GLU.

  • mean_hemoglobin_branch MHB and MAMHB. The intra-subject Mean Hemoglobin Branch (mean_branch="time_mean") is the mean of each row over time (\(\mathrm{AP}_1\)). The inter-subject Multi-scale Aggregated Mean Hemoglobin Branch ("multiscale") averages all rows into one signal (\(\mathrm{AP}_2\), avg_1) and reads it twice, directly and after an avg_size-sample moving average (\(\mathrm{AP}_3\), avg_2), each time with a full-window convolution of \(H/2\) filters, BN and ELU (conv_1/bn_1, conv_2/bn_2); the two halves are concatenated.

  • final_layer FC. One linear layer on the \(H\) KA features and the \(H\) branch features.

Temporal, Spatial, and Spectral Encoding

  • Temporal: one 3 s window (40 samples on UFFT, 30 on MA and WG in the authors’ scripts). KE reads all samples of a row at once, so the S4D layers run along the KE filters, not along time; the branches summarize time by means and full-window convolutions. Kernels count samples, so there is no sampling-rate argument.

  • Spatial: no montage or optode geometry. Each row is one S4D channel, LayerNorm normalizes across rows and MAMHB averages them; the row order must be the one used in training.

  • Chromophores: HbO and HbT; compute_hbt=False takes the rows as given.

  • Spectral: none in the model; the authors band-pass (UFFT, MA) or low-pass (WG) the signals first.

Pretrained Weights

None: the authors release code, not weights.

Differences from the reference implementation

x is (batch, n_chans, n_times); to_fnirs_tensor adds the authors’ singleton axis, and HbT is formed inside the model (the authors sum before their z-score; to reproduce it, pass their [HbO, HbT] arrays with compute_hbt=False). The S4D kernel is applied as a causal convolution in real arithmetic instead of a complex FFT, equal up to float32 rounding and free of complex tensors, so torch.jit.script and torch.export run on it; nn.Dropout1d replaces the authors’ DropoutNd, and their unused _optim weight-decay hints are not set. Names are the authors’; mapping renames fc.0.*, so their state_dict loads with strict=True. mean_branch is their mode, and avg_size, d_state and the activations expose values their code fixes. A training batch of one sample runs the BN layers on their running statistics, and ch_mask raises a ValueError.

Parameters:
  • n_outputs (int | None) – Number of outputs of the model. This is the number of classes in the case of classification.

  • n_chans (int | None) – Number of EEG channels.

  • chs_info (list[dict] | None) – Information about each individual EEG channel. This should be filled with info["chs"]. Refer to mne.Info for more details.

  • n_times (int | None) – Number of time samples of the input window.

  • input_window_seconds (float | None) – Length of the input window in seconds.

  • sfreq (float | None) – Sampling frequency of the EEG recordings.

  • n_layers (int) – Number of S4D blocks, the figure’s \(N\); the authors use 4, and 1 for WG subject-independent.

  • d_state (int) – State size of each S4D kernel (the authors’ d_state), even: the kernel keeps d_state // 2 conjugate pairs.

  • avg_size (int) – Width of the \(\mathrm{AP}_3\) moving average of the "multiscale" branch (the authors’ avg_size_2), at most n_times.

  • drop_prob (float) – Dropout in and after each S4D layer (the authors’ dropout).

  • prenorm (bool) – LayerNorm before each S4D layer instead of after the residual (the authors’ default is post-norm).

  • compute_hbt (bool) – Replace HbR by HbT = HbO + HbR (even n_chans).

  • channel_order (Optional[Literal['stacked', 'interleaved']]) – [HbO..., HbR...] (the authors’) or [HbO_1, HbR_1, ...] (MNE); None reads chs_info, else assumes "stacked". Ignored when compute_hbt=False.

  • mean_branch (Literal['multiscale', 'time_mean']) – The selectable branch (the authors’ mode): MAMHB for "LOSO-CV" (needs an even n_chans) or MHB for "KFold-CV".

  • activation (type[Module]) – Activation inside each S4D layer.

  • branch_activation (type[Module]) – Activation of the "multiscale" branch.

Raises:

ValueError – If some input signal-related parameters are not specified: and can not be inferred.

Notes

The authors’ code trains on UFFT [3] (n_chans = n_times = 40), MA [4] and WG [5] (72 rows, 30 samples), with their BBCI preprocessing, 3 s windows with a 1 s step, HbT per window and a z-score from the training set. Subject-specific runs (MHB) use 5-fold CV over trial blocks, 120 epochs, AdamW (lr 0.001 on UFFT, 0.0005 on MA and WG) decayed by 0.2 at epochs 20, 40 and 60; subject-independent runs (MAMHB) leave one subject out, batch 512, cosine lr over 30 epochs. Last-epoch test accuracy.

In the authors’ training loops, on the same UFFT and WG arrays, splits and seeds (0-2), the port matches the authors’ code: accuracy in %, paired difference in points with its 95 % interval, all including zero.

Cell

This port

Authors’ code

Paired diff [95% CI]

UFFT KFold

71.98

72.00

-0.01 [-0.49, +0.46]

UFFT LOSO

65.81

65.85

-0.04 [-0.32, +0.24]

WG KFold

61.54

61.33

+0.21 [-0.50, +0.91]

WG LOSO

61.92

62.00

-0.07 [-0.54, +0.39]

In the GeoNIRS benchmark [6] (macro-F1 in %, 12 s windows, subject-level 5-fold CV, seeds 810 and 811, "multiscale"), the port gives 67.89 on DS42 action mode and 49.86 on DS19 pain intensity, against 68.98 and 50.79 in GeoNIRS Table 14 and 69.03 and 50.60 for the GeoNIRS copy of fNIRS4D in the same loop.

Added in version 2.1.

License

References

Hugging Face Hub integration

When the optional huggingface_hub package is installed, all models automatically gain the ability to be pushed to and loaded from the Hugging Face Hub. Install with:

pip install braindecode[hub]

Pushing a model to the Hub:

from braindecode.models import FNIRS4D

# Train your model
model = FNIRS4D(n_chans=22, n_outputs=4, n_times=1000)
# ... training code ...

# Push to the Hub; ``datasets`` and ``base_model`` go to the card
model.push_to_hub(
    repo_id="username/my-fnirs4d-model",
    commit_message="Initial model upload",
    model_card_kwargs={"datasets": ["BNCI2014_001"]},
)

Loading a model from the Hub:

from braindecode.models import FNIRS4D

# Load pretrained model
model = FNIRS4D.from_pretrained("username/my-fnirs4d-model")
# Data the weights were trained on, from the Hub model card
print(model.card_data.datasets if model.card_data else None)

# Load with a different number of outputs (head is rebuilt automatically)
model = FNIRS4D.from_pretrained("username/my-fnirs4d-model", n_outputs=4)

Extracting features and replacing the head:

import torch

x = torch.randn(1, model.n_chans, model.n_times)
# Extract encoder features (consistent dict across all models)
out = model(x, return_features=True)
features = out["features"]

# Replace the classification head
model.reset_head(n_outputs=10)

Saving and restoring full configuration:

import json

config = model.get_config()            # all __init__ params
with open("config.json", "w") as f:
    json.dump(config, f)

model2 = FNIRS4D.from_config(config)    # reconstruct (no weights)

All model parameters (both EEG-specific and model-specific such as dropout rates, activation functions, number of filters) are automatically saved to the Hub and restored when loading.

See Loading and Adapting Pretrained Foundation Models for a complete tutorial.

Methods

forward(x, return_features=False, ch_mask=None)[source]#

Forward pass of the fNIRS4D model.

Parameters:
  • x (Tensor) – Input tensor with shape (batch_size, n_chans, n_times): HbO and HbR rows, or the rows as given with compute_hbt=False.

  • return_features (bool) – If True, return the features that feed the FC layer.

  • ch_mask (Tensor | None) – Not supported: every layer is sized by n_chans.

Returns:

Logits with shape (batch_size, n_outputs), or with return_features=True the dict {"features", "cls_token"}, features of shape (batch_size, 2 * n_chans) and cls_token None.

Return type:

Tensor | dict[str, Tensor | None]

reset_head(n_outputs)[source]#

Replace the classification head for a new number of outputs.

This is called automatically by from_pretrained() when the user passes an n_outputs that differs from the saved config. Override in subclasses that need a model-specific head structure. Implementations keep changed constructor arguments in sync with self._update_init_kwargs, so that a saved model can be loaded back. Implementations requiring positive outputs can also use self._set_n_outputs to validate and record the new value.

Parameters:

n_outputs (int) – New number of output classes.

Return type:

None

Examples

>>> from braindecode.models import BENDR
>>> model = BENDR(n_chans=22, n_times=1000, n_outputs=4)
>>> model.reset_head(10)
>>> model.n_outputs
10

Added in version 1.4.