braindecode.models.FNIRS4D#
- class braindecode.models.FNIRS4D(n_outputs=None, n_chans=None, chs_info=None, n_times=None, input_window_seconds=None, sfreq=None, *, n_layers=4, d_state=64, avg_size=10, drop_prob=0.5, prenorm=False, compute_hbt=True, channel_order=None, mean_branch='multiscale', activation=<class 'torch.nn.modules.activation.GELU'>, branch_activation=<class 'torch.nn.modules.activation.ELU'>)[source]#
fNIRS4D from Wang et al. (2026).
Convolution Recurrent
fig/fig2.pngof wzhlearning/fNIRS4D (eb13e5e, Apache-2.0), resized: preprocessing, “fNIRS4D for Signal Classification” with its “Selectable Branches” MAMHB (inter-subject) and MHB (intra-subject), and the three task paradigms. The paper (IEEE TIM 2026) is not open access; its caption is not reproduced.#From the abstract [1]: “many studies focus solely on deep neural networks (DNNs) but neglect the importance of fNIRS domain knowledge. We refine domain knowledge into neuroscientific knowledge and expert prior. This article aims to integrate domain knowledge into DNNs, including activation patterns, hemodynamic responses, and feature engineering.”
The backbone is a state space model: “Diagonally structured state space sequence model (S4D), simple and efficient diagonal versions of SSMs, are introduced to the fNIRS domain. We incorporate domain knowledge into an S4D-based lightweight fNIRS classification model named fNIRS4D.” The S4D layer is that of Gu et al. [2]. The repository’s
fig/fig1.pngties each kind of knowledge to a block: activation patterns to the “Mixed Hemoglobin” input, hemodynamic responses to Knowledge Embedding (KE), feature engineering to the selectable mean hemoglobin branches, with Knowledge Aggregation (KA) over all of them. Only the abstract, README, figures and code are public, and the text below comes from them.Architecture Overview
With \(H\) =
n_chansrows (d_modelin the authors’ S4D), \(T\) =n_timessamples, \(N\) =n_layersand the figure’s names:input: HbO and HbR rows (batch, H, T) Mixed Hemoglobin [HbO, HbT], to_fnirs_tensor (batch, 1, H, T) Selectable branch (mean_hemoglobin_branch): MHB or MAMHB (batch, H) Knowledge Embedding (fNIRS_knowledge_embedding): Conv_1 (1 x T), IN, Conv_1x1, Rearrange (batch, H, H) Knowledge Aggregation: N x (S4D, dropout, residual, LN) (batch, H, H) AP_4: mean over the sequence (batch, H) concatenate with the branch, final_layer FC (batch, n_outputs)
Macro Components
compute_hbtMixed Hemoglobin. The rows become HbO and total hemoglobin, \(\mathrm{HbT} = \mathrm{HbO} + \mathrm{HbR}\), as in the authors’dataloader.py; HbO/HbR pairs come fromchannel_orderorchs_info.fNIRS_knowledge_embeddingKnowledge Embedding (KE). A convolution with \(H\) filters of \(1 \times T\), so each filter reads the whole window of one row (the kernel shape ofFNIRSNet’s DHRConv), instance normalization and a \(1 \times 1\) convolution. The result is rearranged so that each row becomes a sequence of \(H\) filter responses, the sequence the S4D layers read.s4d_layers,normsKnowledge Aggregation (KA). \(N\) S4D blocks, each S4D, dropout, a residual sum and a LayerNorm over the rows (post-norm, or before the layer withprenorm=True), then the mean over the sequence (\(\mathrm{AP}_4\)). Each S4D layer runs one diagonal state space model per row. Gu et al. start from the continuous system (their Eq. (1)), discretize it with the zero-order hold (Section 3.1),\[x'(t) = A x(t) + B u(t), \quad y(t) = C x(t), \qquad \bar{A} = e^{\Delta A}, \quad \bar{B} = (\Delta A)^{-1} (e^{\Delta A} - I) \, \Delta B,\]and apply it as a convolution with a Vandermonde kernel (Eqs. (6)-(7)):
\[y = u * \bar{K}, \qquad \bar{K}_\ell = \sum_{n} C_n \bar{A}_n^{\ell} \bar{B}_n .\]\(A\) is diagonal and complex, initialized with S4D-Lin, \(A_n = -\tfrac{1}{2} + i \pi n\) (Eq. (9)); the step \(\Delta\) is learned per row from a log-uniform start in [0.001, 0.1]. As in the authors’
S4DKernel, \(B = 1\), the sum keepsd_state // 2entries and the kernel is twice its real part (“same as adding conjugate pairs”, Listing 1 of Gu et al.). The layer adds a skip term \(D u\), then GELU, dropout, and a \(1 \times 1\) convolution with a GLU.mean_hemoglobin_branchMHB and MAMHB. The intra-subject Mean Hemoglobin Branch (mean_branch="time_mean") is the mean of each row over time (\(\mathrm{AP}_1\)). The inter-subject Multi-scale Aggregated Mean Hemoglobin Branch ("multiscale") averages all rows into one signal (\(\mathrm{AP}_2\),avg_1) and reads it twice, directly and after anavg_size-sample moving average (\(\mathrm{AP}_3\),avg_2), each time with a full-window convolution of \(H/2\) filters, BN and ELU (conv_1/bn_1,conv_2/bn_2); the two halves are concatenated.final_layerFC. One linear layer on the \(H\) KA features and the \(H\) branch features.
Temporal, Spatial, and Spectral Encoding
Temporal: one 3 s window (40 samples on UFFT, 30 on MA and WG in the authors’ scripts). KE reads all samples of a row at once, so the S4D layers run along the KE filters, not along time; the branches summarize time by means and full-window convolutions. Kernels count samples, so there is no sampling-rate argument.
Spatial: no montage or optode geometry. Each row is one S4D channel, LayerNorm normalizes across rows and MAMHB averages them; the row order must be the one used in training.
Chromophores: HbO and HbT;
compute_hbt=Falsetakes the rows as given.Spectral: none in the model; the authors band-pass (UFFT, MA) or low-pass (WG) the signals first.
Pretrained Weights
None: the authors release code, not weights.
Differences from the reference implementation
xis(batch, n_chans, n_times);to_fnirs_tensoradds the authors’ singleton axis, and HbT is formed inside the model (the authors sum before their z-score; to reproduce it, pass their [HbO, HbT] arrays withcompute_hbt=False). The S4D kernel is applied as a causal convolution in real arithmetic instead of a complex FFT, equal up to float32 rounding and free of complex tensors, sotorch.jit.scriptandtorch.exportrun on it;nn.Dropout1dreplaces the authors’DropoutNd, and their unused_optimweight-decay hints are not set. Names are the authors’;mappingrenamesfc.0.*, so theirstate_dictloads withstrict=True.mean_branchis theirmode, andavg_size,d_stateand the activations expose values their code fixes. A training batch of one sample runs the BN layers on their running statistics, andch_maskraises aValueError.- Parameters:
n_outputs (
int|None) – Number of outputs of the model. This is the number of classes in the case of classification.chs_info (
list[dict] |None) – Information about each individual EEG channel. This should be filled withinfo["chs"]. Refer tomne.Infofor more details.n_times (
int|None) – Number of time samples of the input window.input_window_seconds (
float|None) – Length of the input window in seconds.sfreq (
float|None) – Sampling frequency of the EEG recordings.n_layers (
int) – Number of S4D blocks, the figure’s \(N\); the authors use 4, and 1 for WG subject-independent.d_state (
int) – State size of each S4D kernel (the authors’d_state), even: the kernel keepsd_state // 2conjugate pairs.avg_size (
int) – Width of the \(\mathrm{AP}_3\) moving average of the"multiscale"branch (the authors’avg_size_2), at mostn_times.drop_prob (
float) – Dropout in and after each S4D layer (the authors’dropout).prenorm (
bool) – LayerNorm before each S4D layer instead of after the residual (the authors’ default is post-norm).compute_hbt (
bool) – Replace HbR by HbT = HbO + HbR (evenn_chans).channel_order (
Optional[Literal['stacked','interleaved']]) –[HbO..., HbR...](the authors’) or[HbO_1, HbR_1, ...](MNE);Nonereadschs_info, else assumes"stacked". Ignored whencompute_hbt=False.mean_branch (
Literal['multiscale','time_mean']) – The selectable branch (the authors’mode): MAMHB for"LOSO-CV"(needs an evenn_chans) or MHB for"KFold-CV".activation (
type[Module]) – Activation inside each S4D layer.branch_activation (
type[Module]) – Activation of the"multiscale"branch.
- Raises:
ValueError – If some input signal-related parameters are not specified: and can not be inferred.
Notes
The authors’ code trains on UFFT [3] (
n_chans=n_times= 40), MA [4] and WG [5] (72 rows, 30 samples), with their BBCI preprocessing, 3 s windows with a 1 s step, HbT per window and a z-score from the training set. Subject-specific runs (MHB) use 5-fold CV over trial blocks, 120 epochs, AdamW (lr 0.001 on UFFT, 0.0005 on MA and WG) decayed by 0.2 at epochs 20, 40 and 60; subject-independent runs (MAMHB) leave one subject out, batch 512, cosine lr over 30 epochs. Last-epoch test accuracy.In the authors’ training loops, on the same UFFT and WG arrays, splits and seeds (0-2), the port matches the authors’ code: accuracy in %, paired difference in points with its 95 % interval, all including zero.
Cell
This port
Authors’ code
Paired diff [95% CI]
UFFT KFold
71.98
72.00
-0.01 [-0.49, +0.46]
UFFT LOSO
65.81
65.85
-0.04 [-0.32, +0.24]
WG KFold
61.54
61.33
+0.21 [-0.50, +0.91]
WG LOSO
61.92
62.00
-0.07 [-0.54, +0.39]
In the GeoNIRS benchmark [6] (macro-F1 in %, 12 s windows, subject-level 5-fold CV, seeds 810 and 811,
"multiscale"), the port gives 67.89 on DS42 action mode and 49.86 on DS19 pain intensity, against 68.98 and 50.79 in GeoNIRS Table 14 and 69.03 and 50.60 for the GeoNIRS copy of fNIRS4D in the same loop.Added in version 2.1.
References
Hugging Face Hub integration
When the optional
huggingface_hubpackage is installed, all models automatically gain the ability to be pushed to and loaded from the Hugging Face Hub. Install with:pip install braindecode[hub]
Pushing a model to the Hub:
from braindecode.models import FNIRS4D # Train your model model = FNIRS4D(n_chans=22, n_outputs=4, n_times=1000) # ... training code ... # Push to the Hub; ``datasets`` and ``base_model`` go to the card model.push_to_hub( repo_id="username/my-fnirs4d-model", commit_message="Initial model upload", model_card_kwargs={"datasets": ["BNCI2014_001"]}, )
Loading a model from the Hub:
from braindecode.models import FNIRS4D # Load pretrained model model = FNIRS4D.from_pretrained("username/my-fnirs4d-model") # Data the weights were trained on, from the Hub model card print(model.card_data.datasets if model.card_data else None) # Load with a different number of outputs (head is rebuilt automatically) model = FNIRS4D.from_pretrained("username/my-fnirs4d-model", n_outputs=4)
Extracting features and replacing the head:
import torch x = torch.randn(1, model.n_chans, model.n_times) # Extract encoder features (consistent dict across all models) out = model(x, return_features=True) features = out["features"] # Replace the classification head model.reset_head(n_outputs=10)
Saving and restoring full configuration:
import json config = model.get_config() # all __init__ params with open("config.json", "w") as f: json.dump(config, f) model2 = FNIRS4D.from_config(config) # reconstruct (no weights)
All model parameters (both EEG-specific and model-specific such as dropout rates, activation functions, number of filters) are automatically saved to the Hub and restored when loading.
See Loading and Adapting Pretrained Foundation Models for a complete tutorial.
Methods
- forward(x, return_features=False, ch_mask=None)[source]#
Forward pass of the fNIRS4D model.
- Parameters:
- Returns:
Logits with shape
(batch_size, n_outputs), or withreturn_features=Truethe dict{"features", "cls_token"},featuresof shape(batch_size, 2 * n_chans)andcls_tokenNone.- Return type:
- reset_head(n_outputs)[source]#
Replace the classification head for a new number of outputs.
This is called automatically by
from_pretrained()when the user passes ann_outputsthat differs from the saved config. Override in subclasses that need a model-specific head structure. Implementations keep changed constructor arguments in sync withself._update_init_kwargs, so that a saved model can be loaded back. Implementations requiring positive outputs can also useself._set_n_outputsto validate and record the new value.Examples
>>> from braindecode.models import BENDR >>> model = BENDR(n_chans=22, n_times=1000, n_outputs=4) >>> model.reset_head(10) >>> model.n_outputs 10
Added in version 1.4.