braindecode.models.VEMG2Pose#

class braindecode.models.VEMG2Pose(n_outputs=None, n_chans=None, chs_info=None, n_times=None, input_window_seconds=None, sfreq=None, *, encoder_channels=256, feature_dim=64, hidden_size=512, lstm_layers=2, decoder='lstm', decoder_hidden_sizes=(512, 512), stem_kernel_sizes=(11, 5), stem_strides=(5, 2), tds_subsample_kernels=(17, 9), tds_subsample_strides=(4, 2), tds_kernel_widths=(9, 5), tds_blocks=2, tds_channels=16, rollout_rate=50.0, num_position_steps=500, parameterization='hybrid', output_scalar=0.01, activation=<class 'torch.nn.modules.activation.LeakyReLU'>, drop_prob=0.0)[source]#

VEMG2Pose from Salter et al (2024) [salter2024].

Convolution Recurrent

emg2pose wristband and predicted hand pose

Figure 1 of Salter et al. (2024). The paper describes vemg2pose in prose (Section 3.5, Appendix C) and publishes no architecture diagram, so this is its overview figure.#

Time-depth-separable (TDS) convolutional encoder [hannun2019] feeding a state-conditioned LSTM that rolls the hand pose out at 50 Hz. The decoder emits a position and a velocity for every joint at each step: it takes the position for the first num_position_steps, then integrates the velocity from there on. That hybrid is what the leading v names.

Architecture Overview

(B, n_chans, T) → two strided Conv1d blocks (kernel 11 stride 5, then 5 / 2, LayerNorm over channels) → two TDS stages, each a strided subsampling convolution followed by tds_blocks pairs of a time-depth-separable 2-D convolution and a residual feed-forward block → a linear projection to feature_dim → linear resampling to rollout_rate → LSTM rollout conditioned on the previous pose → (B, T - left_context, n_outputs).

Every convolution is valid (no padding), so the encoder consumes a left_context of 1790 samples at the default schedule. It returns predictions for the valid interval [left_context, T).

Macro Components

VEMG2Pose.encoder

Operations. _Conv1dBlock ×2 then _TdsStage ×2. A stage is a strided Conv1d + LayerNorm, then tds_blocks × ( TDSConv2dBlock, TDSFullyConnectedBlock), both residual and each closing with a LayerNorm over the channel axis. Role. Compresses 2 kHz sEMG to feature_dim features at sfreq / prod(strides), i.e. 25 Hz for the paper’s input.

VEMG2Pose.lstm / VEMG2Pose.final_layer

Operations. Two-layer LSTM over [feature ; previous pose], then LeakyReLU and a linear map to 2 * n_outputs, scaled by output_scalar. The halves are the position and the velocity. Role. Autoregressive pose rollout with state feedback.

Temporal, Spatial, and Spectral Encoding

  • Time: Strided convolutions then an explicit 50 Hz rollout.

  • Channels/space: The first convolution mixes all electrodes; the TDS blocks then convolve within a channel/width factorisation.

  • Frequency: Learned from the raw waveform; nothing is filtered out.

Additional Mechanisms

Passing y0 conditions the rollout on a known initial pose (the paper’s tracking regime). Omitting it starts from zeros, which is the regression regime the released regression_vemg2pose.ckpt was trained under.

Pre-trained weights

Meta’s released checkpoints are rehosted with these parameter names:

# vemg2pose: recurrent decoder
VEMG2Pose.from_pretrained("braindecode/VEMG2Pose-emg2pose")
VEMG2Pose.from_pretrained("braindecode/VEMG2Pose-emg2pose-tracking")
# emg2pose: same encoder, stateless MLP decoder
VEMG2Pose.from_pretrained("braindecode/EMG2Pose-emg2pose")
VEMG2Pose.from_pretrained("braindecode/EMG2Pose-emg2pose-tracking")

Each config records its own decoder and parameterization, so the right rollout is restored automatically.

The rehosted weights reproduce the reference implementation exactly and remain under emg2pose’s CC BY-NC-SA 4.0 license.

Added in version 1.8.

Parameters:

  • n_outputs (int) – Number of outputs of the model. This is the number of classes in the case of classification.

  • n_chans (int) – Number of EEG channels.

  • chs_info (list of dict) – Information about each individual EEG channel. This should be filled with info["chs"]. Refer to mne.Info for more details.

  • n_times (int) – Number of time samples of the input window.

  • input_window_seconds (float) – Length of the input window in seconds.

  • sfreq (float) – Sampling frequency of the EEG recordings.

  • encoder_channels (int) – Width of the strided stem convolutions and of the TDS stages. The default is 256.

  • feature_dim (int) – Width the encoder projects to before the decoder. The default is 64.

  • hidden_size (int) – LSTM hidden width. The default is 512.

  • lstm_layers (int) – Depth of the decoder LSTM, when decoder='lstm'. The default is 2.

  • decoder (str) – Rollout decoder. 'lstm' (the default) is the recurrent decoder the vemg2pose checkpoints use. 'mlp' is the stateless normalised MLP behind emg2pose’s own emg2pose baseline; both are state-conditioned on the previous pose either way.

  • decoder_hidden_sizes (tuple[int, ...]) – Hidden widths of the MLP decoder, unused when decoder='lstm'. The default is (512, 512).

  • stem_kernel_sizes (tuple[int, ...]) – Kernel of each strided stem convolution. The default is (11, 5).

  • stem_strides (tuple[int, ...]) – Stride of each stem convolution. The default is (5, 2).

  • tds_subsample_kernels (tuple[int, ...]) – Subsampling kernel of each TDS stage. The default is (17, 9).

  • tds_subsample_strides (tuple[int, ...]) – Subsampling stride of each TDS stage. The default is (4, 2).

  • tds_kernel_widths (tuple[int, ...]) – Temporal kernel inside each stage’s TDS blocks, which must be odd. The default is (9, 5).

  • tds_blocks (int) – Conv/feed-forward block pairs per TDS stage. The default is 2.

  • tds_channels (int) – Channel count of the TDS factorisation; the width is encoder_channels // tds_channels. The default is 16.

  • rollout_rate (float) – Rate in Hz at which the decoder is unrolled. The default is 50.0.

  • num_position_steps (int) – Input samples for which the decoder’s position output is used directly, before it switches to integrating velocity. The default is 500 (250 ms at 2 kHz).

  • parameterization (str) – How the decoder output becomes a pose. 'hybrid' (the default, and what regression_vemg2pose.ckpt was trained as) emits a position and a velocity per joint, taking the position for num_position_steps and integrating the velocity thereafter. 'velocity' integrates a single velocity from y0, which is the tracking_vemg2pose.ckpt setting. 'position' reads the pose out directly.

  • output_scalar (float) – Fixed multiplier on the decoder output. The default is 0.01.

  • activation (type[Module]) – Activation before the output projection. The default is nn.LeakyReLU.

  • drop_prob (float) – Dropout inside the convolution blocks. The default is 0.0.

Raises:

ValueError – If some input signal-related parameters are not specified: and can not be inferred.

Notes

If some input signal-related parameters are not specified, there will be an attempt to infer them from the other parameters.

References

[salter2024]

Salter, Warren, Schlager, Spurr, Han, Bhasin, Rohin, et al. (2024). emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation. NeurIPS Datasets and Benchmarks. arXiv:2412.02725.

[hannun2019]

Hannun et al. (2019). Sequence Modeling with Time-Depth Separable Convolutions. arXiv:1904.01619.

Hugging Face Hub integration

When the optional huggingface_hub package is installed, all models automatically gain the ability to be pushed to and loaded from the Hugging Face Hub. Install with:

pip install braindecode[hub]

Pushing a model to the Hub:

from braindecode.models import VEMG2Pose

# Train your model
model = VEMG2Pose(n_chans=22, n_outputs=4, n_times=1000)
# ... training code ...

# Push to the Hub; ``datasets`` and ``base_model`` go to the card
model.push_to_hub(
    repo_id="username/my-vemg2pose-model",
    commit_message="Initial model upload",
    model_card_kwargs={"datasets": ["BNCI2014_001"]},
)

Loading a model from the Hub:

from braindecode.models import VEMG2Pose

# Load pretrained model
model = VEMG2Pose.from_pretrained("username/my-vemg2pose-model")
# Data the weights were trained on, from the Hub model card
print(model.card_data.datasets if model.card_data else None)

# Load with a different number of outputs (head is rebuilt automatically)
model = VEMG2Pose.from_pretrained("username/my-vemg2pose-model", n_outputs=4)

Extracting features and replacing the head:

import torch

x = torch.randn(1, model.n_chans, model.n_times)
# Extract encoder features (consistent dict across all models)
out = model(x, return_features=True)
features = out["features"]

# Replace the classification head
model.reset_head(n_outputs=10)

Saving and restoring full configuration:

import json

config = model.get_config()            # all __init__ params
with open("config.json", "w") as f:
    json.dump(config, f)

model2 = VEMG2Pose.from_config(config)    # reconstruct (no weights)

All model parameters (both EEG-specific and model-specific such as dropout rates, activation functions, number of filters) are automatically saved to the Hub and restored when loading.

See Loading and Adapting Pretrained Foundation Models for a complete tutorial.

Methods

forward(x, y0=None)[source]#

Decode hand pose from raw EMG.

Parameters:

  • x (Tensor) – Raw sEMG window at sfreq.

  • y0 (Tensor | None) – Ground-truth initial pose (Tracking mode). Falls back to zeros (Regression mode) when omitted, which is how the released regression checkpoint was trained.

Return type:

Tensor

reset_head(n_outputs)[source]#

Swap the output layer for a new output dimensionality.

The decoded pose feeds back as decoder input, so changing n_outputs also rebuilds the recurrent stack; its weights are re-initialised.

Parameters:

n_outputs (int) – New number of output classes.

Return type:

None

Examples

>>> from braindecode.models import BENDR
>>> model = BENDR(n_chans=22, n_times=1000, n_outputs=4)
>>> model.reset_head(10)
>>> model.n_outputs
10

Added in version 1.4.

tracking_forward(x, y0)[source]#

Explicit Tracking-mode entry point (ground-truth anchor required).

Parameters:

  • x (Tensor) – The description is missing.

  • y0 (Tensor) – The description is missing.

Return type:

Tensor

Examples using braindecode.models.VEMG2Pose#

Loading and Adapting Pretrained Foundation Models

Loading and Adapting Pretrained Foundation Models