Preprint · 2026
What Masking Geometry Works Best for EEG Foundation Models?
For MAE or JEPA pre-training, how do the temporal length (L), the spatial radius (r) and the ratio (ρ) of the mask influence model performance?
We answer this question with a controlled sweep of 58 models and linear probes of their learned representations.
The formalism: Three parameters cover the masking strategies of the EEG literature
Blocks of radius r on the scalp and length L in time are sampled such that a fraction ≈ ρ of the signal is hidden.
- K
- – blocks
- Masked
- –
Cursor
Loading 3-D view…
3-D view unavailable (WebGL is disabled in this browser).
L = 1, r = one gives random patch masking, r = all hides whole time segments and L = 33 (the whole 30 s window) hides whole channels. Everything in between is a spatio-temporal block mask.
The study: 58 models that differ only in their mask and pretext
MAE reconstructs the hidden EEG, JEPA predicts how a teacher network encodes it. Each frozen encoder is then scored by a linear probe on the 12 datasets of OpenEEGBench.
The model reconstructs the raw EEG of the hidden patches.
Everything outside the coloured box is shared by MAE and JEPA.
The model predicts how a teacher network encodes the hidden patches, not the raw EEG.
Everything outside the coloured box is shared by MAE and JEPA.
The mask, decoder and teacher are gone. The frozen encoder turns a new EEG recording into features, and a linear model predicts the label from them. The same protocol is used for all 58 models on the 12 OpenEEGBench datasets.
Kept from pre-training, and frozen: the tokeniser and the encoder.
Shared
EEG
One pre-training window.
30 s · 32 random channels · 1 s patches of 200 samples, 0.1 s overlap · robust scaling per window
EEG
A window from one of the 12 OpenEEGBench tasks, never masked.
6–64 channels · 1–30 s · any montage: only electrode positions are needed
Shared
Linear (tokeniser)
Maps each 1 s patch of one channel to one token, without mixing channels.
512 dims · + fixed encoding of 3D position and time
Linear (tokeniser)
The pre-trained tokeniser, frozen.
Shared
Token grid
All tokens of the window, one per channel and 1 s patch, before masking.
Token grid
All tokens of the window. Nothing is masked.
Pre-training
Masking strategy (L, r)
The object of the study: blocks of radius r on the scalp and L patches in time, sampled such that ρ ≈ 55 % of the tokens are hidden.
Pre-training
Mask
One random mask, redrawn for every window. Solid cells are the visible context, hatched cells the hidden targets.
Pre-training
Remove masked tokens
The encoder sees only the visible tokens, with no placeholder for the hidden ones (as in the original MAE).
Pre-training
Visible tokens
The 1 − ρ = 45 % of tokens left after masking, the model’s only context.
Shared
Transformer Encoder
The model kept after pre-training. It mixes the visible tokens across channels and time.
4 layers · 8 heads · 512 dims · ~12.7 M parameters (REVE-Small)
Transformer Encoder (frozen)
The pre-trained encoder, weights frozen. It now sees every token of the window.
Shared
Encoded visible tokens
Contextual embeddings of the visible tokens.
Encoded tokens
Last-layer embeddings, one per token.
channels × patches × 512 values
Pre-training
Transformer Decoder
Puts a learned mask token at each hidden position and fills it in from the encoder output. Pre-training only, identical for MAE and JEPA.
2 layers
Pre-training
Decoder outputs
Predictions for the hidden positions, still in the 512-dim latent space.
JEPA predictions
Predictions for the hidden positions, compared directly with the teacher’s embeddings.
MAE only
MAE-only branch
Target: the raw EEG. No teacher, no extra loss.
MAE only
Remove unmasked signal patches
Keeps only the hidden patches, so the loss covers only what the model could not see.
MAE only
MAE targets
The raw signal of the hidden patches, taken from the input. Fixed during training.
200 samples per patch
MAE only
Linear (output head)
Maps each decoder output back to a 200-sample signal patch.
MAE only
MAE predictions
The reconstructed signal at each hidden position.
MAE only
L2 MAE
Mean squared error on the hidden patches.
JEPA only
JEPA-only branch
Target: the teacher’s embedding of the hidden tokens.
JEPA only
EMA update
The teacher is not trained by gradients: after each step it moves slightly towards the student.
θteacher ← τ θteacher + (1 − τ) θstudent · τ: 0.999 → 1
JEPA only
Transformer Encoder (Teacher)
A copy of the encoder that sees the full, unmasked window. It is not trained via gradient backpropagation, but is updated via EMA. Discarded after pre-training.
JEPA only
Teacher embeddings
The teacher’s last-layer embedding of every token, computed with full context.
JEPA only
Remove unmasked tokens
Keeps only the teacher embeddings at the hidden positions.
JEPA only
Gradient stop
No gradient reaches the teacher. The teacher is updated via EMA instead.
JEPA only
JEPA targets
Teacher embeddings of the hidden tokens. They shift as the teacher follows the student.
512 dims
JEPA only
L2 JEPA
Mean squared error to the teacher embeddings, on hidden positions only.
Downstream only
Flattened features
All encoded tokens of a window, concatenated.
31,744 to 146,432 features depending on the dataset
Downstream only
Gaussian random projection
A fixed random map that keeps the probe cheap. Each probe seed draws a new one, the only randomness of the evaluation.
5,000 features · entries ~ 𝒩(0, 1/5000)
Downstream only
Projected features
The probe’s input.
5,000 features per window
Downstream only
Ridge regression (linear probe)
A linear model solved in closed form, without gradient descent. It measures what the frozen features already contain.
λ among 17 values, 10−8 to 108, chosen on the validation split
Downstream only
Prediction
A class, or a value for seed-vig (regression).
balanced accuracy · R² for seed-vig · normalised per dataset, averaged over the 12 · relative to REVE = a cell’s colour in the results
29 mask geometries (ρ fixed at 0.55) × 2 pretexts = 58 models
The results: MAE and JEPA agree on the best mask, L = 2 s and r = 9 cm
Each cell is one pre-trained model, evaluated on the 12 datasets of OpenEEGBench.
Both pretexts pick L = 2 s, r = 9 cm. Both fail when blocks are too local (r = one), too global (r = all) or, in most cases, too long (L = 8, 16).
11 MAE and 9 JEPA masks are statistically tied with the best (circles). A poor choice costs JEPA 2.4× more than MAE.
From epoch 3 on, JEPA gets worse as it trains: its encoder learns a lookup table of channel positions instead of reading the EEG. Standard collapse detectors miss it.
At the best mask, both models reach the level of the public REVE-Base checkpoint with 12.7 M parameters instead of 69 M. One pre-training run costs ~14 H100-hours, against 260 A100-hours for REVE-Base.
JEPA
| 1 | 2 | 4 | 8 | 16 | 33 | |
|---|---|---|---|---|---|---|
| one | −0.03 | +0.02 | −0.01 | −0.02 | −0.02 | −0.07 |
| 6 cm | +0.07 | +0.03 | +0.07 | +0.02 | −0.05 | −0.04 |
| 9 cm | +0.07 | +0.10 | +0.06 | −0.01 | −0.03 | −0.04 |
| 12 cm | −0.07 | +0.04 | +0.06 | −0.18 | −0.27 | +0.02 |
| all | −0.41 | −0.67 | −0.37 | −0.36 | −0.31 | not trained |
MAE
| 1 | 2 | 4 | 8 | 16 | 33 | |
|---|---|---|---|---|---|---|
| one | −0.02 | −0.01 | +0.02 | +0.04 | +0.02 | −0.00 |
| 6 cm | −0.02 | +0.06 | +0.07 | +0.02 | −0.13 | +0.04 |
| 9 cm | +0.01 | +0.07 | +0.05 | +0.03 | −0.16 | +0.07 |
| 12 cm | +0.01 | +0.03 | +0.02 | +0.03 | −0.10 | +0.05 |
| all | −0.05 | −0.08 | −0.10 | −0.14 | −0.25 | not trained |
Mean normalised score over the 12 OpenEEGBench datasets and the probe seeds, minus that of the public REVE-Base checkpoint (0 = REVE; linear probe on frozen features). Framed: the best mask of each pretext.
Mean rank (1 = best) of each mask among the 29 of its pretext, under a hierarchical bootstrap over datasets and probe seeds (10,000 replicates). Framed: the best mean rank. Circles: masks statistically tied with it (epoch 10 only, as in the paper).
Hover a cell for its training curve · empty cell: r = all with L = 33 would hide the whole window (not trained) · epochs 1–9: 3 probe seeds, epoch 10: 5
Why the mask matters: EEG is redundant across electrodes, much less across time
A good mask hides enough neighbours to make the task non-trivial, but leaves enough context to solve it: hence a moderate radius and a short length.
Same channel, time-shifted
Same time, different channels
Use the models: All 58 checkpoints are public
For most uses, take the best mask; the other 56 are there to study how masking shapes representations.
Pick a checkpoint
At the best mask, MAE and JEPA both score above REVE: either is a good starting point. The code below follows your pick.
Pretext
Length L
Radius r
PierreGtch/eeg-fm-masking_mae_r9cm_L2 +0.07 vs REVE · rank 8.6 of 29 · best mask
Load a pre-trained model
Install the code (pip install git+https://github.com/PierreGtch/eeg-fm-masking), then load any checkpoint by its id:
import json
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from eeg_fm_masking.oeb.wrapper import ContextualEncoderBenchmarkWrapper
repo = "PierreGtch/eeg-fm-masking_mae_r9cm_L2"
config = json.load(open(hf_hub_download(repo, "config.json")))
model = ContextualEncoderBenchmarkWrapper(
n_chans=n_chans, n_times=n_times, n_outputs=n_outputs, sfreq=200.0,
chs_info=chs_info, # MNE channel info (info["chs"]), positions in metres
**config,
)
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")), strict=False)Evaluate on OpenEEGBench
OpenEEGBench freezes the encoder and fits a ridge probe on its features for each of the 12 datasets, the protocol of the paper. One call runs it on any checkpoint, or on your own model:
import json
from huggingface_hub import hf_hub_download
from open_eeg_bench import benchmark
repo = "PierreGtch/eeg-fm-masking_mae_r9cm_L2"
results = benchmark(
model_cls="eeg_fm_masking.oeb.wrapper.ContextualEncoderBenchmarkWrapper",
hub_repo=repo,
model_kwargs=json.load(open(hf_hub_download(repo, "config.json"))),
finetuning_strategies=["ridge_probe"], # frozen encoder + ridge probe, as in the paper
n_seeds=5,
) # a pandas DataFrame with the test metricsThe video: The paper in two minutes
Citation
@article{guetschel2026masking,
title = {What Masking Geometry Works Best for {EEG} Foundation Models?},
author = {Guetschel, Pierre and Aristimunha, Bruno and El Ouahidi, Yassine and
Delorme, Arnaud and Moreau, Thomas and Tangermann, Michael},
journal = {arXiv preprint arXiv:2609.33487},
year = {2026},
month = sep
}