Preprint · 2026
What Masking Geometry Works Best for EEG Foundation Models?
We describe EEG masking strategies with three parameters, then pre-train 58 foundation models that differ only in their mask and pretext. One mask works best for both MAE and JEPA.
- (L, r, ρ)
- three parameters that cover the masks of the EEG literature
- 58
- models, one pipeline: only the mask and the pretext change
- L = 2 s, r = 9 cm
- best mask, for both MAE and JEPA
One formalism for EEG masks
Most EEG foundation models learn by hiding part of the signal and predicting it. Each one hides it differently, and changes the architecture, data and objective at the same time, so the effect of the mask has never been isolated.
We describe any EEG mask with three parameters: blocks of radius r on the scalp and length L in time, added until a fraction ρ of the signal is hidden.
- K
- – blocks
- Masked
- –
Cursor
Loading 3-D view…
3-D view unavailable (WebGL is disabled in this browser).
Try it: L = 1, r = one gives random patch masking; r = all hides whole time segments; L = 33 (the whole 30 s window) hides whole channels. Everything in between is a spatio-temporal block mask.
The study: pre-train, then evaluate
We use this formalism to run a controlled sweep. All models share one pipeline and one training budget; only the mask and the training target change. MAE reconstructs the raw EEG of the hidden tokens; JEPA predicts how a teacher network encodes them. After pre-training, every model is scored the same way, on the 12 datasets of the OpenEEGBench benchmark. Switch views and hover the blocks for details.
The network fills in the hidden parts of the EEG signal itself: it must output the missing raw samples.
Everything outside the coloured box is shared. Switching between MAE and JEPA only swaps the training target.
The network predicts an abstract representation of the hidden parts, as computed by a teacher network, rather than the raw signal.
Everything outside the coloured box is shared. Switching between MAE and JEPA only swaps the training target.
The mask, decoder and teacher are gone. The frozen encoder turns a new EEG recording into features, and a linear model predicts the label from them. The same protocol is used for all 58 models on the 12 OpenEEGBench datasets.
Only the tokeniser and the encoder are kept from pre-training, and they stay frozen.
Shared
EEG
A 30 s window of EEG, 32 randomly sampled channels. Each column is one channel; each cell is a 1 s patch of it (200 samples, 0.1 s overlap between patches). Each window is normalised on its own (robust scaling).
EEG
A window from one of the 12 OpenEEGBench tasks (sleep staging, motor imagery, seizure detection, emotion…). Its montage, channel count (6 to 64) and length (1 to 30 s) differ from pre-training: the encoder only needs the electrode positions. It is scaled as in pre-training and never masked.
Shared
Linear (tokeniser)
Turns each 1 s patch of one channel into a 512-dim vector: one token per (channel, time patch). No convolution, no channel mixing. The electrode’s 3D position and the time index are then added as a fixed (non-learned) positional encoding.
Linear (tokeniser)
The same pre-trained tokeniser, frozen, on the same 1 s patches: one token per (channel, time patch). Nothing is masked.
Shared
Token grid
All tokens of the window, before masking. The same grid feeds the student encoder (after masking) and, in JEPA, the teacher (unmasked).
Token grid
All tokens of the window: nothing is masked at evaluation time.
Pre-training
Masking strategy (L, r)
The object of the study. The mask is a union of blocks, each covering all electrodes within a radius r of a random centre electrode, for L consecutive time patches. The number of blocks is set so that about 55 % of the tokens are hidden on average.
Pre-training
Mask
One random draw of the mask, resampled for every window. Solid cells stay visible (the context), hatched cells are hidden (what must be predicted). The dotted lines show everywhere this mask is used.
Pre-training
Dotted arrows: the mask
Not data, only the mask pattern: it tells each block which positions are context and which are targets.
Pre-training
Remove masked tokens
Drops the hidden tokens. The encoder only ever sees the visible ones, never a placeholder for the missing ones (as in the original MAE). This also makes the encoder cheaper to run.
Pre-training
Visible tokens
The ~45 % of tokens that survive the mask. This is all the context the model gets.
Shared
Transformer Encoder
The model we keep after pre-training. 4 layers, 8 heads, 512 dims (REVE-Small architecture, ~12.6 M parameters). It mixes information across channels and time among visible tokens only. Downstream, it is frozen and evaluated with a linear probe on 12 OpenEEGBench datasets.
Transformer Encoder (frozen)
The pre-trained encoder, weights frozen: no parameter is updated downstream. It now sees every token of the full, unmasked window. Its last-layer tokens are the features given to the probe.
Shared
Encoded visible tokens
Contextual embeddings of the visible tokens: each one now carries information from the whole visible context.
Encoded tokens
One 512-dim embedding per channel and per 1 s patch, from the last encoder layer: C channels × T patches × 512 values.
Pre-training
Transformer Decoder
A small 2-layer transformer, used only during pre-training and thrown away afterwards. It places one learned “mask token” at each hidden position (dotted arrow: which positions) and fills it in by attending to the encoder output. Same architecture for MAE and JEPA; only the target differs.
Pre-training
Decoder outputs
The decoder’s guesses for the hidden positions, still in the 512-dim latent space. A last linear layer maps them back to raw EEG.
JEPA predictions
The decoder’s guesses for the hidden positions, in latent space. They are compared directly to the teacher’s embeddings at the same positions.
MAE only
MAE-only branch
The target is the raw EEG signal itself. No teacher, no extra loss term.
MAE only
Remove unmasked signal patches
Keeps only the raw signal of the hidden patches: the loss is computed only where the model could not see the input.
MAE only
MAE targets
The raw 1 s signal patches (200 samples each) at the hidden positions, taken straight from the input. Fixed targets: they do not change during training.
MAE only
Linear (output head)
Projects each 512-dim decoder output back to a 200-sample signal patch, so predictions and targets live in the same space.
MAE only
MAE predictions
The reconstructed raw signal at each hidden position.
MAE only
L2 MAE
Mean squared error between reconstructed and true samples, averaged over hidden patches only. The single loss term. Because the targets are the data, MAE cannot collapse to a trivial solution, but it also has to spend capacity on noise that is hard to predict.
JEPA only
JEPA-only branch
The target is the teacher’s embedding of the hidden tokens, not the signal. No extra anti-collapse loss (no VICReg-style term), to keep strict parity with MAE.
JEPA only
EMA update
The teacher is not trained by gradients. After each step, its weights are nudged towards the student encoder’s: θteacher ← τ θteacher + (1 − τ) θstudent, with τ increasing from 0.999 to 1 over training. The result is a slowly moving, smoothed copy of the student, which gives stable targets.
JEPA only
Transformer Encoder (Teacher)
A copy of the encoder that sees the full, unmasked window. It shares the tokeniser with the student, so only the transformer layers are duplicated. Discarded after pre-training.
JEPA only
Teacher embeddings
The teacher’s last-layer embedding for every token. Since the teacher saw everything, the embeddings at hidden positions encode what was really there, in context.
JEPA only
Remove unmasked tokens
Keeps only the teacher embeddings at the hidden positions, i.e. the ones the student has to predict.
JEPA only
Gradient stop
No gradient flows back into the teacher through the targets. Otherwise the easiest way to lower the loss would be to move the targets rather than improve the predictions.
JEPA only
JEPA targets
Teacher embeddings of the hidden tokens (512 dims). Unlike MAE targets, they change during training as the teacher follows the student.
JEPA only
L2 JEPA
Mean squared error between predicted and teacher embeddings, over hidden positions only. The single loss term. Moving targets make a trivial solution possible: with r = “all”, JEPA falls into what we call a bias-inflation collapse, which the usual collapse monitors do not catch.
Downstream only
Flattened features
The encoded tokens concatenated into one vector per window: from 31,744 to 146,432 features, depending on the dataset.
Downstream only
Gaussian random projection
A fixed random linear map down to 5,000 features (entries drawn from 𝒩(0, 1/5000)), to keep the probe cheap. Its random draw is the only source of randomness in the evaluation: each probe seed draws a new projection.
Downstream only
Projected features
5,000 features per window: the input of the linear probe.
Downstream only
Ridge regression (linear probe)
A linear model fitted in closed form on the training split, with no gradient descent; its regularisation λ is picked among 17 values from 10−8 to 108 on the validation split. A linear probe measures what the representation already contains, rather than what a fine-tuned network could learn on top of it.
Downstream only
Prediction
A class (e.g. a sleep stage), or a value for the one regression dataset (seed-vig). Test scores (balanced accuracy, R² for seed-vig) are normalised per dataset and averaged over the 12 datasets: that average, relative to REVE, is the colour of each cell in the results below.
29 mask geometries (ρ fixed at 0.55) × 2 pretexts = 58 models
What we found
Each cell is one pre-trained model. Colour = its score relative to REVE, averaged over the 12 datasets. Slide through the pre-training epochs, switch to ranks, or hover a cell to see how it learns.
JEPA
| 1 | 2 | 4 | 8 | 16 | 33 | |
|---|---|---|---|---|---|---|
| one | −0.03 | +0.02 | −0.01 | −0.02 | −0.02 | −0.07 |
| 6 cm | +0.07 | +0.03 | +0.07 | +0.02 | −0.05 | −0.04 |
| 9 cm | +0.07 | +0.10 | +0.06 | −0.01 | −0.03 | −0.04 |
| 12 cm | −0.07 | +0.04 | +0.06 | −0.18 | −0.27 | +0.02 |
| all | −0.41 | −0.67 | −0.37 | −0.36 | −0.31 | not trained |
MAE
| 1 | 2 | 4 | 8 | 16 | 33 | |
|---|---|---|---|---|---|---|
| one | −0.02 | −0.01 | +0.02 | +0.04 | +0.02 | −0.00 |
| 6 cm | −0.02 | +0.06 | +0.07 | +0.02 | −0.13 | +0.04 |
| 9 cm | +0.01 | +0.07 | +0.05 | +0.03 | −0.16 | +0.07 |
| 12 cm | +0.01 | +0.03 | +0.02 | +0.03 | −0.10 | +0.05 |
| all | −0.05 | −0.08 | −0.10 | −0.14 | −0.25 | not trained |
Mean normalised score over the 12 OpenEEGBench datasets and the probe seeds, minus that of the public REVE-Base checkpoint (0 = REVE; linear probe on frozen features). Bold: the best mask of each pretext.
Mean rank (1 = best) of each mask among the 29 of its pretext, under a hierarchical bootstrap over datasets and probe seeds (10,000 replicates). Bold: the best mean rank.
Hover a cell to compare the two pretexts and see its training curve · empty cell: r = all with L = 33 would hide the whole window (not trained) · epochs 1–9: 3 probe seeds (a few evaluations missing), epoch 10: 5 seeds, as in the paper
- Same winner. MAE and JEPA agree on the best mask: L = 2 s, r = 9 cm.
- Many near-winners. Outside the failure zones the exact choice barely matters: 11 MAE and 9 JEPA masks are statistically tied with the best.
- Too local. Masking one electrode (r = one) is too easy: its neighbours give the answer away.
- Too global. Masking every electrode (r = all) leaves only time to predict from, and time is a weak cue. JEPA suffers most.
- Too long. Most long blocks (L = 8 or 16) also hurt: over several seconds, the past says little about the present.
- Whole channels. The exception: hiding entire channels (L = 33) works, especially for MAE. It becomes a different task: rebuild a channel from its neighbours.
- MAE is more forgiving. The gap between the best and the worst mask is 2.4× larger with JEPA than with MAE.
Why does the mask matter?
EEG is highly redundant across neighbouring electrodes, and much less across time. A good mask hides enough neighbours to make the task non-trivial, but leaves enough context to solve it. That is why the winner sits at a moderate radius and a short length.
Same channel, time-shifted
Same time, different channels
Use the models
All 58 checkpoints are public. For most uses, take the best mask; the other 56 are there to study how masking shapes representations.
Pick a checkpoint
At the best mask, MAE and JEPA both score above REVE: either is a good starting point.
Pretext
Length L
Radius r
PierreGtch/eeg-fm-masking
mae-L2-r9cm +0.07 vs REVE · rank 8.6 of 29 · best mask
Open on Hugging Face ↗What the encoder expects
- EEG resampled to 200 Hz
- 3D electrode positions for every channel (any montage: the model reads positions, not channel names)
- Any channel count and window length: pre-trained on 32 channels × 30 s, evaluated on 6 to 64 channels and 1 to 30 s windows
- Each window normalised on its own, as in pre-training (robust scaling, clipped at 15 SD)
- Output: one 512-dim embedding per channel per 1 s token
Minimal example
from eeg_fm_masking import load_encoder
encoder = load_encoder(framework="mae", L=2, r=9)
tokens = encoder(x, positions) # x: (batch, channels, times) at 200 HzEvaluate on OpenEEGBench
Our encoders plug into OpenEEGBench, so you can compare yours against all 58 under the exact same protocol.
Reproduce or extend
- Re-plot without training: every score on this page comes from the CSVs in results/.
- Pre-train your own: one run takes ~14 H100-hours on the open part of the REVE corpus (4.4 TB). Any (L, r) can be set in the config.
Citation
@article{guetschel2026masking,
title = {What Masking Geometry Works Best for {EEG} Foundation Models?},
author = {Guetschel, Pierre and Aristimunha, Bruno and El Ouahidi, Yassine and
Delorme, Arnaud and Moreau, Thomas and Tangermann, Michael},
journal = {arXiv preprint arXiv:0000.00000},
year = {2026}
}