don't forget the fix to non-deployability of silent-speech models
2026-07-07
Abstract
Silent-speech models based on surface EMG/EEG are trained on isolated, fully-articulated words and evaluated on windows containing exactly those words. Deployment does not look like that. A continuous stream is mostly silence and partial words. The leading and trailing edges of an articulation that a fixed window necessarily slices through. A model that has only ever seen whole words is handed a class it was never trained to represent and fails.
This paper makes two claims.
First, handling partial words is the core of the task that should be handled before numbers are reported. Introduced is a transition-label scheme that labels each sample by the fraction of a word it contains; without it, an otherwise identical model collapses from 0.86 to 0.43 accuracy when deployed live.
Second, cross-session use is its own unsolved problem: models are trained and evaluated as though they will only ever run on the session they were fit to, and when session variation is measured at all it is a documented failure, often ignored by much literature and never solved. Under our collection protocol, extra-session validation does not produce a measurable drop in performance.
For the sake of isolation and simplicity the experiment is limited to a small vocabulary and a single subject.
Cross-session stability
The first issue addressed is that models are trained and evaluated as if they will only ever run on the session they were fit on, even though a domain shift exists due to variability such as exact positioning, sweat, day, focus, etc. Variability that is not fully captured with a low-session uncontrolled approach, and thus not robust against.
When session-to-session transfer is measured at all, it is a documented drop of 15–20 points [1,2,3].
Instead it should be a first principle: you construct a test set which consists of purely held-out sessions. The delta to the validation set (that shares sessions with train) is then to be minimized by increasing the diversity and forcing it to either
- a) find session-invariant features
- b) be able to model all possible changes
Data was collected on a custom-built EEG+EMG rig fit to a single user, giving around 3000 full word samples (600 per class) over roughly 100 sessions collected across different times and states, with different distractions and non-articulatory movement. The speech is performed without sound, a closed mouth, and reduced movement of mouth.
Under this protocol, session-to-session variation did not produce measurable degradation for a single subject for 90% accuracy, as seen in the numbers later.
Setup
ATCNet [4] is used for all following experiments and results, which was the strongest architecture in our comparison against EEGNet [5] and a CATNet [6] both with and without transition handling. On the temporal dimension, attention has a better gradient flow than CNN hierarchy and more robustness than an LSTM over raw timesteps (since the timing and duration variation across repetitions).
Transition labels
By default, literature reports strong isolated-word accuracy and treats the task as classification: given a window known to contain a word, name it.
In a live deployment where segmentation is not solved, any fixed-length window will, most of the time, contain a fragment of a word rather than a whole one. Still, the majority case is absent from training, oftentimes even missing a label for silence at all.
In this implementation, windows have a hard set length of 1.6 s. A sample is labelled by the fraction of the target word it contains:
- fraction > 0.7 → the target word's label
- 0.3 ≤ fraction ≤ 0.7 → a transition (word starting / ending)
- fraction < 0.3 → silence
The time of transition 0.3s - 0.7s is 100% label of word_staring or word_ending. Within 0.3 s on either side, the two adjacent labels are mixed as smoothed (soft) labels as not to force it to learn an imaginary boundary.
Train, val, and test all included partials for 2/3 of the data, and the rest being full ones.
At first word_starting and word_ending were used as their own classes alongside silence, on the reasoning that separation would be helpful for a clean decision boundary. However during training the two transition classes were consistently the hardest, and both collapsed into silence:
| split | samples | loss | accuracy | bal_acc | macro_f1 | weighted_f1 |
|---|---|---|---|---|---|---|
| train | 10416 | 0.4952 | 0.8352 | 0.8533 | 0.8408 | 0.8315 |
| val [intra] | 1836 | 0.5272 | 0.8164 | 0.8334 | 0.8238 | 0.8132 |
| test [extra] | 2668 | 0.7351 | 0.7721 | 0.7883 | 0.7805 | 0.7688 |
| label | precision | recall | f1 | support | top confusion |
|---|---|---|---|---|---|
| highlight | 0.9127 | 0.8149 | 0.8611 | 308 | naan 26 (8%) |
| bullshit | 0.8289 | 0.8811 | 0.8542 | 286 | word ending 19 (7%) |
| gogogo | 0.8365 | 0.8642 | 0.8502 | 302 | silence 26 (9%) |
| shitbull | 0.7554 | 0.9026 | 0.8224 | 195 | word starting 8 (4%) |
| naan | 0.7047 | 0.7953 | 0.7473 | 171 | highlight 10 (6%) |
| word starting | 0.8171 | 0.6253 | 0.7084 | 443 | silence 66 (15%) |
| word ending | 0.7825 | 0.5502 | 0.6461 | 438 | silence 116 (26%) |
| silence | 0.6690 | 0.8971 | 0.7665 | 525 | word starting 36 (7%) |
As the separation wasn't required functionally, merging word_starting and word_ending into silence (though keeping the smooth labels), spared the model from learning two soft but still imaginary boundaries, which resulted in the best recorded performance:
| split | samples | loss | accuracy | bal_acc | macro_f1 | weighted_f1 |
|---|---|---|---|---|---|---|
| train | 8675 | 0.3366 | 0.8986 | 0.9079 | 0.8883 | 0.8996 |
| val [intra] | 1540 | 0.4423 | 0.8636 | 0.8749 | 0.8519 | 0.8648 |
| test [extra] | 1583 | 0.3750 | 0.8825 | 0.8885 | 0.8674 | 0.8838 |
| label | precision | recall | f1 | support | top confusion |
|---|---|---|---|---|---|
| highlight | 0.8848 | 0.8450 | 0.8645 | 200 | naan 26 (13%) |
| bullshit | 0.9211 | 0.9589 | 0.9396 | 219 | silence 5 (2%) |
| gogogo | 0.8898 | 0.9545 | 0.9211 | 220 | bullshit 4 (2%) |
| shitbull | 0.7660 | 0.9412 | 0.8446 | 153 | silence 7 (5%) |
| naan | 0.6708 | 0.8182 | 0.7372 | 132 | silence 10 (8%) |
| silence | 0.9459 | 0.8231 | 0.8802 | 616 | shitbull 34 (6%) |
To compare and proof the claim that a model is unprepared for real shifted data, a model was trained and validated on full silence or words only, while tested on the set that included partials, showing a significant drop in performance:
| split | samples | loss | accuracy | bal_acc | macro_f1 | weighted_f1 |
|---|---|---|---|---|---|---|
| train | 4500 | 0.0821 | 0.9722 | 0.9687 | 0.8067 | 0.9722 |
| val [intra] | 774 | 0.1371 | 0.9612 | 0.9552 | 0.7975 | 0.9611 |
| test [extra] | 1540 | 4.0936 | 0.5338 | 0.7347 | 0.5629 | 0.4087 |
Implementation and limitations
Limitations
- Single subject. Cross-session and vocabulary results are within-subject. Cross-subject generalization is not shown.
- Small vocabulary. Five words plus silence; accuracy is not comparable to large-vocabulary systems, and the task is easier on the vocabulary axis than the session axis is hard.
- The imbalance of classes between val & test set.: test set performed better with 0.8885 compared to the validation set's 0.8749, explained by the minority class being overrepresented in it.
- No extensive tuning of the labelling algorithm: The fraction cutoffs (0.3 / 0.7) and the 0.3 s smoothing window were set empirically, and the confusion structure suggests there is more to extract by tuning them, which should be systematically explored and might shift performance both when transitions are merged with silence or not.
Future directions
The cross-session result should be tested across subjects. A research question to answer would be whether, once the extra-session generalizability has been solved on one person, then once a new subject is calibrated to perform on a closed set of sessions, this ability transfers, or has to be relearned from scratch for each subject.
The transition scheme currently resolves a word only once enough of it is present, which is determined by a single hand-picked untested number. Some words may be identifiable from their beginning alone if the opening phoneme is unique against the rest of the vocabulary, or if the EEG carries enough semantic or early motoric content to commit early. Detecting that a word is determinable and doing so before it completes would save latency directly, replacing a transition state.
Constructing multiple samples out of a single utterance to fight low-data regime without overfitting to blocks and sessions.
References
[1] SilentWear: an Ultra-Low Power Wearable System for EMG-based Silent Speech Recognition. arXiv:2603.02847. (Cross-validated 77.5% → inter-session 59.3%, silent speech.)
[2] L. Maier-Hein, F. Metze, T. Schultz, A. Waibel. Session Independent Non-Audible Speech Recognition Using Surface Electromyography. ASRU 2005. (Within-session 97.3% → across-session 76.2% without adaptation.)
[3] F. Meier et al. A Parallel Ultra-Low Power Silent Speech Interface based on a Wearable, Fully-dry EMG Neckband. arXiv:2509.21964. (Silent 68±3% CV → 54±7% leave-one-session-out.)
[4] H. Altaheri, G. Muhammad, M. Alsulaiman. Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification. IEEE Transactions on Industrial Informatics, 19(2):2249–2258, 2023. doi:10.1109/TII.2022.3197419.
[5] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, B. J. Lance. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering, 15(5):056013, 2018. doi:10.1088/1741-2552/aace8c.
[6] Y. Zhuang, C. Huang, Z. Yu, Y. Zou, J. Ju. CAT-Net: A Cross-Attention Tone Network for Cross-Subject EEG-EMG Fusion Tone Decoding. arXiv:2511.10935, 2025.
☘