Our paper, Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning, has been accepted at Interspeech 2026. The work asks a practical accessibility question: can we improve speech recognition for dysarthric speakers without repeatedly asking them to record large amounts of training data?

Dysarthric speech is highly variable. It can be affected by conditions such as cerebral palsy, Parkinson's disease, ALS, and other neurological conditions. Mainstream automatic speech recognition systems are usually trained on very large typical-speech datasets, so they often struggle when speech contains atypical articulation, timing, phonation, or fatigue-related variation.

The data burden problem

The central bottleneck is not only model architecture. It is data. Dysarthric speech datasets are difficult to collect at scale because recruitment is challenging, recording sessions can be tiring for participants, transcription takes time, and clinical labels often require expert input. As a result, many benchmark datasets remain small compared with the corpora used to train general-purpose ASR systems.

Synthetic speech offers one way to expand training data, but many prior approaches still require speaker-specific fine-tuning, parallel recordings, or multiple enrollment utterances. That can reintroduce the same burden the augmentation method was meant to reduce.

Our approach

We tested whether zero-shot voice cloning can serve as a low-burden augmentation strategy for dysarthric ASR. Using Higgs Audio V2, we cloned speakers from the TORGO dataset using a single reference utterance per speaker, averaging about 7.2 seconds of speech. We then generated a synthetic training set, TORGO-Synth, from linguistically diverse LibriSpeech text prompts.

Pipeline diagram showing TORGO reference speech, Higgs Audio V2 cloning, TORGO-Synth generation, Whisper fine-tuning, and evaluation on TORGO and SAP-1102.
Figure 1. Overview of the voice cloning, ASR fine-tuning, and evaluation pipeline.

The ASR backbone was Whisper-medium. We compared four settings: zero-shot Whisper with no fine-tuning, fine-tuning on real TORGO speech, fine-tuning on cloned speech, and hybrid fine-tuning using both real and cloned speech. Evaluation was performed on held-out real TORGO speech, with an additional cross-corpus test on SAP-1102.

Duration distribution charts for cloned utterances in TORGO-Synth, shown for all speakers and by severity.
Figure 2. Duration distribution of cloned utterances in TORGO-Synth.
t-SNE projection of TORGO speaker embeddings comparing original reference speech and cloned speech clusters.
Figure 3. t-SNE projection of TORGO speaker embeddings. Stars denote original reference utterances and dots denote cloned speech.

What we found

The cloned speech provided a strong adaptation signal. On TORGO, zero-shot Whisper had a word error rate of 31.62%. Fine-tuning on cloned speech reduced this to 26.00%, close to fine-tuning on real speech at 24.44% and hybrid fine-tuning at 25.12%.

Table 1. Speaker-level WER (%) for Whisper-medium across zero-shot and fine-tuning configurations.
SeveritySpeakerZero-shotRealCloneHybrid
SevereM0482.3260.2263.5462.43
Moderate-SevereF0176.6763.3348.3343.33
Moderate-SevereM0145.2827.8334.4335.38
Moderate-SevereM0242.0835.4237.0833.75
Moderate-SevereAverage54.6842.1939.9537.49
ModerateM0523.8126.9833.3330.16
MildF0315.5614.6014.6015.24
MildF042.651.593.172.65
MildM032.761.842.301.84
MildAverage6.996.016.696.58
OverallWER31.6224.4426.0025.12
Overall∆WER (pp)-7.185.626.50

The most interesting result appeared in the moderate-severe speaker group. For those speakers, cloned and hybrid fine-tuning outperformed real-only fine-tuning. This suggests that synthetic cloned speech can add useful phonetic and lexical diversity where authentic recordings are especially scarce and variable.

We also found that more synthetic data is not automatically better. Clone data showed a non-monotonic scaling pattern, with the strongest overall performance around 15 hours of synthetic speech. Beyond that point, performance began to flatten or degrade, likely because the model can start fitting synthetic artifacts rather than robust dysarthric speech patterns.

Table 2. Average WER (%) per severity group across fine-tuning data quantities.
Severity0h5h10h15h20h25h30h40h50h
Severe82.3271.8267.4063.5462.4364.6469.6167.4064.09
Moderate-Severe54.6853.1242.2739.9542.5545.0640.5541.3046.25
Moderate23.8134.9238.1033.3336.5136.5136.5134.9236.51
Mild6.997.626.746.696.637.217.518.288.30
Overall WER31.6230.8127.6926.0026.4728.2327.2228.3728.37

Generalization beyond TORGO

Cross-corpus evaluation on SAP-1102 showed that clone-based fine-tuning can transfer beyond the source dataset. Clone fine-tuning reduced overall WER from 14.50% to 12.84%, and showed particularly strong gains for the cerebral palsy cohort. This is encouraging because accessibility systems must work for new speakers, not only for the dataset used during development.

Cross-corpus generalization bar chart showing WER on SAP-1102 across cerebral palsy, ALS, Parkinson's disease, and overall categories.
Figure 4. Cross-corpus generalization results on SAP-1102 across etiologies.

Why this matters

The goal is not to replace real dysarthric speech data. Real recordings remain essential. But this work shows that zero-shot cloning can help stretch limited data further, potentially reducing participant burden while improving ASR robustness for speakers who are often excluded from mainstream voice technology.

For accessible speech technology, the practical question is always: how do we build systems that improve recognition without demanding unrealistic amounts of effort from the people they are meant to serve? This paper takes one step toward that goal by showing that a single short reference utterance can support useful synthetic augmentation for dysarthric ASR.

Representative audio samples are available at the project audio samples page.