Aspire Eval

---
name: aspire-eval
description: Evaluates the perceptual quality of audio-visual speech enhancement models in real-world noisy environments. It probes how well models generalize to natural reverberation, multi-source background noise, and speaker occlusion compared to synthetic training conditions. Use when the user wants to benchmark on ASPIRE, or asks about evaluating this task. Reports MUSHRA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.00424
  bibtex_key: gogate2019avspeech
  confidence: high
---

# aspire-eval

> AV Speech Enhancement Challenge using a Real Noisy Corpus — Gogate et al. (2019) (arXiv:1910.00424, 2019)

## What this evaluates

Evaluates the perceptual quality of audio-visual speech enhancement models in real-world noisy environments. It probes how well models generalize to natural reverberation, multi-source background noise, and speaker occlusion compared to synthetic training conditions.

## Datasets

- **ASPIRE** — total ?; splits: test (-1)

## Metrics

- `MUSHRA` **(primary)** — range: [0, 100]
  - Subjective quality score on a scale of [0, 100] based on ITU-R BS.1534 (MUSHRA) guidelines. Listeners rate the perceptual quality of enhanced speech relative to a reference degraded signal.

## Input / output format

**Input**: Audio samples of enhanced speech generated by SE models for specific utterances from the ASPIRE corpus.

**Output**: A subjective quality rating on a scale from 0 to 100 per audio sample.

## Scoring recipe

```python
scores = []
for participant in participants:
    for utterance in test_set:
        score = participant.rate(enhanced_audio) # 0-100
        scores.append(score)
scores = filter_outliers(scores, completion_times)
return mean(scores)
```

## Common pitfalls

- Confusing the training corpora (Grid + Chime3 synthetic mixtures) with the evaluation corpus (ASPIRE real-noisy).
- Assuming MUSHRA scores directly translate to objective metrics like PESQ or STOI without calibration.
- Ignoring the outlier removal step based on listening test completion time, which can skew results.

## Evidence (verbatim from paper)

> MUSHRA-style [[8]] listening test was used for subjective evaluation. A total of 20 native English speakers with normal-hearing participated in the listening test. The individual test consists of 20 randomly selected utterances drawn from the ASPIRE corpus. ... Participants were asked to score the quality of each audio sample, on a scale from [0, 100], generated by different SE models for the same sentence.

## Citation

```bibtex
@misc{gogate2019avspeech,
  title={AV Speech Enhancement Challenge using a Real Noisy Corpus},
  author={Gogate et al. (2019)},
  year={2019},
  note={arXiv:1910.00424}
}
```

- arXiv: 1910.00424