Xtreme R Eval

---
name: xtreme-r-eval
description: Evaluates zero-shot cross-lingual transfer by training models on English data and testing them on 50 typologically diverse languages across classification, QA, and retrieval tasks. It probes fine-grained diagnostic capabilities and cross-lingual alignment using structured performance breakdowns. Use when the user wants to benchmark on XQuAD, XCOPA, Mewsli-X, LAReQA, CheckList, or asks about evaluating this task. Reports Exact Match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.07412
  bibtex_key: ruder2021xtremer
  confidence: high
---

# xtreme-r-eval

> XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation — Ruder et al. (2021) (arXiv:2104.07412, 2021)

## What this evaluates

Evaluates zero-shot cross-lingual transfer by training models on English data and testing them on 50 typologically diverse languages across classification, QA, and retrieval tasks. It probes fine-grained diagnostic capabilities and cross-lingual alignment using structured performance breakdowns.

## Datasets

- **XQuAD** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
- **XCOPA** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
- **Mewsli-X** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
- **LAReQA** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
- **CheckList** — total ?; splits: test (-1); repo https://github.com/google-research/xtreme

## Metrics

- `Exact Match` **(primary)** — range: [0, 1]
  - Returns 1.0 if the predicted answer string exactly matches the gold answer string, and 0.0 otherwise. Averaged across all test instances.
- `F1` — range: [0, 1]
  - Token-level F1 score measuring the harmonic mean of precision and recall between predicted and gold answers.

## Input / output format

**Input**: Training: English-language task instances. Evaluation: Target-language test instances (questions, multiple-choice options, or document pairs) for zero-shot inference.

**Output**: Predicted answer string, class label, or retrieved document ID per instance.

## Scoring recipe

```python
def evaluate(predictions, golds):
    em = sum(1.0 if p == g else 0.0 for p, g in zip(predictions, golds)) / len(golds)
    f1 = compute_token_f1(predictions, golds)
    return {'exact_match': em, 'f1': f1, 'avg': (em + f1) / 2}
```

## Common pitfalls

- Using validation sets from multiple target languages for hyperparameter tuning instead of a single target language validation set as recommended.
- Fine-tuning on translated in-language data rather than strictly following the zero-shot cross-lingual transfer protocol.
- Reporting only aggregate scores without providing fine-grained diagnostic breakdowns, which obscures language-specific or attribute-specific weaknesses.

## Evidence (verbatim from paper)

> We average “F1” and “Exact Match” of QA systems. We classify the attribute values into four categories: extra-small (XS), small (S), large (L) and extra-large (XL) values (see the Appendix for detailed interval information).

## Citation

```bibtex
@misc{ruder2021xtremer,
  title={XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation},
  author={Ruder et al. (2021)},
  year={2021},
  note={arXiv:2104.07412}
}
```

- arXiv: 2104.07412