Xglue Eval

---
name: xglue-eval
description: Evaluates cross-lingual transfer capabilities of pre-trained language models across 11 diverse natural language understanding and generation tasks spanning over 100 languages. It measures how well models fine-tuned on English can generalize to zero-shot testing in other languages. Use when the user wants to benchmark on XGLUE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.01401
  bibtex_key: liang2020xglue
  confidence: high
---

# xglue-eval

> XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation — Liang et al. (2020) (arXiv:2004.01401, 2020)

## What this evaluates

Evaluates cross-lingual transfer capabilities of pre-trained language models across 11 diverse natural language understanding and generation tasks spanning over 100 languages. It measures how well models fine-tuned on English can generalize to zero-shot testing in other languages.

## Datasets

- **XGLUE** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels averaged across all target languages for understanding tasks (NER, POS, NC, XNLI, PAWS-X, QADSM, WPR, QAM).
- `BLEU-4` — range: percent
  - 4-gram BLEU score for generation tasks (QG, NTG) and abstractive summarization.
- `ROUGE-L` — range: percent
  - Longest common subsequence recall for abstractive summarization evaluation.

## Input / output format

**Input**: Source text(s) in a target language, optionally paired with a pivot language (English) training set for fine-tuning.

**Output**: Predicted class labels, token-level tags, or generated text sequences.

## Scoring recipe

```python
def compute_metric(predictions_by_lang, gold_by_lang, task_type):
    lang_scores = []
    for lang in predictions_by_lang:
        preds = predictions_by_lang[lang]
        gold = gold_by_lang[lang]
        if task_type == 'understanding':
            lang_scores.append(sum(1 for p, g in zip(preds, gold) if p == g) / len(gold))
        else:
            lang_scores.append(bleu4(preds, gold))
    return mean(lang_scores)
```

## Common pitfalls

- Models are fine-tuned exclusively on English labeled data, then zero-shot tested on all other languages; failing to replicate this pivot-language setup yields non-comparable results.
- Generation tasks use different pre-training objectives (xDAE vs xFNP) and prediction mechanisms (single-token vs multi-token), making direct comparison unfair without noting the architectural differences.
- Averages reported in tables are macro-averages across languages, not weighted by dataset size.

## Evidence (verbatim from paper)

> Given a task, each pre-trained model is fine-tuned using its English training set only, and then applied to all test sets in different languages. ... BLUE-4 is the metric.

## Citation

```bibtex
@misc{liang2020xglue,
  title={XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation},
  author={Liang et al. (2020)},
  year={2020},
  note={arXiv:2004.01401}
}
```

- arXiv: 2004.01401