Xrag Eval

---
name: xrag-eval
description: Evaluates an LLM's ability to perform cross-lingual retrieval-augmented generation by answering questions in a target language using supporting documents in English or mixed languages, while ignoring topically related distractors. It specifically probes cross-document reasoning capabilities and response language consistency. Use when the user wants to benchmark on XRAG, or asks about evaluating this task. Reports response language consistency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.10089
  bibtex_key: liu2025xrag
  confidence: high
---

# xrag-eval

> XRAG: Cross-lingual Retrieval-Augmented Generation — Liu et al. (2025) (arXiv:2505.10089, 2025)

## What this evaluates

Evaluates an LLM's ability to perform cross-lingual retrieval-augmented generation by answering questions in a target language using supporting documents in English or mixed languages, while ignoring topically related distractors. It specifically probes cross-document reasoning capabilities and response language consistency.

## Datasets

- **XRAG** — total ?; splits: test (-1)

## Metrics

- `response language consistency` **(primary)** — range: percent
  - Measures whether the generated answer is in the same language as the input question. Calculated as the proportion of instances where Language(q) == Language(\tilde{a}).

## Input / output format

**Input**: A question q in a target language (German, Spanish, Chinese, or Arabic), a set of two supporting articles D^+, and a set of distracting articles D^- (topically related but unhelpful). In the monolingual retrieval setting, all documents are in English. In the multilingual retrieval setting, documents are in English and the question language.

**Output**: An answer \tilde{a} generated by the LLM, which must be in the same language as the question q.

## Scoring recipe

```python
def score_response_language_consistency(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if detect_language(pred) == detect_language(gold_ans):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models frequently generate answers in English even when the question is in a non-English language, failing the language consistency constraint in monolingual retrieval settings.
- Distracting articles D^- are topically related to the question but cannot answer it, requiring the model to perform cross-document reasoning to correctly ignore them and synthesize information only from D^+.

## Evidence (verbatim from paper)

> The dataset reveals two key challenges: (1) all evaluated LLMs fail to maintain response language consistency in monolingual retrieval, and (2) in multilingual retrieval, the core difficulty lies in cross-lingual information integration rather than non-English text generation.

## Citation

```bibtex
@misc{liu2025xrag,
  title={XRAG: Cross-lingual Retrieval-Augmented Generation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.10089}
}
```

- arXiv: 2505.10089