Xtsc Bench Eval

---
name: xtsc-bench-eval
description: This benchmark evaluates the quality and stability of explanation methods for time series classification models. It probes four key properties: robustness to input perturbations, faithfulness to model predictions, computational complexity of the explanations, and reliability against known ground-truth informative features. Use when the user wants to benchmark on XTSC-Bench Synthetic Datasets, or asks about evaluating this task. Reports Faithfulness Correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.14957
  bibtex_key: hollig2023xtscbench
  confidence: high
---

# xtsc-bench-eval

> XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification — Höllig et al. (2023) (arXiv:2310.14957, 2023)

## What this evaluates

This benchmark evaluates the quality and stability of explanation methods for time series classification models. It probes four key properties: robustness to input perturbations, faithfulness to model predictions, computational complexity of the explanations, and reliability against known ground-truth informative features.

## Datasets

- **XTSC-Bench Synthetic Datasets** — total 120; splits: test (120); repo https://github.com/JHoelli/XTSC-Bench

## Metrics

- `Max Sensitivity` — range: other
  - Measures the maximum change in the explanation vector when the input is perturbed within a radius r: max_{x̄-x≤r} ||E_f(x̄) - E_f(x)||.
- `Average Sensitivity` — range: other
  - Computes the average sensitivity over the input neighborhood: (1/|x|) Σ ||E_f(x̄) - E_f(x)||.
- `Faithfulness Correlation` **(primary)** — range: [-1, 1]
  - Measures the correlation between the sum of attributions over a feature subset S and the difference in model output when those features are replaced by a reference baseline: corr(Σ_{s∈S} E_f(x_{x_s=x̃_s}), f(x) - f(x_{x_s=x̃_s})).
- `Complexity` — range: other
  - Calculates the entropy of the fractional contribution distribution of features: -Σ_{i=1}^d P_g(i) ln(P_g(i)), where P_g(i) = E_f(x)_i / Σ|E_f(x)|.
- `Relevance Rank Accuracy` — range: [0, 1]
  - Ratio of top-K attributed features that fall within the ground truth mask: |X_topK ∩ GT(x)| / |GT(x)|.
- `Relevance Mass Accuracy` — range: [0, 1]
  - Ratio of the sum of attribution values within the ground truth mask to the total sum of all attribution values: Σ_{E_f(x)_i ∈ GT(x)} E_f(x)_i / Σ E_f(x).

## Input / output format

**Input**: Time series tensor (time_steps × features), model f, explanation method E, and optionally ground truth mask GT(x) and reference baseline x̃.

**Output**: Attribution vector E_f(x) of identical shape to input, plus computed metric scores for robustness, faithfulness, complexity, and reliability.

## Scoring recipe

```python
def score(attribution, x, f, gt_mask, baseline, r=0.01, K=10):
    x_bar = x + np.random.uniform(-r, r, size=x.shape)
    attr_bar = explain(x_bar)
    sens_max = np.max(np.abs(attr_bar - attribution))
    sens_mean = np.mean(np.abs(attr_bar - attribution))
    x_masked = x.copy(); x_masked[gt_mask] = baseline[gt_mask]
    faith = np.corrcoef(np.sum(attribution[gt_mask]), f(x) - f(x_masked))[0,1]
    P_g = attribution / np.sum(np.abs(attribution))
    cpx = -np.sum(P_g * np.log(P_g + 1e-9))
    topK = np.argsort(attribution)[-K:]
    racc = len(set(topK) & set(gt_mask)) / len(gt_mask)
    macc = np.sum(attribution[gt_mask]) / np.sum(attribution)
    return sens_max, sens_mean, faith, cpx, racc, macc
```

## Common pitfalls

- Using standard baselines like zeros or training means for time series can be misleading, as zero may represent an informative anomaly or trend.
- The perturbation radius r for sensitivity metrics must be small; too large a perturbation violates the assumption that f(x) ≈ f(x̄) and breaks robustness evaluation.
- Default uniform baselining in the tool may be inappropriate for real-world datasets; custom baselines should be provided to match domain-specific noise characteristics.

## Evidence (verbatim from paper)

> XTSC-Bench employs two metrics measuring the robustness of an explanation algorithm E: Max Sensitivity measures the maximum change in the explanation with a small perturbation of the input x. Average Sensitivity denotes the average sensitivity in the neighborhood of x with x̄-x≤r. XTSC-Bench employs faithfulness correlation to measure the correlation between the sum of attributions and the difference in output when setting those features to a reference baseline. XTSC-Bench includes the ground truth based measures relevance rank accuracy and relevance mass accuracy.

## Citation

```bibtex
@misc{hollig2023xtscbench,
  title={XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification},
  author={Höllig et al. (2023)},
  year={2023},
  note={arXiv:2310.14957}
}
```

- arXiv: 2310.14957