Benchtemp Eval

---
name: benchtemp-eval
description: Evaluates the effectiveness and efficiency of Temporal Graph Neural Networks (TGNNs) on link prediction and node classification tasks. It probes model performance across transductive and inductive settings (New-Old/New-New) to ensure fair cross-model comparisons. Use when the user wants to benchmark on BenchTemp (15 datasets), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.16385
  bibtex_key: huang2023benchtemp
  confidence: high
---

# benchtemp-eval

> BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks — Huang et al. (2023) (arXiv:2308.16385, 2023)

## What this evaluates

Evaluates the effectiveness and efficiency of Temporal Graph Neural Networks (TGNNs) on link prediction and node classification tasks. It probes model performance across transductive and inductive settings (New-Old/New-New) to ensure fair cross-model comparisons.

## Datasets

- **BenchTemp (15 datasets)** — total ?; splits: test (-1); repo https://github.com/qianghuangwhu/benchtemp

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. Computed as the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `AP` — range: [0, 1]
  - Average Precision, the area under the precision-recall curve.
- `Runtime` — range: other
  - Seconds per epoch required for training.
- `Epochs` — range: other
  - Number of epochs until early stopping triggers.
- `RAM` — range: other
  - Maximum CPU RAM usage during training.
- `GPU Memory` — range: other
  - Maximum GPU memory usage during training.

## Input / output format

**Input**: Temporal graph data (nodes, edges, timestamps) formatted for link prediction or node classification under transductive or inductive (New-Old/New-New) settings.

**Output**: Predicted link/node probabilities. Efficiency logs: runtime per epoch, epochs to convergence, peak RAM/GPU memory.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

def compute_ap(y_true, y_pred):
    from sklearn.metrics import average_precision_score
    return average_precision_score(y_true, y_pred)

# Run 3 times, report mean ± std. Early stop: patience=3, tol=1e-3, timeout=48h.
```

## Common pitfalls

- Early stopping uses a patience of 3 and tolerance of 1e-3, which may cause models to terminate prematurely if validation metrics fluctuate.
- A 48-hour timeout is enforced; models that do not converge or encounter runtime errors are marked with '—' or '*' and excluded from averages.
- Standard deviations may be reported as zero if a model fails to complete multiple epochs within the timeout, leading to only one valid run.

## Evidence (verbatim from paper)

> Evaluation Metrics. We use the Evaluator module, choosing AUC and AP for the link prediction task, and AUC for the node classification task, following the prior works [1-7]. In addition, we report efficiency metrics, as shown in Table 4. Protocol. We run each job three times (unless timed out) and report the mean and standard deviation. We use an EarlyStopMonitor with a patience of 3 and tolerance of 10^{-3}, and set a timeout (48 hours).

## Citation

```bibtex
@misc{huang2023benchtemp,
  title={BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2308.16385}
}
```

- arXiv: 2308.16385