Almanacs Eval

---
name: almanacs-eval
description: Evaluates whether language model explanations (e.g., weights, qualitative descriptions) enable a second predictor model to accurately simulate and predict the behavior of a synthetic linear model across safety-relevant scenarios. The benchmark specifically probes simulatability and robustness to distributional shift between training and test variable values. Use when the user wants to benchmark on ALMANACS Synthetic Dataset, or asks about evaluating this task. Reports probability.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.12747
  bibtex_key: mills2023almanacs
  confidence: medium
---

# almanacs-eval

> ALMANACS: A Simulatability Benchmark for Language Model Explainability — Mills et al. (2023) (arXiv:2312.12747, 2023)

## What this evaluates

Evaluates whether language model explanations (e.g., weights, qualitative descriptions) enable a second predictor model to accurately simulate and predict the behavior of a synthetic linear model across safety-relevant scenarios. The benchmark specifically probes simulatability and robustness to distributional shift between training and test variable values.

## Datasets

- **ALMANACS Synthetic Dataset** — total ?; splits: train (-1), test (-1); repo https://github.com/edmundmills/ALMANACS

## Metrics

- `probability` **(primary)** — range: [0, 1]
  - The predictor outputs a float between 0 and 1 representing the likelihood of a 'Yes' answer. Performance is evaluated by comparing this predicted probability against the true probability generated by the synthetic linear model.

## Input / output format

**Input**: Template-based scenario questions with five variable placeholders, accompanied by either WEIGHTS (numerical weights and variable-to-score mappings) or QUALITATIVE (natural language descriptions of variable influence and ranking) explanations.

**Output**: A JSON object containing a 'reasoning' string and a 'probability' float representing the model's estimated likelihood of a 'Yes' answer.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions: list of predicted probabilities from predictor
    # gold: list of true probabilities from synthetic model
    # Evaluate how closely predicted probabilities match the ground truth
    mse = sum((p - g) ** 2 for p, g in zip(predictions, gold)) / len(predictions)
    return mse
```

## Common pitfalls

- The synthetic model relies on all-distilroberta-v1 embeddings and per-placeholder PCA, which may not capture the semantic complexity of real-world LLM reasoning.
- Predictors must estimate missing variable scores from limited examples, introducing compounding errors when multiplied by weights.
- The benchmark explicitly tests distributional shift between train and test variables, meaning standard in-distribution fine-tuning will not generalize.

## Evidence (verbatim from paper)

> Given a sample S with scores {s_1, s_2, ..., s_5} for the values filled in the template, the prediction y which corresponds to the probability of a 'yes' answer is computed as: y = σ(∑ w_i s_i) where σ denotes the sigmoid function.

## Citation

```bibtex
@misc{mills2023almanacs,
  title={ALMANACS: A Simulatability Benchmark for Language Model Explainability},
  author={Mills et al. (2023)},
  year={2023},
  note={arXiv:2312.12747}
}
```

- arXiv: 2312.12747