Bbh Eval

---
name: bbh-eval
description: Evaluates a model's zero-shot in-context learning capability on reasoning-heavy multiple-choice tasks. It compares self-generated demonstrations against direct prompting and chain-of-thought baselines to measure accuracy gains. Use when the user wants to benchmark on BIG-Bench Hard (BBH), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.15035
  bibtex_key: chen2023selficl
  confidence: high
---

# bbh-eval

> Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations — Chen et al. (2023) (arXiv:2305.15035, 2023)

## What this evaluates

Evaluates a model's zero-shot in-context learning capability on reasoning-heavy multiple-choice tasks. It compares self-generated demonstrations against direct prompting and chain-of-thought baselines to measure accuracy gains.

## Datasets

- **BIG-Bench Hard (BBH)** — total 5511; splits: test (5511)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice tasks: the proportion of test instances where the model's predicted answer exactly matches the gold label.

## Input / output format

**Input**: Task description followed by the test input instance. For CoT baselines, a reasoning trigger phrase 'Let's think step by step.' is appended.

**Output**: A single predicted answer choice from the multiple-choice options.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- BBH contains 27 tasks total, but the evaluation strictly uses only the 23 multiple-choice tasks; including non-multiple-choice tasks will break the accuracy metric.
- Evaluation temperature must be set to 0 for deterministic results, as specified in the implementation details.
- Prompt format varies significantly between baselines (ZS-Direct vs ZS-CoT); mixing prompt templates will invalidate head-to-head comparisons.

## Evidence (verbatim from paper)

> We adopt the BIG-Bench Hard (BBH) benchmark for our evaluation. BBH contains a total of 27 tasks, from which we select 23 tasks that are multiple-choice tasks as our evaluation testbed for SELF-ICL. Each BBH tasks has around 150 ~ 250 examples, and the total number of instances is 5,511. The accuracy delta indicates the accuracy difference between SELF-ICL and the baseline method (blue/orange indicates our method wins/loses).

## Citation

```bibtex
@misc{chen2023selficl,
  title={Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.15035}
}
```

- arXiv: 2305.15035