Actionbench Eval

---
name: actionbench-eval
description: Evaluates a text-to-image model's ability to customize generated images with a specific subject's appearance while accurately transferring a target action from an exemplar image, without appearance leakage or subject deformation. Use when the user wants to benchmark on ActionBench, or asks about evaluating this task. Reports total accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.15841
  bibtex_key: huang2023learningdisentangledidentifiers
  confidence: high
---

# actionbench-eval

> Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation — Siteng Huang et al. (2023) (arXiv:2311.15841, 2023)

## What this evaluates

Evaluates a text-to-image model's ability to customize generated images with a specific subject's appearance while accurately transferring a target action from an exemplar image, without appearance leakage or subject deformation.

## Datasets

- **ActionBench** — total ?; splits: test (-1)

## Metrics

- `action accuracy` — range: percent
  - Percentage of generated images where the action matches the exemplar action, as judged by human evaluators.
- `subject accuracy` — range: percent
  - Percentage of generated images where the subject matches the provided textual name without obvious deformations or defects.
- `total accuracy` **(primary)** — range: percent
  - Percentage of generated images that are considered totally correct, meaning both the action and the subject are correctly generated according to human evaluators.

## Input / output format

**Input**: Exemplar image(s) showing a specific action, and a textual name/description of the target subject.

**Output**: A single 512x512 RGB image generated by the text-to-image model.

## Scoring recipe

```python
correct_count = 0
total_count = 0
for pair in ActionBench:
    images = model.generate(exemplar=pair.exemplar, subject_name=pair.name, n=4)
    for img in images:
        total_count += 1
        action_ok = evaluator.check_action_consistency(img, pair.exemplar)
        subject_ok = evaluator.check_subject_fidelity(img, pair.name)
        if action_ok and subject_ok:
            correct_count += 1
total_accuracy = (correct_count / total_count) * 100
```

## Common pitfalls

- Relies entirely on human evaluation without reporting inter-annotator agreement or rater calibration.
- Only 4 images are sampled per subject-action pair, which may not capture generation variance or diversity.
- Binary 'totally correct' criterion ignores partial successes or graded quality assessments.

## Evidence (verbatim from paper)

> Tab. 1 reports the action, subject and total accuracy for all methods. ... A generated image will only be considered totally correct if both the action and the character are correctly generated.

## Citation

```bibtex
@misc{huang2023learningdisentangledidentifiers,
  title={Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation},
  author={Siteng Huang et al. (2023)},
  year={2023},
  note={arXiv:2311.15841}
}
```

- arXiv: 2311.15841