Adaptive Llm Testing Eval

---
name: adaptive-llm-testing-eval
description: This evaluation probes the effectiveness of diversity-based adaptive test selection strategies for black-box LLM applications. It measures how quickly and reliably different prioritization methods detect failures in prompt templates compared to random baselines, while also assessing the diversity of generated outputs. Use when the user wants to benchmark on BBH & P3 Prompt Templates, or asks about evaluating this task. Reports APFD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.13480
  bibtex_key: yoon2025adaptivetesting
  confidence: high
---

# adaptive-llm-testing-eval

> Adaptive Testing for LLM-Based Applications: A Diversity-based Approach — Yoon, Feldt, and Yoo (2025) (arXiv:2501.13480, 2025)

## What this evaluates

This evaluation probes the effectiveness of diversity-based adaptive test selection strategies for black-box LLM applications. It measures how quickly and reliably different prioritization methods detect failures in prompt templates compared to random baselines, while also assessing the diversity of generated outputs.

## Datasets

- **BBH & P3 Prompt Templates** — total 46; splits: test (-1)

## Metrics

- `APFD` **(primary)** — range: percent
  - Adapted from Average Percentage of Faults Detected. Calculates the cumulative percentage of failures detected as tests are executed in prioritized order, averaged across the execution sequence. Ranges from 0 to 100.
- `Output Diversity` — range: other
  - Average number of unique words across all generated outputs for a given input subset.

## Input / output format

**Input**: Prompt templates with standardized instructions and input examples, formatted to constrain LLM outputs to specific patterns (e.g., 'The answer is [answer].').

**Output**: Natural language text generated by the LLM, expected to contain the correct answer in a specified format.

## Scoring recipe

```python
def compute_apfd(outputs, golds):
    failures = [1 if out != gold else 0 for out, gold in zip(outputs, golds)]
    cumulative = []
    count = 0
    for f in failures:
        count += f
        cumulative.append(count)
    n = len(failures)
    t = n
    return 100 * (1 - (sum(cumulative) / (n * t)) + (1 / (2 * t)))

def compute_diversity(outputs):
    unique_counts = [len(set(out.split())) for out in outputs]
    return sum(unique_counts) / len(unique_counts)
```

## Common pitfalls

- Applying traditional structural coverage metrics to closed-source LLMs instead of using failure detection.
- Assuming a single distance metric generalizes well across all task types without considering task-dependent effectiveness.
- Comparing adaptive strategies against random baselines without accounting for differing input pool sizes (100 vs 1,000) or insufficient repetitions.

## Evidence (verbatim from paper)

> To this end, we adopt the average percentage of failure detection (APFD), a straightforward adaptation of the Average Percentage of Faults Detected metric[[33]], which has been widely used for evaluating test prioritization techniques. APFD, which ranges from 0 to 100, indicates the rate of failure detection, with higher values reflecting faster identification of failures.

## Citation

```bibtex
@misc{yoon2025adaptivetesting,
  title={Adaptive Testing for LLM-Based Applications: A Diversity-based Approach},
  author={Yoon, Feldt, and Yoo (2025)},
  year={2025},
  note={arXiv:2501.13480}
}
```

- arXiv: 2501.13480