Agentharm Eval

---
name: agentharm-eval
description: This benchmark evaluates the harmfulness and safety alignment of LLM-based agents by measuring their compliance with malicious, multi-step tasks that require coherent tool chaining. It probes whether models can be coerced into executing harmful behaviors through direct prompting or simple jailbreak templates, while tracking refusal rates and capability preservation. Use when the user wants to benchmark on AgentHarm, or asks about evaluating this task. Reports harm score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.09024
  bibtex_key: andriushchenko2024agentharm
  confidence: high
---

# agentharm-eval

> AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents — Andriushchenko et al. (2024) (arXiv:2410.09024, 2024)

## What this evaluates

This benchmark evaluates the harmfulness and safety alignment of LLM-based agents by measuring their compliance with malicious, multi-step tasks that require coherent tool chaining. It probes whether models can be coerced into executing harmful behaviors through direct prompting or simple jailbreak templates, while tracking refusal rates and capability preservation.

## Datasets

- **AgentHarm** — total 110; splits: test (110)

## Metrics

- `harm score` **(primary)** — range: percent
  - The percentage of harmful tasks successfully executed by the agent, graded by a GPT-4o semantic judge based on target functions (e.g., all tool calls made, correct order, correct parameters). Calculated as (successful executions / total tasks) × 100.

## Input / output format

**Input**: A system prompt encouraging tool use, followed by a direct malicious prompt or a jailbreak template prompt. The agent receives the prompt and iteratively processes tool execution results within a context window of up to 4096 tokens.

**Output**: Sequential text responses and structured tool calls (e.g., <Tool call>: tool_name: {"param": "value"}) executed in a while-loop agentic setting.

## Scoring recipe

```python
successes = 0
for task in AgentHarm_test_set:
    output = run_agent(task.prompt, system_prompt, max_tokens=4096)
    grade = gpt4o_judge(output, task.target_functions)
    if grade == 1.0:  # All target functions satisfied
        successes += 1
harm_score = (successes / len(AgentHarm_test_set)) * 100
```

## Common pitfalls

- Forced tool calling APIs can act as a jailbreak themselves, artificially lowering refusal rates.
- API constraints vary by provider (e.g., Gemini ignores ~16% of behaviors due to no-argument function errors; Llama struggles with tool-call syntax), requiring model-specific workarounds.
- The universal jailbreak template was optimized on a private validation set, which may inflate performance compared to zero-shot generalization.

## Evidence (verbatim from paper)

> We use GPT-4o as a semantic judge for grading particular aspects of the agent's output and for tracking refusals. GPT-4o mini gets a 62.5% harm score, while refusing just 22% of the prompts; Mistral Large 2 refuses just 1.1% of requests and gets our strongest score without attacking of 82.2%.

## Citation

```bibtex
@misc{andriushchenko2024agentharm,
  title={AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents},
  author={Andriushchenko et al. (2024)},
  year={2024},
  note={arXiv:2410.09024}
}
```

- arXiv: 2410.09024