Bark Multi Agent Behavior Eval

---
name: bark-multi-agent-behavior-eval
description: Evaluates the robustness of autonomous driving behavior planners (MCTS, RL, IDM, MOBIL) in interactive multi-agent traffic. It probes how well models handle prediction inaccuracies, parameter variations, and complex merging constraints without fine-tuning. Use when the user wants to benchmark on BARK Sampling Scenarios, INTERACTION, or asks about evaluating this task. Reports collision_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.02604
  bibtex_key: bernhard2020bark
  confidence: high
---

# bark-multi-agent-behavior-eval

> BARK: Open Behavior Benchmarking in Multi-Agent Environments — Bernhard et al. (2020) (arXiv:2003.02604, 2020)

## What this evaluates

Evaluates the robustness of autonomous driving behavior planners (MCTS, RL, IDM, MOBIL) in interactive multi-agent traffic. It probes how well models handle prediction inaccuracies, parameter variations, and complex merging constraints without fine-tuning.

## Datasets

- **BARK Sampling Scenarios** — total 2400; splits: test (2400); repo https://github.com/bark-simulator/bark
- **INTERACTION** — total 1; splits: test (1)

## Metrics

- `collision_rate` **(primary)** — range: percent
  - Percentage of scenarios resulting in a collision out of the total evaluated scenarios. Calculated as (number of collisions / total scenarios) * 100.
- `success_rate` — range: percent
  - Percentage of scenarios where the controlled agent successfully reaches the goal within the maximum allowed simulation steps.
- `average_steps` — range: other
  - Mean number of simulation steps taken per scenario, computed only over successfully completed runs.

## Input / output format

**Input**: Initial traffic state including ego vehicle position/velocity, surrounding vehicles' positions/velocities, map geometry, and goal definition (e.g., target lane). For simulation benchmarks, the full recorded scenario sequence with other agents' trajectories.

**Output**: Action sequence (steering, acceleration, braking) or lane-change decision per time step. Evaluation outputs binary outcomes (collision, success, exceeded_time) or final trajectory.

## Scoring recipe

```python
def score(predictions, gold):
    collisions = successes = exceeded = 0
    total_steps = 0
    for pred in predictions:
        if pred.collided: collisions += 1
        elif pred.reached_goal:
            successes += 1
            total_steps += pred.steps
        else: exceeded += 1
    return {
        'collision_rate': collisions / len(predictions),
        'success_rate': successes / len(predictions),
        'avg_steps': total_steps / successes if successes > 0 else 0
    }
```

## Common pitfalls

- Collision metrics alone do not capture scenario deviation; a model might avoid collisions but drastically alter traffic flow or violate ordering constraints.
- Parameter variations (e.g., IDM time headway) are applied to surrounding agents to test prediction robustness, not the ego planner's direct parameters.
- Results are highly sensitive to the fixed maximum simulation steps (30) and time step resolution (0.1s–0.2s), which truncate long-horizon behaviors.

## Evidence (verbatim from paper)

> Figure 5 gives the percentages of each scenario run with the controlled agent reaching the goal, colliding, or exceeding the maximum allowed simulation time steps (>30), as well as the average number of simulation steps in the event that the goal is reached. It is clear that the collision rate rises as the prediction error increases.

## Citation

```bibtex
@misc{bernhard2020bark,
  title={BARK: Open Behavior Benchmarking in Multi-Agent Environments},
  author={Bernhard et al. (2020)},
  year={2020},
  note={arXiv:2003.02604}
}
```

- arXiv: 2003.02604