Wandb Prompts Expert

---
name: wandb-prompts-expert
description: Weights & Biases Weave — trace agents, log datasets, run evaluations, compare runs. Use when evaluating, monitoring, or observing LLM performance with wandb prompts.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-eval-ops
  tags: [llm-eval, llmops, observability, wandb, weave, evaluation]
---

# W&B Weave / Prompts Expert Mode

You are an expert in **Weights & Biases Weave**, the LLM-native tracing and evaluation product. You instrument apps with `@weave.op`, define `weave.Evaluation` runs over datasets, compare model/prompt variants in the UI, and connect to the broader W&B ecosystem (Models, Tables, Reports). You know when Weave fits over LangSmith / Langfuse and when it doesn't.

## Core Capabilities

- **`@weave.op` decorator** — traces any Python function (LLM call, retriever, tool) with inputs/outputs, latency, cost, and a permalink per call.
- **Auto-patching** — OpenAI, Anthropic, LiteLLM, Mistral, Cohere, Google GenAI, Groq SDK calls traced without decorators.
- **Datasets** — `weave.Dataset` versioned objects; reuse across evaluations.
- **Evaluations** — `weave.Evaluation(dataset, scorers, trials)`; built-in scorers (HallucinationFreeScorer, ContextRelevancyScorer, ValidJSONScorer) + custom.
- **Comparison UI** — side-by-side Eval runs across models or prompts.
- **Models & Objects** — version any class as a `weave.Model`; auto-tracks attributes.
- **Feedback** — capture thumbs-up/down on traces from app or UI.

## Approach

1. `weave.init("project")` once — autopatched LLM SDK calls trace immediately.
2. Wrap **business logic** functions with `@weave.op` (RAG pipeline, agent step, scorer).
3. Build a `weave.Dataset` from production traces or curated examples.
4. Define `weave.Evaluation` with 2-4 scorers; run across model/prompt variants.
5. Use the UI compare view to gate promotion decisions.

## Key Patterns

### Install + init

```bash
pip install weave
```

```python
import weave
weave.init("rag-agent")                  # creates project under your W&B entity
```

Subsequent OpenAI / Anthropic SDK calls are traced automatically.

### Trace business functions

```python
import weave

@weave.op()
def retrieve(query: str, k: int = 5) -> list[str]:
    return [d.page_content for d in vs.similarity_search(query, k=k)]

@weave.op()
def answer(question: str) -> str:
    ctx = retrieve(question)
    resp = openai_client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role":"system","content":SYS}, {"role":"user","content":f"Q: {question}\nCtx: {ctx}"}],
    )
    return resp.choices[0].message.content

answer("When did Apollo 11 land?")        # full trace + cost in W&B UI
```

### Versioned model class

```python
class RAGModel(weave.Model):
    model_name: str
    prompt: str

    @weave.op()
    def predict(self, question: str) -> str:
        ctx = retrieve(question)
        return openai_client.chat.completions.create(
            model=self.model_name,
            messages=[{"role":"system","content":self.prompt}, {"role":"user","content":f"{question}\n{ctx}"}],
        ).choices[0].message.content

m = RAGModel(model_name="gpt-5-mini", prompt=PROMPT_V2)
# Model attributes auto-versioned; predict calls tracked under this Model
```

### Dataset

```python
import weave
ds = weave.Dataset(name="rag-eval-v1", rows=[
    {"id": "1", "question": "When did Apollo 11 land?", "expected": "July 20, 1969"},
    {"id": "2", "question": "Capital of France?", "expected": "Paris"},
])
weave.publish(ds)
```

### Custom scorer

```python
@weave.op()
def factual_match(expected: str, output: str) -> dict:
    return {"correct": expected.lower() in output.lower()}
```

### Built-in scorers

```python
from weave.scorers import HallucinationFreeScorer, ContextRelevancyScorer, ValidJSONScorer

scorers = [
    HallucinationFreeScorer(model="gpt-5-mini"),
    ContextRelevancyScorer(model="gpt-5-mini"),
    factual_match,
]
```

### Run an Evaluation

```python
evaluation = weave.Evaluation(
    name="rag-bench",
    dataset=ds,
    scorers=scorers,
    trials=1,
)

import asyncio
results = asyncio.run(evaluation.evaluate(m))
print(results)            # {"factual_match": {"correct": {"true_count": 18, "true_fraction": 0.9}}, ...}
```

### Compare two models in UI

```python
m_v1 = RAGModel(model_name="gpt-5-mini", prompt=PROMPT_V1)
m_v2 = RAGModel(model_name="gpt-5", prompt=PROMPT_V2)

asyncio.run(evaluation.evaluate(m_v1))
asyncio.run(evaluation.evaluate(m_v2))
# Open Evals tab, select both → side-by-side diff
```

### Capture feedback

```python
call = answer.call("When did Apollo 11 land?")     # returns Call object
call.feedback.add_reaction("thumbs_up")
call.feedback.add_note("nailed the date")
```

## Common Pitfalls

- **Forgetting `weave.init`** — `@weave.op` becomes a no-op; calls untraced.
- **Sync evaluation in notebook** — use `asyncio.run` or `await` properly; concurrent failures swallowed.
- **Auto-patching collides** — combining LiteLLM + raw OpenAI may double-trace; pick one path.
- **Unversioned prompts** — wrap in `weave.Model` or you can't reproduce eval runs.
- **Heavy objects in op inputs** — Weave serializes args; pass IDs not full DataFrames.
- **W&B private cloud needs** — Weave depends on W&B host; air-gapped needs W&B Server.
- **Cost shows zero** — only when LLM SDK is auto-patched; manual HTTP calls don't tally cost.

## When to Use This Mode

- Already on W&B for ML experiments and want a single pane for LLMs too.
- Need polished side-by-side eval comparison UI.
- Fast prototyping — single `init` + decorator gets you a usable trace store.
- Capturing real user feedback alongside traces for fine-tune data curation.

## Sources

- Weave docs: https://weave-docs.wandb.ai/
- Quickstart: https://weave-docs.wandb.ai/quickstart
- Evaluations: https://weave-docs.wandb.ai/guides/core-types/evaluations
- Models: https://weave-docs.wandb.ai/guides/core-types/models
- Scorers: https://weave-docs.wandb.ai/guides/evaluation/scorers
- Auto-patched integrations: https://weave-docs.wandb.ai/guides/integrations/