# Creating Online Evaluations

> Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn this failure into an eval". Covers choosing the target and eval type (hog / llmjudge / sentiment), configuring a provider, model, and usable provider key for an llmjudge eval, scoping which generations trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

## Facts
- Page: https://tashan.sh/capability/skill-posthog-creating-online-evaluations
- tashan id: skill:PostHog/creating-online-evaluations
- Source: https://github.com/PostHog/ai-plugin
- Type: skill
- Category: ai
- tashan score: not scored (catalogued only — too little public evidence)
- Adoption: 9.0
- Upkeep: 96.0
- Freshness: 92.0
- Evidence coverage: 84% of the inputs this score can use
- Health: active
- Instruction depth: not yet graded
- Official: no

## Install

```sh
cp -r creating-online-evaluations ~/.claude/skills/
```

## Security audit
Not scanned. We audit npm-published capabilities; this one has no npm package we can resolve, or has not reached the queue. This is not a clean bill of health.

---
Measured 2026-08-20 by tashan (https://tashan.sh) from public evidence. Scorer s5.