Creating Online Evaluations
Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn this failure into an eval". Covers choosing the target and eval type (hog / llmjudge / sentiment), configuring a provider, model, and usable provider key for an llmjudge eval, scoping which generations trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. …
Works with: Claude Code (native) · Cursor, Codex CLI (manual)
native: this artifact type is that client's own format
Category: AI & Agents — see all ranked ›
Install (Claude Code):
cp -r creating-online-evaluations ~/.claude/skills/- Adoption: 1 repos
- Upkeep: 96.0
- Freshness: 92.0
- Evidence coverage: 84% of the inputs this score can use — the rest are unknown, and the score is discounted for it
- Health: active
- Contributors: 10
Security audit
Not scanned yet. We audit npm-published capabilities for known advisories, install-time scripts and permission surface; this one has no npm package we can resolve, or has not reached the queue.
Its own instructions
Its SKILL.md says when to use it, shows worked examples and covers setup.
Read from the capability’s own SKILL.md. This is not a grade and does not compare to the instruction-depth verdict on an MCP server — a skill has no tools to document, so that rubric does not apply to it.
You searched for one. Check the rest of your stack:
npx tashan-cli doctorReads the config already on your machine and names what is dead, deprecated or running code at install time. No account, nothing uploaded.
Pro adds the history to tashan doctor, so a run over your own config says which of yours gained an advisory, started running an install script, or lost its last maintainer — and what to move to.
- Every score since we started measuring, for any capability
- The named replacement when something you run is dying — not just that it is
tashan doctorover the config you already have, on your machine
Start a 7-day trial › Everything measured on this page stays free.
source ↗ · skill:PostHog/creating-online-evaluations
Everything on this page is public evidence and free. What it cannot know is whether you run this — check your whole config, free, in the browser. tashan Pro adds the series behind each row and names a replacement for anything dying.
Already running this? Check your whole config — free, in your browser, nothing installed. Or npx tashan-cli doctor locally, which sends nothing at all.
Measured 2026-08-20 · scorer s5 · how · something wrong here?