# LLM Eval Harness

> - Test/evaluate any LLM behind an OpenAI- or Anthropic-compatible endpoint: availability (maxtokens-aware), request fidelity (does system prompt/tools/history REACH the model, or does the gateway silently drop it), speed (TTFT+tok/s), concurrency (before a workshop), Anthropic protocol compliance, quality regression, vendor bug reports, deployment gates, resident canaries. Reach for this BEFORE hand-rolling a curl loop against any LLM endpoint — that instinct is the trap: it skips the N=10 sampling, Connection:close, and env-var key handling this bakes in. Use when someone tests/benchmarks/测评/压测 a model/endpoint, onboards a provider, decides whether to switch or temporarily fail over to an alternate channel (e.g. outage or quota exhaustion), writes a supported-models list, debugs "model ignores system prompt", or verifies a tok/s claim. Triggers on "benchmark this model", "测一下这个模型/渠道/API", "接入新模型先测一下", "system prompt 不生效", "这个渠道能不能用/稳不稳", "临时切换过去顶一阵子" — even without "eval", even wrapped in business narrative.

## Facts
- Page: https://tashan.sh/capability/skill-daymade-llm-eval-harness
- tashan id: skill:daymade/llm-eval-harness
- Source: https://github.com/daymade/claude-code-skills
- Type: skill
- Category: other
- tashan score: not scored (catalogued only — too little public evidence)
- Adoption: 9.0
- Upkeep: 97.0
- Freshness: 93.0
- Evidence coverage: 84% of the inputs this score can use
- Health: active
- Instruction depth: not yet graded
- License: MIT
- Official: no

## Install

```sh
cp -r llm-eval-harness ~/.claude/skills/
```

## Security audit
Not scanned. We audit npm-published capabilities; this one has no npm package we can resolve, or has not reached the queue. This is not a clean bill of health.

---
Measured 2026-09-12 by tashan (https://tashan.sh) from public evidence. Scorer s5.