evaluation-methodology
PluginEval 怎麼替 skill 和 plugin 打分：static、judge、Monte Carlo 三層
中文翻譯
---
name: evaluation-methodology
description: "PluginEval 的品質方法論，涵蓋維度、rubric 與計分公式。適用於理解 plugin 品質如何被衡量、解讀某個維度的低分、決定如何提升 skill 的 triggering accuracy 或 orchestration fitness、為你的 marketplace 設定分數門檻，或向 Neon 這類外部夥伴說明品質徽章的時候。"
---
# 評估方法論
PluginEval 結合最多三層，替 skill 或 plugin 打 0 到 100 分。Static 層是 lint，速度快又具確定性，適合用來檢查結構。LLM judge 層和 Monte Carlo 層是實驗性質，沒有拿人工標註驗證過，所以它們的數字只能當作粗略的訊號。以 trace 為基礎的 eval 計畫，請看 repo 根目錄的 `evals/README.md`。
Judge rubric 的錨點在 [references/rubrics.md](references/rubrics.md)。每個 anti-pattern 的修法和每個維度的提升技巧，在 [references/improving-scores.md](references/improving-scores.md)。
## 評估深度
| 深度 | 層 | 信心標籤 |
|---|---|---|
| `quick` | static | Estimated |
| `standard`（`score` 的預設） | static 與 judge | Assessed |
| `deep`（`certify` 使用） | static、judge，以及跑 50 次的 Monte Carlo | Certified |
| `thorough` | static、judge，以及跑 100 次的 Monte Carlo | Certified+ |
這個標籤代表實際跑了哪個深度，不代表有跟人工判斷對照過。Plugin 目錄不管你要求哪個深度，都只會跑 static 層。
## Static 層（lint）
Static 層讀取 SKILL.md，不呼叫任何模型。它計算七個子分數，前六個會餵進複合維度：
- `frontmatter_quality`（餵進 `triggering_accuracy`）
- `orchestration_wiring`（餵進 `orchestration_fitness`）
- `progressive_disclosure`, `structural_completeness`, `token_efficiency`, `ecosystem_coherence`
- `harness_portability`
`harness_portability` 不對應任何維度，所以不會改變 skill 的複合分數。不過它占 static 層自身分數的 6%。Plugin 的分數是由這層分數算出來的，所以 portability 的發現可能會稍微拉低 plugin 的分數。它的發現不會被算作 anti-pattern。
Static 層會標出六種 anti-pattern：`OVER_CONSTRAINED`、`EMPTY_DESCRIPTION`、`MISSING_TRIGGER`、`BLOATED_SKILL`、`ORPHAN_REFERENCE` 和 `DEAD_CROSS_REF`。每個標記會讓分數扣 5%，最低扣到 50% 為止：
```text
penalty = max(0.5, 1.0 - 0.05 * anti_pattern_count)
```
報告會為每個標記印出嚴重程度，但扣分只算標記的數量，不看嚴重程度。
## LLM judge 層（實驗性質）
Judge 層會呼叫模型四次，回傳四個 0 到 1 的整體分數：
- `triggering_accuracy`：Haiku 讀 description 並寫 10 個測試 prompt，5 個應該觸發、5 個不應該。它預測每個 prompt 的結果，並回報自己算出的 F1。沒有任何東西會拿這些預測去對照真實的觸發情況。
- `orchestration_fitness` 和 `scope_calibration`：Sonnet 用五級的 rubric 替 skill 評分。
- `output_quality`：Sonnet 想像三個任務，並評估它預期會得到的輸出。
這三次 Sonnet 呼叫只看得到 SKILL.md 的前 3,000 個字元。只會跑一個 judge，因為沒有任何東西會讀 `judges` 設定。
## Monte Carlo 層（實驗性質）
Haiku 寫 15 個應該觸發這個 skill 的 prompt，這一層重複使用它們，湊到 50 次執行（`thorough` 是 100 次）。每次執行會把 SKILL.md 的文字和一個 prompt 送給模型，這一層記錄四個量測值：
- Activation rate 是有任何非空回覆的執行所占的比例，所以它顯示的是模型有沒有回答，不是 skill 該不該被觸發。
- Output quality 是回覆長度除以 500，上限 1.0。
- Failure rate 是出錯的執行所占的比例。
- Token efficiency 是 `1 - median_tokens / 8000`。
每個 prompt 都是應該觸發的，所以這一層從來不會檢查 skill 有沒有在不相關的請求中保持安靜。這一層的 JSON 包含針對它自己量測值的 Wilson、bootstrap 和 Clopper-Pearson 區間。複合分數的 `ci_lower` 和 `ci_upper` 欄位永遠是 null。
## 複合分數
第一步，每個維度會把存在的各層分數混合起來，並把混合權重在這些層之間重新正規化。第二步，把加權後的維度分數加總，並把維度權重在有分數的維度之間重新正規化。第三步，把總和乘以 100，再乘以 anti-pattern 的扣分係數。
| 維度 | 權重 | Static | Judge | Monte Carlo |
|---|---|---|---|---|
| `triggering_accuracy` | 0.25 | 0.15 | 0.25 | 0.60 |
| `orchestration_fitness` | 0.20 | 0.10 | 0.70 | 無 |
| `output_quality` | 0.15 | 無 | 0.40 | 0.60 |
| `scope_calibration` | 0.12 | 無 | 0.55 | 無 |
| `progressive_disclosure` | 0.10 | 0.80 | 無 | 無 |
| `token_efficiency` | 0.06 | 0.40 | 無 | 0.50 |
| `robustness` | 0.05 | 無 | 無 | 0.80 |
| `structural_completeness` | 0.03 | 0.90 | 無 | 無 |
| `code_template_quality` | 0.02 | 無 | 無 | 無 |
| `ecosystem_coherence` | 0.02 | 0.85 | 無 | 無 |
某個格子寫「無」，代表那一層對該維度不產生分數，即使 `LAYER_BLENDS` 列了權重也一樣。沒有任何一層會產生 `code_template_quality`，所以它永遠是未量測。對 plugin 目錄來說，複合分數是 static 層在這個 plugin 所有 skill 和 agent 上的平均分數，乘以 100，再乘以 plugin 的 anti-pattern 數量對應的扣分係數。
## 徽章與等級
徽章只由複合分數決定。Platinum 要至少 90，Gold 至少 80，Silver 至少 70，Bronze 至少 60。`Badge.from_scores` 接受 Elo 評分，但沒有任何指令會算出 Elo。Plugin 層級的徽章，包含每週 CI 報告裡的那些，只來自 static 層。`standard` 深度以上的 skill 層級徽章，也會納入實驗性質的那幾層。
每個有量測的維度，會在 0 到 100 的量表上得到一個字母等級，從 97 的 A+ 一路到 60 的 D-，低於 60 就是 F。
## 用法
```bash
plugin-eval score ./path/to/skill --depth quick # 只做 static lint
plugin-eval score ./path/to/skill # static 與 judge
plugin-eval certify ./path/to/skill # deep 深度
plugin-eval compare ./skill-a ./skill-b # 預設為 quick 深度
plugin-eval score ./path/to/skill --depth quick --output json --threshold 70
```
在 `standard` 深度以上，`score`、`certify` 和 `compare` 會在 stderr 印出提示，說明 judge 層和 Monte Carlo 層是實驗性質。對 plugin 目錄，CLI 則改為印出警告，說明只會跑 static 層。有指定 `--threshold` 時，只要複合分數低於該值，指令就會以 exit code 1 結束。`plugin-eval init` 會寫出 corpus index，但沒有其他指令會讀它。
reads it.
## 範例
下面是 JSON 輸出的節錄範例。腳本可以從中讀取 `composite.score`：
```json
{
"layers": [{"layer": "static", "score": 0.75, "sub_scores": {}, "anti_patterns": []}],
"composite": {"score": 76.9, "ci_lower": null, "ci_upper": null, "badge": "silver",
"confidence_label": "Estimated", "dimensions": []},
"elo": null
}
```
## 疑難排解
- 加了內容之後分數下降的話，檢查 JSON 裡的 `layers[0].anti_patterns`。
- 如果 quick 深度下 `triggering_accuracy` 偏低，在 description 加上觸發用語，例如「Use this skill when」，後面接幾個以逗號分隔的使用情境。
- Judge 分數在每次執行之間會變動，因為模型每次都會寫新的測試 prompt 和任務。需要可重複比較的時候，請用 static 層。
- 如果 stderr 說 judge 無法量測某些維度，請用 `uv sync --extra llm` 安裝 LLM extra。
## 相關
`eval-judge` agent 在 Claude Code 內替四個 judge 維度評分，`eval-orchestrator` agent 則負責執行 CLI 並合併結果。英文原文
---
name: evaluation-methodology
description: "PluginEval quality methodology, covering dimensions, rubrics, and scoring formulas. Use this skill when understanding how plugin quality is measured, when interpreting a low score on a specific dimension, when deciding how to improve a skill's triggering accuracy or orchestration fitness, when setting score thresholds for your marketplace, or when explaining quality badges to external partners like Neon."
---
# Evaluation methodology
PluginEval scores a skill or a plugin from 0 to 100 by combining up to three layers. The static
layer is a lint. It's fast and deterministic, and it's useful for checking structure. The LLM
judge and Monte Carlo layers are experimental, not validated against human labels, so treat their
numbers as rough signals. For the trace-based eval program, see `evals/README.md` at the
repository root.
The judge rubric anchors are in [references/rubrics.md](references/rubrics.md). Fixes for each
anti-pattern and tips for each dimension are in
[references/improving-scores.md](references/improving-scores.md).
## Evaluation depths
| Depth | Layers | Confidence label |
|---|---|---|
| `quick` | static | Estimated |
| `standard` (default for `score`) | static and judge | Assessed |
| `deep` (used by `certify`) | static, judge, and Monte Carlo with 50 runs | Certified |
| `thorough` | static, judge, and Monte Carlo with 100 runs | Certified+ |
The label names the depth that ran, not a check against human judgment. A plugin directory gets
the static layer only, whatever depth you ask for.
## Static layer (lint)
The static layer reads SKILL.md and makes no model calls. It computes seven sub-scores, and the
first six feed composite dimensions:
- `frontmatter_quality` (feeds `triggering_accuracy`)
- `orchestration_wiring` (feeds `orchestration_fitness`)
- `progressive_disclosure`, `structural_completeness`, `token_efficiency`, `ecosystem_coherence`
- `harness_portability`
`harness_portability` maps to no dimension, so it doesn't change a skill's composite score. It
does carry 6% of the static layer's own score. A plugin's score is built from that layer score,
so portability findings can lower a plugin's score a little. Its findings are not counted as
anti-patterns.
The static layer flags six anti-patterns: `OVER_CONSTRAINED`, `EMPTY_DESCRIPTION`,
`MISSING_TRIGGER`, `BLOATED_SKILL`, `ORPHAN_REFERENCE`, and `DEAD_CROSS_REF`. Each flag cuts the
score by 5%, down to a floor of 50%:
```text
penalty = max(0.5, 1.0 - 0.05 * anti_pattern_count)
```
The report prints a severity for each flag, but the penalty counts flags and ignores severity.
## LLM judge layer (experimental)
The judge layer makes four model calls and returns four holistic scores from 0 to 1:
- `triggering_accuracy`: Haiku reads the description and writes 10 test prompts, 5 that should
trigger and 5 that should not. It predicts the outcome for each prompt and reports its own F1.
Nothing checks those predictions against real triggering.
- `orchestration_fitness` and `scope_calibration`: Sonnet rates the skill on a five-point rubric.
- `output_quality`: Sonnet imagines three tasks and rates the output it expects.
The three Sonnet calls see only the first 3,000 characters of SKILL.md. Only one judge runs,
because nothing reads the `judges` setting.
## Monte Carlo layer (experimental)
Haiku writes 15 prompts that should trigger the skill, and the layer repeats them to reach 50
runs (100 at `thorough`). Each run sends the SKILL.md text and one prompt to the model, and the
layer records four measures:
- Activation rate is the share of runs with any non-empty reply, so it shows whether the model
answered, not whether the skill should have fired.
- Output quality is reply length divided by 500, capped at 1.0.
- Failure rate is the share of runs that errored.
- Token efficiency is `1 - median_tokens / 8000`.
Every prompt is one that should trigger, so the layer never checks that the skill stays out of
unrelated requests. The layer's JSON includes Wilson, bootstrap, and Clopper-Pearson intervals
for its own measures. The composite `ci_lower` and `ci_upper` fields are always null.
## Composite score
First, for each dimension, the engine blends the layer scores that exist, and it renormalizes
the blend weights over those layers. Second, it sums the weighted dimension scores, and it
renormalizes the dimension weights over the dimensions that have a score. Third, it multiplies
the sum by 100 and by the anti-pattern penalty.
| Dimension | Weight | Static | Judge | Monte Carlo |
|---|---|---|---|---|
| `triggering_accuracy` | 0.25 | 0.15 | 0.25 | 0.60 |
| `orchestration_fitness` | 0.20 | 0.10 | 0.70 | none |
| `output_quality` | 0.15 | none | 0.40 | 0.60 |
| `scope_calibration` | 0.12 | none | 0.55 | none |
| `progressive_disclosure` | 0.10 | 0.80 | none | none |
| `token_efficiency` | 0.06 | 0.40 | none | 0.50 |
| `robustness` | 0.05 | none | none | 0.80 |
| `structural_completeness` | 0.03 | 0.90 | none | none |
| `code_template_quality` | 0.02 | none | none | none |
| `ecosystem_coherence` | 0.02 | 0.85 | none | none |
A cell reads "none" when its layer produces no score for the dimension, even where
`LAYER_BLENDS` lists a weight. No layer produces `code_template_quality`, so it's always
unmeasured. For a plugin directory, the composite is the static layer's mean score across the
plugin's skills and agents, times 100, times the penalty for the plugin's anti-pattern count.
## Badges and grades
Badges come from the composite score alone. Platinum needs at least 90, Gold at least 80, Silver
at least 70, and Bronze at least 60. `Badge.from_scores` accepts an Elo rating, but no command
computes one. Plugin-level badges, including the ones in the weekly CI report, come from the
static layer alone. Skill-level badges at `standard` depth or deeper also include the
experimental layers.
Each measured dimension gets a letter grade on the 0 to 100 scale, from A+ at 97 down to D- at
60, and F below 60.
## Usage
```bash
plugin-eval score ./path/to/skill --depth quick # static lint only
plugin-eval score ./path/to/skill # static and judge
plugin-eval certify ./path/to/skill # deep depth
plugin-eval compare ./skill-a ./skill-b # quick depth by default
plugin-eval score ./path/to/skill --depth quick --output json --threshold 70
```
At `standard` depth or deeper, `score`, `certify`, and `compare` print a note on stderr that the
judge and Monte Carlo layers are experimental. For a plugin directory, the CLI prints a warning
that only the static layer runs instead. With `--threshold`, the command exits with code 1 when the
composite is below the value. `plugin-eval init` writes a corpus index, but no other command
reads it.
## Examples
An abridged example of the JSON output follows. Scripts can read `composite.score` from it:
```json
{
"layers": [{"layer": "static", "score": 0.75, "sub_scores": {}, "anti_patterns": []}],
"composite": {"score": 76.9, "ci_lower": null, "ci_upper": null, "badge": "silver",
"confidence_label": "Estimated", "dimensions": []},
"elo": null
}
```
## Troubleshooting
- When a score drops after you add content, check `layers[0].anti_patterns` in the JSON.
- If `triggering_accuracy` is low at quick depth, add a trigger phrase such as "Use this skill
when" to the description, followed by several comma-separated contexts.
- Judge scores change between runs, because the model writes new test prompts and tasks each
time. Use the static layer for comparisons you want to repeat.
- If stderr says the judge could not measure some dimensions, install the LLM extra with
`uv sync --extra llm`.
## Related
The `eval-judge` agent scores the four judge dimensions inside Claude Code, and the
`eval-orchestrator` agent runs the CLI and merges the results.