benchmark-optimization-loop

affaan-m ｜測試與驗證｜原文｜ MIT

把「讓它變快」變成有基準線、正確性關卡的實測最佳化迴圈

中文翻譯

---
name: benchmark-optimization-loop
description: 把「讓它變快」的要求，轉成有界限、可量測的最佳化迴圈：先量基準線，產生每個只驗證一個假設的變體，用正確性關卡對每個變體做 benchmark，再把最快且安全的變體升級，並附上可重現的指令。被要求加速某個東西、嘗試很多變體、做遞迴最佳化、對延遲/吞吐量/成本做 benchmark，或靠重複量測的測試挑出最佳實作時使用。
license: MIT
metadata:
  origin: ECC
tools: Read, Write, Edit, Bash, Grep, Glob
---

# Benchmark Optimization Loop

用這個 skill 把「讓它快 20 倍」或「嘗試 50 輪遞迴最佳化」
轉成有界限、可量測、而且真的能改善系統的迴圈。

## 必要的基準線

這些東西沒齊之前，不要開始最佳化：

- 要最佳化的操作；
- 必須維持通過的正確性關卡；
- 指標：實際耗時、p95 延遲、每秒列數、每次執行的成本、記憶體、錯誤率；
- 目前的基準線；
- 搜尋預算：最多幾個變體、最長時間、最高花費、最大的資料影響。

如果使用者要求不切實際的目標，保留那份企圖心，但要讓迴圈
有界限、可量測。

## 迴圈

1. 量測基準線。
2. 依證據找出瓶頸。
3. 產生變體，每個變體只驗證一個假設。
4. 用相同形狀的輸入執行這些變體。
5. 淘汰在正確性、安全性或可重現性上失敗的變體。
6. 升級最快且安全的變體。
7. 把勝出的路徑寫成 script、指令、測試、設定或文件固定下來。
8. 重跑基準線和勝出者，確認差異。

## 變體表

像這樣追蹤變體：

```text
Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited
```

## 遞迴搜尋

遞迴或超參數的工作：

- 把每次執行都持久化到帳本裡；
- 跟前一個被接受的勝出者比較，不要只跟上一次執行比；
- 保留一個 holdout 或重放檢查；
- 當改善幅度落在雜訊範圍內、正確性失敗、成本超出
  預算，或搜尋開始改動超過它能解釋的變數數量時，就停下來。

除非搜尋空間真的窮舉過了，否則要用「實測最佳的安全變體」這種說法，不要說「全域最佳解」。

## 升級關卡

變體要通過以下條件，才能成為新的預設值：

- 正確性測試通過；
- 效能差異已重複驗證或已有解釋；
- 回復的方式很明確；
- 這個修改已經寫進版本控制或持久的 runbook；
- 最後的摘要包含確切的指令和量測數據。

英文原文

---
name: benchmark-optimization-loop
description: Convert 'make it faster' requests into a bounded measured optimization loop — baseline first, generate one-hypothesis variants, benchmark each against a correctness gate, and promote the fastest safe variant with reproducible commands. Use when asked to speed something up, try many variants, run recursive optimization, benchmark latency/throughput/cost, or pick the best implementation by repeated measured tests.
license: MIT
metadata:
origin: ECC
tools: Read, Write, Edit, Bash, Grep, Glob
---

# Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive
optimizations" into a bounded measured loop that can actually improve a system.

## Required Baseline

Do not optimize until these exist:

- the operation being optimized;
- the correctness gate that must stay green;
- the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;
- the current baseline;
- the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop
bounded and measurable.

## Loop

1. Measure the baseline.
2. Identify bottlenecks from evidence.
3. Generate variants that test one hypothesis each.
4. Run variants with the same input shape.
5. Reject variants that fail correctness, safety, or reproducibility.
6. Promote the fastest safe variant.
7. Codify the winning path in a script, command, test, config, or doc.
8. Rerun the baseline and winner to confirm the delta.

## Variant Table

Track variants like this:

## Recursive Search

For recursive or hyperparameter work:

- persist every run to a ledger;
- compare against the prior accepted winner, not only the previous run;
- keep a holdout or replay check;
- stop when improvement is within noise, correctness fails, cost exceeds the
budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless
the search space was actually exhaustive.

## Promotion Gate

A variant cannot become the new default until:

- correctness tests pass;
- the performance delta is repeated or explained;
- rollback is obvious;
- the change is encoded in source control or a durable runbook;
- the final summary includes exact commands and measurements.