ecc-tools-cost-audit
用證據追查 ECC Tools 的 PR 暴增、配額繞過與燒錢
中文翻譯
--- name: ecc-tools-cost-audit description: 以證據為先的 ECC Tools 燒錢與計費稽核工作流。在調查 ECC Tools repo 中失控的 PR 建立、配額繞過、premium 模型外洩、重複 job，或 GitHub App 成本暴增時使用。 metadata: origin: ECC --- # ECC Tools 成本稽核 當使用者懷疑 ECC Tools GitHub App 在燒成本、建立過多 PR、繞過用量限制，或把免費使用者導向 premium 分析路徑時，就用這個 skill。 這是針對兄弟 repo [ECC-Tools](../../ECC-Tools) 的聚焦操作者工作流。它不是通用的計費 skill，也不是整個 repo 的 code review。 ## Skill 組合 相關時，把這些 ECC 原生 skill 拉進工作流： - `autonomous-loops`：用於跨 webhook、queue、計費與重試的有界多步驟稽核 - `agentic-engineering`：把請求路徑追蹤成一個個可證明的獨立單元 - `customer-billing-ops`：repo 行為和客戶影響的計算必須清楚分開時 - `search-first`：在發明 helper 或重寫 repo 內既有工具之前 - `security-review`：動到 auth、用量 gate、entitlement 或 secret 時 - `verification-loop`：證明重跑安全與修復後的確切狀態 - `tdd-workflow`：修復需要在 worker、router 或計費路徑補 regression 覆蓋時 ## 何時使用 - 使用者提到 ECC Tools 的燒錢速度、PR 遞迴、過度建立 PR、用量限制繞過，或 premium 模型外洩 - 任務在兄弟 repo `ECC-Tools` 裡，且取決於 webhook handler、queue worker、用量預留、PR 建立邏輯，或付費 gate 的執行 - 客戶回報 app 建了太多 PR、計費錯誤，或分析了程式碼卻沒有產出可用的結果 ## 範圍護欄 - 在兄弟 repo `ECC-Tools` 裡工作，不要在 `ECC` 裡 - 除非使用者明確要求修復，否則先以唯讀方式開始 - 追蹤分析燒錢時，不要去動不相關的計費、結帳或 UI 流程 - 在證明不是之前，把 app 產生的 branch 和 app 產生的 PR 當成紅旗的遞迴路徑 - 明確分開三件事： - repo 端的燒錢根因 - 面向客戶的計費影響 - 需要排進 backlog 後續處理的產品或 entitlement 缺口 ## 工作流 ### 1. 凍結 repo 範圍 - 切換到兄弟 repo `ECC-Tools` - 先檢查 branch 和本地 diff - 找出稽核的確切範圍： - webhook router - queue producer - queue consumer - PR 建立路徑 - 用量預留 / 計費路徑 - 模型路由路徑 ### 2. 先追蹤入口，再提出理論 - 先檢查 `src/index.*` 或主要的 entrypoint - 在建議修復之前，先對應出每一條 enqueue 路徑 - 確認哪些 GitHub event 共用同一個 queue 類型 - 確認 push、pull_request、synchronize、comment 或手動重跑的 event，會不會匯集到同一條昂貴的路徑 ### 3. 追蹤 worker 與副作用 - 檢查處理分析的 queue consumer 或排程 worker - 確認排進 queue 的分析是不是一定會以下列結尾： - 建立 PR - 建立 branch - 更新檔案 - 呼叫 premium 模型 - 用量增加 - 如果分析會花掉 token，卻在輸出被保存之前失敗，就歸類為「燒錢但輸出壞掉」 ### 4. 稽核高訊號的燒錢路徑 #### PR 倍增 - 檢查 PR helper 與 branch 命名 - 檢查去重、synchronize event 的處理，以及重用既有 PR - 如果 app 產生的 branch 可以重新進入分析，就把它視為優先級 0 的遞迴風險 #### 配額繞過 - 檢查配額是在哪裡被檢查、用量又是在哪裡被預留或遞增 - 如果配額在 enqueue 之前檢查，但用量只在 worker 內才計費，就把前門同時放行當成真正的 race #### Premium 模型外洩 - 檢查模型選擇、等級分支與 provider 路由 - 驗證有 premium key 時，免費或有上限的使用者是否仍能打到 premium 分析器 #### 重試燒錢 - 檢查重試 loop、重複的 queue job，以及確定性失敗的重跑 - 如果同一個非暫時性的錯誤能反覆花掉分析成本，要在改善品質之前先修這個 ### 5. 依燒錢順序修復 如果使用者要求改程式碼，依下列順序排修復優先級： 1. 停止自動的 PR 倍增 2. 停止配額繞過 3. 停止 premium 外洩 4. 停止重複 job 的 fanout 與無意義的重試 5. 補上重跑/更新的安全缺口 除非同一個根因明顯橫跨多個檔案，否則這一輪只做一到三個直接的修復。 ### 6. 用最小的證明步驟驗證 - 只重跑涵蓋被改動路徑的目標測試或整合切片 - 驗證燒錢路徑現在是： - 被擋下 - 被去重 - 降級成較便宜的分析 - 或提早被拒絕 - 精確說明最終狀態： - 本地已修改 - 本地已驗證 - 已 push - 已 deploy - 仍然被卡住 ## 高訊號的失敗模式 ### 1. 所有觸發共用同一個 queue 類型 如果 push、PR sync 和手動稽核都排進同一種 job，而 worker 又一律建立 PR，分析就等於 PR 垃圾訊息。 ### 2. enqueue 之後才預留用量 如果用量在前門檢查，卻只在 worker 內遞增，並行的請求可以全部通過 gate 並超出配額。 ### 3. 免費層走到 premium 路徑 如果免費的排隊 job 在 key 存在時仍能路由到 Anthropic 或其他 premium provider，那就是真正的花費外洩，即使使用者從沒看到 premium 的結果。 ### 4. App 產生的 branch 重新進入 webhook 如果 `pull_request.synchronize`、branch push 或 comment 觸發的執行在 app 自己的 branch 上被觸發，app 就會遞迴分析自己的輸出。 ### 5. 在確保可持久化之前就做昂貴的工作 如果系統花掉 token 之後，卻在建立 PR、更新檔案或 branch 衝突時失敗，它就是在燒成本卻沒有交付價值。 ## 常見陷阱 - 不要一開始就在 repo 裡亂逛；先釐清 webhook -> queue -> worker - 不要把客戶計費的推論和有程式碼支撐的產品事實混在一起 - 在最高燒錢的路徑被控制住之前，不要先修價值較低的品質問題 - 在窄範圍的證明步驟重跑過之前，不要宣稱燒錢已修好 - 除非使用者要求，否則不要 push 或 deploy - 如果不相關的 repo 本地變更已在進行中，不要去動它們 ## 驗證 - 根因要引用確切的檔案路徑與程式碼區域 - 修復依燒錢影響排序，不是依程式碼整潔度 - 要寫出證明用的指令 - 最終狀態要區分本地修改、驗證、push 與 deploy
英文原文
--- name: ecc-tools-cost-audit description: Evidence-first ECC Tools burn and billing audit workflow. Use when investigating runaway PR creation, quota bypass, premium-model leakage, duplicate jobs, or GitHub App cost spikes in the ECC Tools repo. metadata: origin: ECC --- # ECC Tools Cost Audit Use this skill when the user suspects the ECC Tools GitHub App is burning cost, over-creating PRs, bypassing usage limits, or routing free users into premium analysis paths. This is a focused operator workflow for the sibling [ECC-Tools](../../ECC-Tools) repo. It is not a generic billing skill and it is not a repo-wide code review pass. ## Skill Stack Pull these ECC-native skills into the workflow when relevant: - `autonomous-loops` for bounded multi-step audits that cross webhooks, queues, billing, and retries - `agentic-engineering` for tracing the request path into discrete, provable units - `customer-billing-ops` when repo behavior and customer-impact math must be separated cleanly - `search-first` before inventing helpers or re-implementing repo-local utilities - `security-review` when auth, usage gates, entitlements, or secrets are touched - `verification-loop` for proving rerun safety and exact post-fix state - `tdd-workflow` when the fix needs regression coverage in the worker, router, or billing paths ## When To Use - user says ECC Tools burn rate, PR recursion, over-created PRs, usage-limit bypass, or premium-model leakage - the task is in the sibling `ECC-Tools` repo and depends on webhook handlers, queue workers, usage reservation, PR creation logic, or paid-gate enforcement - a customer report says the app created too many PRs, billed incorrectly, or analyzed code without producing a usable result ## Scope Guardrails - work in the sibling `ECC-Tools` repo, not in `ECC` - start read-only unless the user clearly asked for a fix - do not mutate unrelated billing, checkout, or UI flows while tracing analysis burn - treat app-generated branches and app-generated PRs as red-flag recursion paths until proved otherwise - separate three things explicitly: - repo-side burn root cause - customer-facing billing impact - product or entitlement gaps that need backlog follow-up ## Workflow ### 1. Freeze repo scope - switch into the sibling `ECC-Tools` repo - check branch and local diff first - identify the exact surface under audit: - webhook router - queue producer - queue consumer - PR creation path - usage reservation / billing path - model routing path ### 2. Trace ingress before theorizing - inspect `src/index.*` or the main entrypoint first - map every enqueue path before suggesting a fix - confirm which GitHub events share a queue type - confirm whether push, pull_request, synchronize, comment, or manual re-run events can converge on the same expensive path ### 3. Trace the worker and side effects - inspect the queue consumer or scheduled worker that handles analysis - confirm whether a queued analysis always ends in: - PR creation - branch creation - file updates - premium model calls - usage increments - if analysis can spend tokens and then fail before output is persisted, classify it as burn-with-broken-output ### 4. Audit the high-signal burn paths #### PR multiplication - inspect PR helpers and branch naming - check dedupe, synchronize-event handling, and existing-PR reuse - if app-generated branches can re-enter analysis, treat that as a priority-0 recursion risk #### Quota bypass - inspect where quota is checked versus where usage is reserved or incremented - if quota is checked before enqueue but usage is charged only inside the worker, treat concurrent front-door passes as a real race #### Premium-model leakage - inspect model selection, tier branching, and provider routing - verify whether free or capped users can still hit premium analyzers when premium keys are present #### Retry burn - inspect retry loops, duplicate queue jobs, and deterministic failure reruns - if the same non-transient error can spend analysis repeatedly, fix that before quality improvements ### 5. Fix in burn order If the user asked for code changes, prioritize fixes in this order: 1. stop automatic PR multiplication 2. stop quota bypass 3. stop premium leakage 4. stop duplicate-job fanout and pointless retries 5. close rerun/update safety gaps Keep the pass bounded to one to three direct fixes unless the same root cause clearly spans multiple files. ### 6. Verify with the smallest proving steps - rerun only the targeted tests or integration slices that cover the changed path - verify whether the burn path is now: - blocked - deduped - downgraded to cheaper analysis - or rejected early - state the final status exactly: - changed locally - verified locally - pushed - deployed - still blocked ## High-Signal Failure Patterns ### 1. One queue type for all triggers If pushes, PR syncs, and manual audits all enqueue the same job and the worker always creates a PR, analysis equals PR spam. ### 2. Post-enqueue usage reservation If usage is checked at the front door but only incremented in the worker, concurrent requests can all pass the gate and exceed quota. ### 3. Free tier on premium path If free queued jobs can still route into Anthropic or another premium provider when keys exist, that is real spend leakage even if the user never sees the premium result. ### 4. App-generated branches re-enter the webhook If `pull_request.synchronize`, branch pushes, or comment-triggered runs fire on app-owned branches, the app can recursively analyze its own output. ### 5. Expensive work before persistence safety If the system can spend tokens and then fail on PR creation, file update, or branch collision, it is burning cost without shipping value. ## Pitfalls - do not begin with broad repo wandering; settle webhook -> queue -> worker first - do not mix customer billing inference with code-backed product truth - do not fix lower-value quality issues before the highest-burn path is contained - do not claim burn is fixed until the narrow proving step was rerun - do not push or deploy unless the user asked - do not touch unrelated repo-local changes if they are already in progress ## Verification - root causes cite exact file paths and code areas - fixes are ordered by burn impact, not code neatness - proving commands are named - final status distinguishes local change, verification, push, and deployment