stat-hypothesis-testing
教你做假設檢定、選對統計方法，並分清顯著跟有用
中文翻譯
---
name: "stat-hypothesis-testing"
description: "進行統計假設檢定，包括虛無/對立假設的擬定、p 值、第一型/第二型錯誤，以及檢定統計量的選擇。當使用者需要判斷結果是否具統計顯著性、挑選合適的統計檢定、正確解讀 p 值或評估研究發現時使用這個 skill，就算他們說的是「這個結果顯著嗎」、「我該用哪種統計檢定」或「這個 p 值是什麼意思」也一樣。"
metadata:
category: "WP-21 設計/資訊/傳播/公衛"
tags: ["statistics", "hypothesis-testing", "p-value", "inference"]
---
# 假設檢定
## 框架
```
IRON LAW: Statistical Significance ≠ Practical Significance
（鐵律：統計顯著 ≠ 實務顯著）
p 值 < 0.05 的意思是：在虛無假設成立的前提下，這個結果不太可能出現。
它「不代表」這個結果重要、很大，或有實務意義。
只要樣本夠大，0.1% 的轉換率差異也會變成「統計顯著」，
但實務上一點價值都沒有。
報告 p 值時，一定要同時報告效果量（effect size）。
```
```
IRON LAW: State Hypotheses BEFORE Looking at Data
（鐵律：先定假設，再看資料）
H₀（虛無假設）和 H₁（對立假設）必須在分析資料之前就定義好。
看完資料才選假設 = p-hacking = 學術造假。
「我們發現一個有趣的模式，來用同一批資料測測看」是無效的做法。
```
### 核心概念
| 概念 | 定義 |
|---------|-----------|
| **H₀（虛無假設）** | 預設假設：沒有效果、沒有差異 |
| **H₁（對立假設）** | 你想證明的：確實有效果/差異 |
| **p-value** | 假設 H₀ 為真，看到這個結果（或更極端結果）的機率 |
| **α（顯著水準）** | 拒絕 H₀ 的門檻（通常是 0.05） |
| **第一型錯誤（α）** | H₀ 其實為真卻拒絕它（偽陽性） |
| **第二型錯誤（β）** | H₁ 為真卻沒能拒絕 H₀（偽陰性） |
| **檢定力 Power（1-β）** | 偵測到真實效果的機率（目標：≥ 0.8） |
| **效果量 Effect size** | 差異的大小（Cohen's d、odds ratio、R²） |
### 檢定選擇指南
| 資料類型 | 組別 | 檢定 |
|-----------|--------|------|
| 連續、常態、2 組 | 獨立 | 獨立樣本 t 檢定 |
| 連續、常態、2 組 | 成對/前後測 | 成對樣本 t 檢定 |
| 連續、常態、3 組以上 | 獨立 | 單因子 ANOVA |
| 連續、非常態 | 2 組 | Mann-Whitney U |
| 類別 | 2 組以上 | 卡方檢定 |
| 連續、看關聯性 | 2 個變數 | Pearson 相關（常態）/ Spearman（非常態） |
| 二元結果 | 預測變數 | Logistic regression |
### 檢定流程
1. **陳述假設**：H₀ 和 H₁，要帶具體參數
2. **選檢定**：依資料類型、分布和組別來選（用上面的指南）
3. **設 α**：通常 0.05（不一樣的話要說明理由）
4. **計算**：跑檢定，取得檢定統計量和 p 值
5. **決策**：p < α → 拒絕 H₀；p ≥ α → 無法拒絕 H₀
6. **報告**：效果量 + 信賴區間 + p 值（不要只寫「顯著」）
## 輸出格式
```markdown
# Hypothesis Test: {Research Question}
## Hypotheses
- H₀: {null — no effect/difference}
- H₁: {alternative — there IS an effect/difference}
- α = {0.05 or other}
## Test Selection
- Test: {name}
- Rationale: {why this test fits the data}
- Assumptions checked: {normality, independence, equal variance}
## Results
- Test statistic: {value}
- p-value: {value}
- Effect size: {value and interpretation}
- 95% CI: [{lower}, {upper}]
## Decision
{Reject / Fail to reject H₀}
## Interpretation
{What this means in practical terms, with effect size context}
```
## 常見陷阱
- **「無法拒絕 H₀」≠「H₀ 為真」**：沒有證據不等於證據不存在。你可能只是檢定力不夠，偵測不到真實的效果。
- **多重比較會膨脹第一型錯誤**：用 α=0.05 測 20 個假設，光靠運氣就預期會有 1 個偽陽性。要套用 Bonferroni 或 FDR 校正。
- **檢定前先檢查假設條件**：t 檢定假設常態和變異數相等，違反假設會讓結果失效。假設不成立時，改用無母數的替代方法。
- **樣本數決定檢定力**：樣本太小會漏掉真實效果（第二型錯誤）。收資料「之前」就要先算出所需樣本數。
- **p 值「不是」H₀ 為真的機率**：它是在 H₀ 成立下出現這份資料的機率，兩者本質上不同（基率謬誤）。
## 參考資料
- 樣本數計算，見 `references/sample-size.md`
- 無母數檢定替代方案，見 `references/nonparametric-tests.md`英文原文
---
name: "stat-hypothesis-testing"
description: "Conduct statistical hypothesis testing including null/alternative hypothesis formulation, p-values, Type I/II errors, and test statistic selection. Use this skill when the user needs to determine whether a result is statistically significant, choose the right statistical test, interpret p-values correctly, or evaluate research findings — even if they say 'is this result significant', 'which statistical test should I use', or 'what does this p-value mean'."
metadata:
category: "WP-21 設計/資訊/傳播/公衛"
tags: ["statistics", "hypothesis-testing", "p-value", "inference"]
---
# Hypothesis Testing
## Framework
```
IRON LAW: Statistical Significance ≠ Practical Significance
A p-value < 0.05 means the result is unlikely under the null hypothesis.
It does NOT mean the result is important, large, or practically meaningful.
With a large enough sample, a 0.1% conversion rate difference becomes
"statistically significant" but is practically worthless.
ALWAYS report effect size alongside p-value.
```
```
IRON LAW: State Hypotheses BEFORE Looking at Data
H₀ (null) and H₁ (alternative) must be defined before data analysis.
Choosing hypotheses after seeing the data = p-hacking = scientific fraud.
"We found an interesting pattern, let's test it on the same data" is invalid.
```
### Core Concepts
| Concept | Definition |
|---------|-----------|
| **H₀ (Null)** | Default assumption: no effect, no difference |
| **H₁ (Alternative)** | What you want to show: there IS an effect/difference |
| **p-value** | Probability of seeing this result (or more extreme) IF H₀ is true |
| **α (significance level)** | Threshold for rejecting H₀ (typically 0.05) |
| **Type I error (α)** | Rejecting H₀ when it's actually true (false positive) |
| **Type II error (β)** | Failing to reject H₀ when H₁ is true (false negative) |
| **Power (1-β)** | Probability of detecting a real effect (target: ≥ 0.8) |
| **Effect size** | Magnitude of the difference (Cohen's d, odds ratio, R²) |
### Test Selection Guide
| Data Type | Groups | Test |
|-----------|--------|------|
| Continuous, normal, 2 groups | Independent | Independent t-test |
| Continuous, normal, 2 groups | Paired/before-after | Paired t-test |
| Continuous, normal, 3+ groups | Independent | One-way ANOVA |
| Continuous, non-normal | 2 groups | Mann-Whitney U |
| Categorical | 2+ groups | Chi-square test |
| Continuous, relationship | 2 variables | Pearson correlation (normal) / Spearman (non-normal) |
| Binary outcome | Predictors | Logistic regression |
### Testing Process
1. **State hypotheses**: H₀ and H₁ with specific parameters
2. **Choose test**: Based on data type, distribution, and groups (use guide above)
3. **Set α**: Usually 0.05 (justify if different)
4. **Calculate**: Run the test, get test statistic and p-value
5. **Decide**: p < α → reject H₀; p ≥ α → fail to reject H₀
6. **Report**: Effect size + confidence interval + p-value (not just "significant")
## Output Format
```markdown
# Hypothesis Test: {Research Question}
## Hypotheses
- H₀: {null — no effect/difference}
- H₁: {alternative — there IS an effect/difference}
- α = {0.05 or other}
## Test Selection
- Test: {name}
- Rationale: {why this test fits the data}
- Assumptions checked: {normality, independence, equal variance}
## Results
- Test statistic: {value}
- p-value: {value}
- Effect size: {value and interpretation}
- 95% CI: [{lower}, {upper}]
## Decision
{Reject / Fail to reject H₀}
## Interpretation
{What this means in practical terms, with effect size context}
```
## Gotchas
- **"Fail to reject H₀" ≠ "H₀ is true"**: Absence of evidence is not evidence of absence. You may lack power to detect a real effect.
- **Multiple comparisons inflate Type I error**: Testing 20 hypotheses at α=0.05 → expect 1 false positive by chance. Apply Bonferroni or FDR correction.
- **Check assumptions before testing**: t-test assumes normality and equal variance. Violating assumptions invalidates results. Use non-parametric alternatives when assumptions fail.
- **Sample size determines power**: Small samples miss real effects (Type II error). Calculate required sample size BEFORE collecting data.
- **p-value is NOT the probability that H₀ is true**: It's the probability of the data given H₀. These are fundamentally different things (base rate fallacy).
## References
- For sample size calculation, see `references/sample-size.md`
- For non-parametric test alternatives, see `references/nonparametric-tests.md`