mkt-ab-testing
設計行銷 A/B 測試，一次只改一個變數再看結果
中文翻譯
---
name: "mkt-ab-testing"
description: "為登陸頁、電子郵件活動、廣告素材和定價設計並執行行銷 A/B 測試，附上正確的測試設計與結果分析。使用者需要測試行銷版本的差異、透過實驗提升轉換率，或要在兩種活動做法之間做決定時使用，就算他們說的是「哪個版本表現比較好」、「測試這個登陸頁」、「A/B 測試我們的電子郵件主旨」或「我們該不該改 CTA」也一樣。"
metadata:
category: "WP-09 數位行銷"
tags: ["digital-marketing", "ab-testing", "conversion", "experimentation"]
---
# 行銷 A/B 測試
## 框架
```
IRON LAW: One Variable at a Time
If you change the headline AND the image AND the CTA simultaneously,
you cannot know which change caused the result. Test ONE variable per
experiment. If you need to test multiple changes, use sequential tests
or multivariate testing (MVT) with sufficient traffic.
```
### 該測什麼（依影響程度）
| 元素 | 預期提升 | 需要的流量 | 優先順序 |
|---------|-------------|---------------|---------|
| **優惠／定價** | 10-50% | 中 | 最高 |
| **標題／主旨** | 5-30% | 低 | 高 |
| **CTA（文字、顏色、位置）** | 5-20% | 低 | 高 |
| **頁面版面** | 5-15% | 中 | 中 |
| **圖片／影片** | 3-15% | 中 | 中 |
| **表單欄位** | 5-25%（減少欄位 = 轉換率更高） | 低 | 中 |
| **社會證明的位置** | 3-10% | 中 | 較低 |
### 測試設計
1. **假設**：「把 [變數] 從 [A] 改成 [B]，會讓 [指標] 提升 [X%]，因為 [理由]」
2. **主要指標**：「一個」決定勝負的指標（轉換率、每位訪客營收、註冊率）
3. **護欄指標**：絕對不能變差的指標（跳出率、頁面載入時間、每位用戶營收）
4. **流量分配**：對照組與變體組 50/50（標準做法）
5. **樣本數**：開始前先算好（公式請看 stat-ab-testing）
6. **期間**：至少 1-2 個完整的工作週（涵蓋星期幾的效應）
### 常見的行銷測試
| 測試 | 對照組（A） | 變體組（B） | 指標 |
|------|-----------|------------|--------|
| 電子郵件主旨 | "Your weekly update" | "3 trends you missed this week" | 開信率 |
| 登陸頁 CTA | "Sign Up" | "Start Free Trial" | 點擊率 |
| 定價頁 | 顯示 3 個方案 | 顯示 2 個方案 + 「最受歡迎」徽章 | 轉換率 |
| 廣告素材 | 產品照 | 帶有產品的生活風格照 | CTR → 轉換 |
| 表單長度 | 8 個欄位 | 4 個欄位 | 表單完成率 |
### 分析與決策
| 結果 | 決定 | 行動 |
|--------|---------|--------|
| B 勝出、p < 0.05、提升幅度有意義 | 上線 B | 部署變體，開始下一個測試 |
| B 勝出、p < 0.05、但提升很小（<1%） | 不上線 | 提升幅度不值得承擔改動的風險 |
| 沒有顯著差異 | 保留 A | A 是已知的基準；去測別的東西 |
| B 在主要指標勝出，但護欄指標輸了 | 調查 | 可能需要重新設計變體 |
## 輸出格式
```markdown
# A/B Test Plan: {Test Name}
## Hypothesis
Changing {variable} from {A} to {B} will increase {metric} by {X%} because {reasoning}.
## Design
- Primary metric: {metric}
- Guardrail: {metric(s)}
- Split: 50/50
- Sample size: {N per variant}
- Duration: {days/weeks}
## Results
| Metric | Control | Variant | Diff | CI (95%) | Significant? |
|--------|---------|---------|------|----------|-------------|
| {primary} | {value} | {value} | {±%} | [{lower}, {upper}] | Y/N |
## Decision
{Ship / Don't ship / Extend} — {rationale}
```
## 注意事項
- **不要因為「看起來不錯」就提早停止**：偷看結果、一看到顯著就停，會讓偽陽性率膨脹到 30% 以上。要跑到計畫的樣本數。
- **星期幾的效應**：週一的訪客和週六的訪客行為不同。測試一定要跑至少 1-2 個完整的週。
- **新奇效應**：新設計可能因為好奇而暫時提升。要等 2 週以上，看效果是否持續。
- **贏家的詛咒**：因為統計雜訊，測試估計出來的提升幅度常常比真實的提升大。部署後的實際影響要預期會比較小。
- **不要什麼都測，要測重要的**：花力氣跑 20 個按鈕顏色的小測試，卻忽略定價頁，是把精力放錯地方。先測影響大的元素。
## 腳本
| 腳本 | 說明 | 用法 |
|--------|-------------|-------|
| `scripts/ab_test.py` | 雙比例 z 檢定，附效果量與樣本數規劃 | `python scripts/ab_test.py --help` |
執行 `python scripts/ab_test.py --verify` 可以跑內建的健全性測試。
## 參考資料
- 統計方法論（樣本數、p 值）請看 stat-ab-testing skill
- 多變量測試設計請看 `references/mvt-design.md`英文原文
---
name: "mkt-ab-testing"
description: "Design and execute marketing A/B tests for landing pages, email campaigns, ad creatives, and pricing with proper test design and result analysis. Use this skill when the user needs to test marketing variations, improve conversion rates through experimentation, or decide between two campaign approaches — even if they say 'which version performs better', 'test this landing page', 'A/B test our email subject line', or 'should we change our CTA'."
metadata:
category: "WP-09 數位行銷"
tags: ["digital-marketing", "ab-testing", "conversion", "experimentation"]
---
# Marketing A/B Testing
## Framework
```
IRON LAW: One Variable at a Time
If you change the headline AND the image AND the CTA simultaneously,
you cannot know which change caused the result. Test ONE variable per
experiment. If you need to test multiple changes, use sequential tests
or multivariate testing (MVT) with sufficient traffic.
```
### What to Test (by Impact)
| Element | Expected Lift | Traffic Needed | Priority |
|---------|-------------|---------------|---------|
| **Offer/Pricing** | 10-50% | Medium | Highest |
| **Headline/Subject line** | 5-30% | Low | High |
| **CTA (text, color, placement)** | 5-20% | Low | High |
| **Page layout** | 5-15% | Medium | Medium |
| **Image/Video** | 3-15% | Medium | Medium |
| **Form fields** | 5-25% (reduction = higher CVR) | Low | Medium |
| **Social proof placement** | 3-10% | Medium | Lower |
### Test Design
1. **Hypothesis**: "Changing [variable] from [A] to [B] will increase [metric] by [X%] because [reasoning]"
2. **Primary metric**: ONE metric that determines winner (conversion rate, revenue per visitor, signup rate)
3. **Guardrail metrics**: Metrics that must NOT degrade (bounce rate, page load time, revenue per user)
4. **Traffic split**: 50/50 between control and variant (standard)
5. **Sample size**: Calculate before starting (see stat-ab-testing for formula)
6. **Duration**: Minimum 1-2 full business weeks (capture day-of-week effects)
### Common Marketing Tests
| Test | Control (A) | Variant (B) | Metric |
|------|-----------|------------|--------|
| Email subject | "Your weekly update" | "3 trends you missed this week" | Open rate |
| Landing page CTA | "Sign Up" | "Start Free Trial" | Click rate |
| Pricing page | Show 3 plans | Show 2 plans + "most popular" badge | Conversion rate |
| Ad creative | Product photo | Lifestyle photo with product | CTR → conversion |
| Form length | 8 fields | 4 fields | Form completion rate |
### Analysis & Decision
| Result | Decision | Action |
|--------|---------|--------|
| B wins, p < 0.05, meaningful lift | Ship B | Deploy variant, start next test |
| B wins, p < 0.05, tiny lift (<1%) | Don't ship | Lift not worth the change risk |
| No significant difference | Keep A | A is the known quantity; test something else |
| B wins on primary but loses on guardrail | Investigate | May need to redesign variant |
## Output Format
```markdown
# A/B Test Plan: {Test Name}
## Hypothesis
Changing {variable} from {A} to {B} will increase {metric} by {X%} because {reasoning}.
## Design
- Primary metric: {metric}
- Guardrail: {metric(s)}
- Split: 50/50
- Sample size: {N per variant}
- Duration: {days/weeks}
## Results
| Metric | Control | Variant | Diff | CI (95%) | Significant? |
|--------|---------|---------|------|----------|-------------|
| {primary} | {value} | {value} | {±%} | [{lower}, {upper}] | Y/N |
## Decision
{Ship / Don't ship / Extend} — {rationale}
```
## Gotchas
- **Don't stop early because it "looks good"**: Peeking at results and stopping when you see significance inflates false positive rates to 30%+. Run to planned sample size.
- **Day-of-week effects**: Monday visitors behave differently from Saturday visitors. Always run tests for at least 1-2 complete weeks.
- **Novelty effect**: A new design may get a temporary lift from curiosity. Wait 2+ weeks to see if the effect sustains.
- **Winner's curse**: The estimated lift from a test is often larger than the true lift due to statistical noise. Expect the actual impact after deployment to be smaller.
- **Don't test everything — test what matters**: Running 20 small tests on button colors while ignoring the pricing page is misallocating effort. Test high-impact elements first.
## Scripts
| Script | Description | Usage |
|--------|-------------|-------|
| `scripts/ab_test.py` | Two-proportion z-test with effect size and sample-size planning | `python scripts/ab_test.py --help` |
Run `python scripts/ab_test.py --verify` to execute built-in sanity tests.
## References
- For statistical methodology (sample size, p-values), see the stat-ab-testing skill
- For multivariate testing design, see `references/mvt-design.md`