t - 分布

t - 分布，用于根据小样本来估计呈正态分布且方差未知的总体的均值的统计显著性。如果总体方差已知（例如在样本数量足够多时），则应该用正态分布来估计总体均值。

t-test 改进了 z-test 在样本量小于 30 时误差大的问题。

$\overline{X}_n = \frac{\sum_i^n X_i}{n}$

$S_n = \frac{1}{n-1}\sum_i^n (X_i - \overline X)^2$

$\frac{\overline X_n - \mu_0}{\sigma / \sqrt n}$

$t = \frac{\overline X_n - \mu_0}{S_n / \sqrt n}$

$t = \frac{\overline X_1 - \overline X_2}{S_{\overline X_1 - \overline X_2}}$

应用2

For example, given a sample with a sample variance 2 and sample mean of 10, taken from a sample set of 11 (10 degrees of freedom), using the formula

$\overline X \pm t_{\alpha, n-1}\cdot \frac{S_n}{n}$

we can determine that at 90% confidence, we have a true mean lying below 10.58 and over 9.41.

In other words, on average, 80% of the times that upper and lower thresholds are calculated by this method, the true mean is both below the upper threshold and above the lower threshold. This is not the same thing as saying that there is an 80% probability that the true mean lies between a particular pair of upper and lower thresholds that have been calculated by this method.

最小样本数计算

https://www.cnstat.org/samplesize/11/

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

http://www.evanmiller.org/ab-testing/sample-size.html

https://zhuanlan.zhihu.com/p/40919260

Statistical Power （$1 -\beta$）统计学功效：在假设检验中， 拒绝原假设后， 接受正确的替换假设的概率。在假设检验中有 $\alpha$ 错误和 $\beta$ 错误。$\alpha$ 错误是 FP 错误， $\beta$ 错误是 FN 错误。

Significiant Level 显著性水平：$\alpha$。同时也是 Type I Error 出现的概率，FP，第一类错误意味着新的产品对业务其实没有提升，我们却错误的认为有提升。

Minimum Detectable Effect 最小改善程度：$\delta = (t_{\alpha/2} + t_{\beta} \sigma \sqrt{2/n})$

A/B 测试最小样本： $n = 16 \frac{\sigma^2}{\delta^2}$