We will continue with the basic terms of hypothesis testing. The first one is the significance level. Alpha, as popularly known, is set by the person in charge of the testing and signifies the strength of the evidence required to establish the tester’s proposition (alternative hypothesis). We are familiar with the alpha of 0.05 (5%). But how does one choose the right level?
The famous analogy is the court cases. For civil cases (deal with personal rights), scholars define 51% of the evidence to support a claim. On the other hand, criminal cases may require far more, say, more than 90% of the evidence, for a verdict against the suspect. It may go to 99% when the potential punishment of the guilty is severe. You may look back at an older post to see the significance of where the judges draw their lines.
In the same way, the analyst may decide on a stricter significance level or a lower probability of rejecting the null hypothesis if the stakes are high. Putting it differently, an alpha of 0.01 means a 1% probability that the test will produce a statistically significant result if the null hypothesis is correct.
Reference
Jim Frost, “Hypothesis Testing: An Intuitive Guide for Making Data Driven Decisions”
