In the previous post, we saw the problem of indiscriminately using the threshold p-value of 0.05 to test significance. Benjamin and Berger, in their 2019 publication in The American Statistician, urge the scientific community to be more transparent and make three recommendations to manage such situations.
Recommendation 1: Reduce alpha from 5% to 0.5%
It is probably a pragmatic solution for people using p-value-based null hypothesis testing. We know 0.005 corresponds to a Bayes Factor of ca. 25, which can produce good posterior odds for prior odds as low as 1:10. But what happens if the prior odds are lower than 1:10 (say, 1:100 or 1:1000)?
Recommendation 2: Report Bayes Factor
Bayes factor gives a different perspective on the validity of the alternative hypothesis (finding) against the null hypothesis (default). Once it is reported, the readers will have a feel of the strength of the discovery.
| Bayes Factor (BF10) | Interpretation |
| > 100 | Decisive evidence for H1 |
| 10 – 100 | Strong evidence for H1 |
| 3.2 – 10 | Substantial evidence for H1 |
| 1 – 3.2 | No real evidence for H1 |
Recommendation 3: Report Prior and Posterior Odds
The best service to the community is when researchers estimate (and report) the prior odds for the discovery and how the evidence has transformed them to the posterior.
Reference
[1] Benjamin, D. J.; Berger, J. O., “Three Recommendations for Improving the Use of p-Values”, The American Statistician, 73:sup1, 186-191, DOI: 10.1080/00031305.2018.1543135
[2] Kass, R. E.; Raftery, A. E., Bayes Factors, Journal of the American Statistical Association, 1995, 90(43), 773
