Abuses of Power: Persistent Problems of Post-Hoc Power

A post-hoc power calculation offers no new information.

Statistical power is a concept in null hypothesis significance testing. It is the probability a test rejects the null hypothesis, making the ‘correct’ decision. In that situation, this is when the test gets a ‘significant’ result.

(Image: Alison Yuhan Yao/Towards Data Science. The bottom-right box should say: “False Negative”.)

Statistical power concerns study design. Clinical trials need to be big enough for researchers to find there is a treatment effect if one exists. Such a study designed to have “80% power” will reject the null hypothesis in four of five trials. Sample size calculations form an important part of planning a study or experiment.

One practice is the calculation of “post-hoc power”. This is also called “achieved”, “retrospective”, or “observed” power. This calculation claims to estimate the test’s power, given an observed effect size. A 2019 paper published in the Journal of Surgical Research focuses on post-hoc power. The paper suggested adding this calculation to statistical guidelines. Other journals support this approach when publishing analyses. Common software tools like SPSS also feature “post-hoc power”.

Statisticians have documented many times post-hoc power calculations are not useful. The problem is the p-value determines “observed” power. Post-hoc…