No open-access location is known for this record yet.
No open-access location is known for this record yet.
Dear Editor, In a recent paper entitled ‘Choosing optimal method to combine p-values’ by Sungho Won et al., the authors proposed a ‘most powerful’ method to combine p-values, under the strong assumption that the effect sizes or their ratios are known 1. They used the ‘standardized expected difference in means’ as the weights in the Liptak method with one-sided tests. They also extended these weights to combine p-values from two-sided tests. Through simulations they compared the performance of their proposed methods with others, including the Liptak method and the Fisher method. They claimed that the new methods outperformed others. However, through simulations with the same settings used by Won et al., we found that their proposed method does not outperform the methods of Fisher and Liptak. On the contrary, our simulation results show that Won et al.'s method actually performs worse than the methods mentioned above. Tables I–IV and Figures 1-4 give the simulation results under four different situations that were considered in Figure 2 of Won et al. 1. Power estimated at significance level 0.01 from Won's method, Liptak method, Fisher method, and Generalized Fisher (G. Fisher) method when the mean differences are 0.1 for both X and Y with same sample sizes. Power estimated at significance level 0.01 from Won's method, Liptak method, Fisher method, and Generalized Fisher (G.Fisher) method when the mean differences are 0.1 for both X and Y with different sample sizes N2 = 5N1. Power estimated at significance level 0.01 from Won's method, Liptak method, Fisher method, and Generalized Fisher (G.Fisher) method when the mean differences are 0.02 for both X and Y with same sample sizes. Power estimated at significance level 0.01 from Won's method, Liptak method, Fisher method, and Generalized Fisher (G.Fisher) method when the mean differences are 0.02 for both X and Y with different sample sizes N2 = 5N1. One of the possible reasons that Won et al.'s method had the highest power in their simulations is that they used an incorrect method to obtain p-values. Under the assumption that the effect sizes or their ratios are known, the appropriate way to obtain p-values is from one-sided, rather than two-sided, t-test. Therefore, the one-sided Won et al. method, which is the same as the Liptak method, should be used. From our simulations (Tables I–IV, and Figures 1-4 ), it can be seen that the Liptak method indeed outperforms the Fisher method. However, Won et al. did not include a very important method, the generalized Fisher method due to Lancaster 2, 3. In our simulations, we included this method and chose the sample sizes as the degrees of freedom in the Lancaster method. In our simulations, the Lancaster method performed similarly to the Liptak method. Since the generalized Fisher method did not make any assumption about the effect sizes, it is more robust than Won et al.'s method (i.e. Liptak method), and it should be considered under these situations. In summary, if the true effect sizes or their ratios are known, or even under a weaker assumption that the signs (negative or positive) of effects are known, a one-sided test is more powerful and more appropriate to obtain p-values than two-sided test. The method by Won et al. is at best as powerful as the Liptak method. However, the generalized Fisher method due to Lancaster is as powerful as, but more robust than the Liptak method and should be considered.
Loading…
Stored only when both papers are in the corpus — 7 citations exist upstream.
Imported from OpenAlex (CC0) — nothing here is generated.