Fig. 1

Evidence pyramid. Suggested levels of evidence for results of benchmark studies designed for the comparison of statistical methods using real data. A neutral study is conducted by researchers that do not have a preference for any particular method and are (at least as a collective) approximately equally experienced with each of the considered methods. A non-neutral study is one in which the researchers have a potential conscious or subconscious interest in the demonstration of the superiority of a given method (the “preferred method”) or have greater experience in one or more of the methods (again, the “preferred method”) to the extent that it may bias the results. A non-preferred method is a statistical method from a non-neutral study but not that or those method(s) thought to be preferred. Bias in non-neutral studies can advantage preferred methods and disadvantage non-preferred methods, or both