What is a Valid Scientific Study? An Analysis of Selection Criteria Used by the What Works Clearinghouse
Meta-analyses and reviews of the educational research literature have identified hundreds of efficacy studies. Yet the What Works Clearinghouse reports that very few of these analyses meet its selection criteria and standards of evidence. This report examines why these differences occur. It finds that the WWC procedures differ markedly from standard practices within the social sciences. The WWC gives no academic or scholarly justification for their policies. Moreover, an empirical, quantitative analysis of the utility of the WWC approach indicates that it provides no “value added” to estimates of a curriculum’s impact. The costs of applying the WWC standards are far from minimal and result in highly selective and potentially biased summaries of the literature. It is suggested that the public would be better served if the WWC adopted the standard methodological practices of the social sciences.