Freedom of Information Request (FOIA) - National Institute for Direct Instruction (NIFDI)

2014-006

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:54:49 -0800

2014-005

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:54:28 -0800

2014-004

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:54:05 -0800

2014-003

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:53:42 -0800

2014-003 (Documents from related issues)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:53:19 -0800

2014-002

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:52:51 -0800

2014-001

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:52:25 -0800

2013-003

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:52:02 -0800

2012-001(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:51:40 -0800

2010-003(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:51:17 -0800

2009-004(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:50:53 -0800

2009-001(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:50:18 -0800

2008-012(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:49:55 -0800

2008-006(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:49:32 -0800

2007-003(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:49:07 -0800

2007-001(pt. 2)

ccox@nifdi.org (Christina Cox) — Fri, 30 Jan 2015 15:47:33 -0800

2014-5: Examining the Inaccuracies and Mystifying Policies and Standards of the What Works Clearinghouse: Findings from a FOIA Request

ccox@nifdi.org (Christina Cox) — Fri, 17 Oct 2014 13:14:17 -0700

Reviewing documentation related to 62 Quality Reviews of What Works Clearinghouse (WWC) publications, this report summarizes the reasons for the reviews, the revisions made and not made, and the inconsistent application of WWC policies during the publication of these reports and during the Quality Reviews. This information was obtained in the fall of 2013 via the Freedom of Information Act (FOIA). These Quality Reviews were performed in response to the concerns of 54 organizations, study authors, program developers, teachers, and education researchers.Forty-one different instructional programs and study reviews were examined in these Quality Reviews. This documentation revealed a wide range of concerns, particularly the misinterpretation of study findings. This issue was given specific attention, especially in relation to how the WWC accounts for the fidelity of implementation when determining their rating of effectiveness for specific programs. With the information provided from the FOIA request and the publicly available information three conclusions appear clear. 1) The WWC suffers from a lack of transparency in their policies and guidelines, 2) the conclusions they create in their reports can be misleading, and 3) the reports are potentially damaging to program developers and ultimately the success of students.

2014-4: Reading Mastery for Beginning Readers: An Analysis of Errors in a What Works Clearinghouse Report

ccox@nifdi.org (Christina Cox) — Fri, 17 Oct 2014 13:12:56 -0700

A November, 2013, report of the What Works Clearinghouse stated that it could find “no studies of Reading Mastery that fall within the scope of the Beginning Reading review protocol [and] meet What Works Clearinghouse (WWC) evidence standards” (WWC, 2013b, p. 1). This NIFDI technical report documents substantial errors in the WWC’s compilation of studies to examine and in the interpretations of individual studies. Effect sizes (Cohen’s d) were computed for results of more than three dozen studies identified by the WWC but rejected for analysis. All of these studies conformed to standard methodological criteria regarding valid research designs and would have been accepted in scholarly reviews of the literature. The average effect size associated with Reading Mastery (RM) was .57. This value is more than twice the .25 level traditionally used to denote educational significance. The results replicate meta-analyses that have found strong evidence of the efficacy of Reading Mastery. Given the high rate of error in this and other WWC reports consumers are advised to consult reviews of studies in the standard research literature rather than WWC summaries.

2014-3: What is a Valid Scientific Study? An Analysis of Selection Criteria Used by the What Works Clearinghouse

ccox@nifdi.org (Christina Cox) — Fri, 17 Oct 2014 13:10:37 -0700

Meta-analyses and reviews of the educational research literature have identified hundreds of efficacy studies. Yet the What Works Clearinghouse reports that very few of these analyses meet its selection criteria and standards of evidence. This report examines why these differences occur. It finds that the WWC procedures differ markedly from standard practices within the social sciences. The WWC gives no academic or scholarly justification for their policies. Moreover, an empirical, quantitative analysis of the utility of the WWC approach indicates that it provides no “value added” to estimates of a curriculum’s impact. The costs of applying the WWC standards are far from minimal and result in highly selective and potentially biased summaries of the literature. It is suggested that the public would be better served if the WWC adopted the standard methodological practices of the social sciences.

2013-002

ccox@nifdi.org (Christina Cox) — Tue, 07 Oct 2014 11:13:02 -0700

This Quality Review addressed concerns about the WWC's review of the instructional program Read Naturally. The inquirer was concerned about the misinterpretation of study findings. The WWC determined it was necessary to revise the report to clarify the description of Read Naturally.