Technical Reports

Some Direct Instruction schools and classrooms are more successful than others. A major reason for these differences is implementation fidelity – the extent to which teachers and schools administer the programs as they were designed. Teachers and schools that conform most closely to DI implementation protocols have the most success.

This report uses data from different studies and settings to examine two general factors that make DI implementations more effective: 1) administrative decisions and practices and 2) experience with the program. The data show that DI students make significantly more progress at mastery and have significantly higher achievement when

• their teachers implement the programs with greater fidelity
• their teachers have been trained for the specific programs they are teaching
• their teachers are given time and support to prepare lessons
• they are taught for the recommended time each week
• those deemed at risk are given extra instructional time (“double dosing”)
• their teachers have more experience in teaching the programs
• DI has been implemented at their schools for a longer period of time
• they start learning with DI in kindergarten

This report examines the high school accomplishments and preparation for college of students with varying degrees of exposure to Direct Instruction in elementary school. Results indicate that students taught with Direct Instruction early in their school career were significantly more likely to be prepared to enter higher education – enrolling in a traditional academic program, finishing a college prep mathematics class, and  taking Advanced Placement courses and/or college entrance examinations. Among those enrolled in traditional academic programs, those with early exposure to Direct Instruction also ranked higher in their high school graduating class and were more likely to have a GPA high enough to qualify for college admission.

This report examines changes in mathematics and reading skills of students in Kment Elementary School in Roseville, Michigan after the Direct Instruction programs Connecting Math Concepts: Comprehensive Edition (CMCCE) and Reading Mastery: Signature Edition (RMSE) were adopted. The first section examines changes in mathematics scores and the second examines changes in reading scores. The data provide consistent and strong evidence that Kment students’ had improved skills in both math and reading after using these Direct Instruction programs.

Reviewing documentation related to 62 Quality Reviews of What Works Clearinghouse (WWC) publications, this report summarizes the reasons for the reviews, the revisions made and not made, and the inconsistent application of WWC policies during the publication of these reports and during the Quality Reviews. This information was obtained in the fall of 2013 via the Freedom of Information Act (FOIA). These Quality Reviews were performed in response to the concerns of 54 organizations, study authors, program developers, teachers, and education researchers.Forty-one different instructional programs and study reviews were examined in these Quality Reviews. This documentation revealed a wide range of concerns, particularly the misinterpretation of study findings. This issue was given specific attention, especially in relation to how the WWC accounts for the fidelity of implementation when determining their rating of effectiveness for specific programs. With the information provided from the FOIA request and the publicly available information three conclusions appear clear. 1) The WWC suffers from a lack of transparency in their policies and guidelines, 2) the conclusions they create in their reports can be misleading, and 3) the reports are potentially damaging to program developers and ultimately the success of students.

A November, 2013, report of the What Works Clearinghouse stated that it could find “no studies of Reading Mastery that fall within the scope of the Beginning Reading review protocol [and] meet What Works Clearinghouse (WWC) evidence standards” (WWC, 2013b, p. 1). This NIFDI technical report documents substantial errors in the WWC’s compilation of studies to examine and in the interpretations of individual studies. Effect sizes (Cohen’s d) were computed for results of more than three dozen studies identified by the WWC but rejected for analysis. All of these studies conformed to standard methodological criteria regarding valid research designs and would have been accepted in scholarly reviews of the literature. The average effect size associated with Reading Mastery (RM) was .57. This value is more than twice the .25 level traditionally used to denote educational significance. The results replicate meta-analyses that have found strong evidence of the efficacy of Reading Mastery. Given the high rate of error in this and other WWC reports consumers are advised to consult reviews of studies in the standard research literature rather than WWC summaries.

Meta-analyses and reviews of the educational research literature have identified hundreds of efficacy studies. Yet the What Works Clearinghouse reports that very few of these analyses meet its selection criteria and standards of evidence. This report examines why these differences occur. It finds that the WWC procedures differ markedly from standard practices within the social sciences. The WWC gives no academic or scholarly justification for their policies. Moreover, an empirical, quantitative analysis of the utility of the WWC approach indicates that it provides no “value added” to estimates of a curriculum’s impact. The costs of applying the WWC standards are far from minimal and result in highly selective and potentially biased summaries of the literature. It is suggested that the public would be better served if the WWC adopted the standard methodological practices of the social sciences.

Arthur Academies are a set of six charter elementary schools in the greater Portland, Oregon metropolitan area. All of the Academies use the Direct Instruction programs, Reading Mastery Signature Edition and Connecting Math Concepts. The Academies provide extensive teacher training and coaching to maintain high fidelity to the implementation guidelines of the programs. This report examines data from the Stanford Achievement Test (SAT) and Oregon Assessment of Knowledge and Skills (OAKS) on the reading and mathematics achievement of Arthur students from six consecutive school years: 2007-2008 through 2012-2013. The data indicate that at the start of kindergarten, Arthur students had achievement scores that were similar to or slightly lower than students in the nation as a whole. However, by the end of their kindergarten year, the situation had reversed and the average Arthur student scored much higher than peers in the nation. In all cases, the changes over time, relative to the national norms, were statistically significant. This high level of achievement persisted, and even increased, through later grades. In all cases, the percentage of students scoring at high levels was substantially higher than would be expected given national norms. 

Students’ performance on state developed and norm-referenced assessments is a major concern of educators. Schools may devote many hours to “test preparation,” hoping that this will increase students’ scores. Yet, policymakers have expressed concern over this practice, suggesting that it can result in invalid measures of student achievement, divert attention from crucial subject matter, and provide less than optimal models for students.

Data presented in this report suggest that a rational alternative to extensive test preparation is to ensure that students make adequate progress, at mastery, through Reading Mastery Signature Edition (RMSE) and Connecting Math Concepts: Comprehensive Edition (CMCCE). The data suggest that when students have attained mastery at their assigned grade level in these programs the probability that they will pass the state assessments is very high. The influence of cumulative mastery of the curriculum is independent of students’ prior achievement, providing hope for students who have struggled with low assessment scores in the past.

The What Works Clearinghouse (WWC) is a federally funded program established in 2002 that evaluates educational interventions and publishes reports and summary ratings. The reports have received extensive criticism, including concerns such as examining only a small proportion of the available evidence, errors in the review process, and a lack of peer review and comparisons of results to related literature.

Two WWC reports issued in July 2013 illustrate the severe problems that can permeate the process and result in the dissemination of erroneous conclusions. In one case, the WWC’s errors resulted in a positive rating for a program that has been determined, by more inclusive and careful reviews, to be ineffective and inefficient. In the other case the WWC’s errors resulted in a negative conclusion regarding a program that has been judged, by more inclusive and careful reviews, to be highly effective. This document describes these errors and preliminary steps needed to prevent their reoccurrence.

This report examines the relationship between schools’ decisions to maintain the use of Direct Instruction programs and the variables of training, coaching, and the fidelity of implementing Direct Instruction programs. In order to better understand this relationship the report examines the decisions of schools in the Guam Public School System (GPSS) on whether or not to maintain their Direct Instruction programs.

In 2003, 24 of the 25 elementary schools in the GPSS began to systematically implement Direct Instruction with the support of the National Institute for Direct Instruction. In the fall of 2008, 15 of these schools decided to stop using Direct Instruction. Results indicated that schools with more extensive support had higher levels of fidelity and schools with higher levels of fidelity were more likely to retain Direct Instruction.

The findings in this report support other literature on the critical role of technical assistance in promoting teachers’ skills and their fidelity of implementation as well as the importance of providing continuous and intensive support.

In the fall of 2003, 24 schools in the Guam Public School System (GPSS) systematically implemented Direct Instruction with accompanying training and support. Initially used with lower grades, it was expanded to incorporate the upper grades. In 2008, fifteen of these schools stopped using Direct Instruction, but two additional schools began using Direct Instruction at that time.

This article examines the changes in the average Stanford Achievement Test (SAT) scores in reading of first and fifth grade students in elementary schools in the GPSS from the Spring of 1993 through the Spring of 2011. The analysis of this data focuses on the difference in the average SAT scores in schools and years when Direct Instruction was implemented and when it was not using multiple base-line interrupted time series techniques.

Three sets of results are reported: 1) an analysis of variation in first grade achievement using all years of data, 2) an analysis of variation in fifth grade achievement using all years of data, and 3) an analysis of fifth grade achievement limited to cohorts of students for whom data on first grade achievement were also available.

Results showed there were significantly higher achievement scores during school years when students were exposed to Direct Instruction. The difference in scores between students exposed and not exposed to Direct Instruction was statistically significant and educationally important. Additionally data indicated a significantly smaller decline in average test scores from first to fifth grade results when Direct Instruction was implemented in the fifth grade.

In this report, Stockard analyzes the conclusions of the What Works Clearinghouse (WWC) on the effectiveness of Direct Instruction, specifically examining the strong differences between their review of Direct Instruction and the extant of scholarly literature, which indicates that Direct Instruction programs are highly effective.

The report initially examines the WWC review procedures for including and excluding articles for official review, comparing the procedures to those of the social sciences, while highlighting problems in their selection criteria. This section is followed by an examination of the ten studies on Direct Instruction the WWC decided met their inclusion criteria. She argues serious errors occurred in the classification of four of the approved studies. The following two sections of the report focus on analyzing the content of the 2012 WWC report on the effectiveness of the Direct Instruction program Reading Mastery with students with learning disabilities.

Stockard examines twenty research studies that could have been included in the WWC’s review, specifically focusing on the design and conclusions of the studies, the effect size associated with their results, and the potential reasons why the WWC would not include the studies in their review. She concludes that the WWC procedures appear to be responsible for a selective and inaccurate analysis of Direct Instruction literature. Furthermore she suggests consumers should consult sources of summary material other than the WWC, and specifically they should examine the well-conducted and highly regarded meta-analysis literature.

In response to a recent report published by the What Works Clearinghouse (WWC) on the effectiveness of the Direct Instruction program Reading Mastery with students with learning disabilities, Stockard and Wood analyzed the report and highlighted the serious errors in the WWC’s conclusions. 

The WWC examined two studies and concluded that Reading Mastery had “no discernible effects on reading comprehension and potentially negative effects on alphabetics, reading fluency, and writing.” Their conclusion is in stark contrast to dozens of studies of Reading Mastery and other elements of the Direct Instruction corpus of material, which consistently found strong positive effects of the programs on academic achievement for students of all ability levels.

Stockard and Wood thoroughly examined the WWC report and the research articles used to develop its findings and found very serious problems with its conclusions. These problems are documented and explained in detail. This report also discusses the WWC procedures for their reports and the issues that result from them. Additionally, an annotated list of relevant studies not reviewed by the WWC is included to demonstrate the weakness of the WWC conclusions and provide further evidence on the effectiveness of Direct Instruction and Reading Mastery.

This article discusses an experiment on the effectiveness of two reading programs: Reading Mastery (RM) Signature Edition and Harcourt Brace (HB). This experiment was conducted using students in grades K-2 at a Florida elementary school during the 2010-2011 school year. Students were randomly assigned to each program.

The data for assessing the changes in reading skills over time was collected by the school district using the Florida Assessments for Instruction in Reading (FAIR) and the Star Reading assessment (first and second graders only). Students assigned to RM began the school year with slightly lower scores than the students in HB. However by the end of the year, the RM students recorded significantly greater scores than the HB students.

The greater success with RM was represented across all groups that are commonly seen as being at greater risk for having difficulties with reading: special education students, racial-ethnic minorities, students receiving free or reduced lunch, and boys. This report is divided into three sections. The first section provides summary results for all students in the study. The second and third sections discuss in detail the measures used and the analyses while also separating the results of the kindergarten students from the first and second grade students. 

In response to an increased focus by the field of education on the use of evidence-based practices in schools, this article analyzes research on Direct Instruction (DI) programs and practices to help determine its classification as an evidence-based practice. Coughlin reviewed multiple research syntheses, systematic reviews, meta-analyses, and best evidence syntheses on the effectiveness of DI. Her selection provides strong support for the effectiveness of DI with a wide range of students, in different settings, and within multiple subject areas. The various approaches to determining program effectiveness and the conclusions of these analytical approaches are discussed in detail. Coughlin concludes that there has been remarkable consistency in the conclusions drawn about the effectiveness of DI, but additional research is needed to reinforce the already established strong research base and to explore the effectiveness of DI with different populations, conditions, and subject areas.

The purpose of this study was to examine growth in reading comprehension and fluency of fourth grade general education students receiving instruction in Reading Mastery, Signature Edition or Scott Foresman’s Celebrate Reading program.

High achieving students in a Midwestern elementary school were randomly assigned to one of the programs and received instruction for ninety minutes a day. Each group had two teachers (four teachers in all), who were randomly assigned to teach the different groups. The curriculum-based AIMSweb measurement system was used to gather fluency and comprehension data.

While there were no significant differences between the groups at baseline, the RM group had significantly stronger growth in both fluency and comprehension skills over the school year. Effect sizes associated with the differences at spring were .75 for fluency and .41 for comprehension, both higher than the level usually used to denote educational significance.

This report examines changes in reading skills of elementary students in three rural Midwestern school districts after the implementation of Reading Mastery (RM) with support from the National Institute for Direct Instruction (NIFDI). Data from two sets of student cohorts: 1) cohorts that began school before RM was implemented and did not start the curriculum until first grade or later (partial exposure cohorts), and 2) cohorts that had RM beginning in kindergarten with teachers fully complying with the NIFDI model (full exposure cohorts). By the middle of kindergarten, those in the full exposure cohorts had significantly higher DIBELS scores than students in the partial exposure cohort and scores that were equal to or significantly higher than those in the national sample. Differences remained strong and significant through the primary years, and effect sizes surpassed usual standards of educational significance. Data on fourth graders from one district showed statistically significant and educationally important increases in scores on the state reading assessment after exposure to RM.

This report describes the use of Direct Instruction in a Liberian elementary school that serves children from very impoverished backgrounds, with limited supplies and infrastructural support.

The school began using Direct Instruction in the fall of 2004 with training from U.S. based educational consultants and supplies donated by schools in the U.S. In fall, 2009, a random sample of 43 second to sixth graders were administered DIBELS Oral Reading Fluency (ORF) probes. As a comparison group, 19 students in two nearby schools with similar populations, but which had not used Direct Instruction, were also tested.

These data were compared to DIBELS scores obtained from students in three Midwestern elementary schools. The students in the Liberian DI school had markedly higher ORF scores than the students in the other Liberian schools, with differences that were more than a standard deviation in magnitude at each grade level. In comparison to students in the United States, the Liberian DI students had lower ORF scores in second through fourth grade, but ORF scores within one-tenth of a standard deviation of their American counterparts at the upper grades.

When the U.S. data were disaggregated across school districts comparisons indicated that the average Direct Instruction Liberian fifth grader had higher ORF scores than the average fifth grader in two of the Midwest communities. The average Liberian DI sixth grader had a higher score than sixth graders in one of the communities and came close to the average score in another community.

Comparison of DIBELS scores with established U.S. benchmarks indicated that about one-third of the Liberian DI students would be termed “at risk” of having future academic difficulties, while all but one of the students in the comparison schools fell in that category. None of the comparison Liberian students were deemed “low risk,” while a third of the Liberty students were in this group.

These differences were highly significant. There was markedly less variability in DIBELS scores in the Liberian DI school than in the U.S. schools. Results are reported to replicate those found two decades earlier in South Africa, another English speaking African country (Grossen and Kelly, 1992).

Thirty-seven four-year-old students in a Head Start program in a suburban community in the southern United States were randomly assigned to receive 30 minutes of additional instruction each day in their usual language arts curriculum or 30 minutes of instruction with the Direct Instruction program, Funnix Beginning Reading. Instruction for the Funnix group was provided by high school-aged tutors, who were trained and supervised by an experienced teacher.

Pretesting before instruction began indicated no significant differences between the two groups in the DIBELS letter naming fluency or initial sound fluency measures. However, by winter and spring the students in the Funnix group had significantly higher scores on numerous measures of beginning literacy. Significant differences remained with multivariate analyses that controlled for initial levels of skill and minority status. Equivalent results also appeared when a reduced sample that individually matched children on pretest scores and minority status was used. Effect sizes (Cohen’s d) in winter and spring ranged from .51 to 2.24, depending on the measure that was used.

The National Reading Panel recently concluded that pre-literacy and early literacy instruction is appropriate for kindergarten students and an important element of promoting higher achievement in later grades.

This paper examines the relationship of receiving the Direct Instruction (DI) kindergarten curriculum, Reading Mastery, on students’ oral reading fluency in first and second grade. Data from several hundred students in two different schools are analyzed. Achievement was measured using the Dynamic Indicators of Basic Skills (DIBELS).

Results indicate that the students who received Direct Instruction kindergarten had significantly higher achievement in early elementary school. These results replicate those found in other studies, providing consistent evidence of the effectiveness of Direct Instruction kindergarten instruction in promoting later academic achievement.

This study compared gains in achievement from kindergarten through third grade of students in two schools in the same district. One school used Direct Instruction’s Reading Mastery for both General and Special Education students, while the other used a basal reader and occasionally used Reading Mastery with special education students. DIBELS measures were collected at regular intervals. There were no significant differences in scores at the start of kindergarten, but students in the DI school had significantly higher oral reading scores from the middle of first grade through the end of third grade. Differences were stronger for special education students than for general education students.

This report evaluates the procedures of the What Works Clearinghouse (WWC), and specifically examines a WWC report on the effectiveness of the Direct Instruction program Reading Mastery. Stockard’s evaluation is divided into three sections; a criticism of the WWC procedures, a detailed analysis of the WWC’s report on Reading Mastery, and general concerns about the WWC and the misrepresentation of research.

Stockard examined the WWC report on Reading Mastery in terms of their conclusions and the WWC procedures for conducting their reviews. The WWC report determined that no studies met their evidence standards for review and therefore they could not make any conclusion based on research about the effectiveness of Reading Mastery. However, Stockard determined the WWC failed to examine almost 100 research studies that were cited in other well-known reviews of the literature. Many of these studies demonstrated the effectiveness of Direct Instruction and Reading Mastery with a variety of students in different settings.

Apart from the absence of these relevant studies Stockard argues that reviews of some of the studies included were flawed. In conclusion Stockard provides recommendations for changes in WWC procedures.

In response to very low achievement scores, the Baltimore City Public School System (BCPSS) implemented curricular reforms in the late 1990’s. Under the guidance of the National Institute for Direct Instruction the district implemented Direct Instruction in 11 schools as part of a whole school reform.

Initially reading curriculum was implemented and then within the next two years the Direct Instruction mathematics programs DISTAR Arithmetic and Connecting Math Concepts was added. All first and fifth grade students in BCPSS were administered the Comprehensive Test of Basic Skills (CTBS) beginning in the spring of the 1997-1998 school year and concluding in the 2002-2003 school year. Data was retrieved from over 40,000 first grade students on mathematics achievement in computations, concepts and applications, and a composite score.

This study compares students’ mathematics achievement in schools that implemented Direct Instruction (DI schools), and schools without Direct Instruction (Control schools). Results showed that first grade students in schools with Direct Instruction had significantly higher mathematics achievement than students in the control schools. The positive effects of Direct Instruction increased over time as the curricular changes became institutionalized within the schools.

First grade students in both schools recorded higher levels of achievement in 2003 than 1998, but the increase was significantly larger for students in the NIFDI-supported Direct Instruction schools. On average the first grade composite mathematics achievement scores in the Direct Instruction schools increased by 135 percent from 1998 to 2003, but by only 54 percent in the control schools.

This report examines the achievement growth of two cohorts of students who were enrolled in the Baltimore City Public School System in both first grade and fifth grade (in first grade in 1997-98 and fifth grade in 2001-2003 or in first grade in 1998-99 and fifth grade in 2002-03) were included in the study. Students attended 103 different schools. Eleven of these schools implemented Direct Instruction with support from the National Institute for Direct Instruction (NIFDI) and 5 used DI, but with support from another organization. Sample size was approximately 5,000. Schools in the three groups had similar demographic characteristics, although one of the Other DI schools had a markedly lower rate of poverty and a substantially lower rate of student turnover. The dependent variables were comprehension, vocabulary and composite scores on the Comprehensive Test of Basic Skills (CTBS). Results indicated that students who received Direct Instruction in first grade had significantly higher achievement scores in fifth grade than other students. The results were strongest with the measures of vocabulary and the composite score. While students in all schools had higher average achievement in fifth grade than in first grade, the increases were strongest for students in the NIFDI-supported schools and in the low-poverty Other DI school. On average, students in NIFDI supported schools had a 25 percent gain in their composite reading achievement scores from first grade to fifth grade. Students in the control schools had a gain of 5 percent. The magnitude of changes from first to fifth grade far surpassed the traditional criterion of educationally important effects for both the NIFDI-supported school and the low poverty school in the Other DI group. Mixed models were used with controls for school level socio-demographic characteristics.

Responding to very low reading achievement scores, the Baltimore City Public School System (BCPSS) began implementing curricular reforms in the mid-1990s. Direct Instruction was implemented in 16 schools, specifically using the curricular programs; Reading Mastery Classic, Language for Learning, Language for Thinking, and Reasoning and Writing.

Throughout the study 11 of these schools received technical support from the National Institute for Direct Instruction (NIFDI). The remaining five schools discontinued support from NIFDI or used an alternative provider. Beginning in the spring of the 1997-1998 school year through the 2002-2003 school year the Comprehensive Test of Basic Skills (CTBS) was administered to all first graders. Data was retrieved from over 40,000 students on reading vocabulary, comprehension, and a composite score.

This study compared students’ achievement in 1) schools with NIFDI support, 2) schools that implemented Direct Instruction without NIFDI support (other DI schools), and 3) schools without Direct Instruction (the Control schools). Results showed that students in the NIFDI-supported schools had significantly higher achievement than students in the control schools on all three measures of achievement and significantly higher comprehension scores than students in the Other DI schools. Additionally the achievement scores of first grade students in all 3 groups increased from 1998 to 2003. The average first grade composite reading achievement scores for the students in the NIFDI-supported schools increased by 113% during that time period while student scores in the control schools and the Other DI schools increased by 56% or less.