Education Research On Trial: Policy Reform and the Call for Scientific

Description

Read the author's commentary for the Teachers College Record here: http://www.tcrecord.org/Content.asp?ContentID=15915 It is not an exaggeration to say that the field of education has been under attack. Many, particularly in Washington, D.C., have proclaimed the research to be shoddy. They have called for new "scientific" standards for research. Randomized control trials have been promoted. In... Read more

The first set of reprinted documents below traces the development of the charges that education research is low quality and of limited usefulness. The second set highlights the key elements of the federal reforms to improve education research. The final set provides examples of some of the major responses from the education research community.

The Problem

Carl F. Kaestle, "The Awful Reputation of Education Research." . . . . . . . . . . . . . . . . 23

Grover J. (Russ) Whitehurst, "New Wine, New Bottles." Address at the

annual meeting of the American Educational Research Association. . . . . . . . . . . . . . 30

Selection from Request for Proposals for Predoctoral Interdisciplinary

Research Training Programs in the Education Sciences, Issued by the

Institute of Education Sciences in 2004 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Remedies for Improvement

The Definition of "Scientifically Based Research" in the No Child Left

Behind Act of 2001 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

The Definitions of Scientific Validity in the Education Sciences

Reform Act of 2002 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Mission and Functions of the Institute of Education Sciences, as Detailed

in Education Sciences Reform Act of 2002 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

The What Works Clearinghouse Standards for Evaluating Existing

Research Studies of Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Reactions from the Education Research Community . . . . . . . . . . . . . . . . . . . . . . 53

A selection from the 2002 report from the National Research Council,

Scientific Research in Education. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Margaret Eisenhart and Lisa Towne, "Contestation and Change in

National Policy on ‘Scientifically Based’ Education Research." . . . . . . . . . . . . . . . . 64

The American Educational Research Association’s "Standards for

Reporting on Empirical Social Science Research in AERA Publications." . . . . . . . . 72

Part II: The Politics of Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

The Politics of Science: Battles for Scientific Authority in the Field of

Education Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

Pamela Barnhouse Walters

Walters argues that the current debates about the quality of education research and the best ways to improve it do not turn only on issues of the scientific merits of competing positions. The debates are part of political and social struggles between groups of scientific experts, and between policymakers and scientific experts, over who gets to decide what counts as science and to claim scientific legitimacy within the research field.

A History of Efforts to Improve the Quality of Federal Education

Research: From Gardner’s Task Force to the Institute of Education

Sciences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Maris Vinovskis

Vinovskis shows that the current critique of the quality of education research is related in important ways to recurring dissatisfaction on the part of federal lawmakers and bureaucrats with the decisions and priorities of the federal agencies that provide the bulk of federal funding for education research.

Part III: Seeking Rigor; Finding Rigor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Assessing Quality in Educational Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Barbara Schneider

Schneider addresses the question of whether the quality of education research is as bad as its critics charge by comparing the scientific standards and processes in place at major education journals with the standards and processes in place in journals in other fields generally considered to be more scientific. She finds the education journals to be comparably rigorous.

Can Non-Randomized Studies Provide Evidence of Causal Effects?

A Case Study Using the Regression Discontinuity Design . . . . . . . . . . . . . . . . . . 228

Larry V. Hedges and Jennifer Hanis

While in sympathy with the call to make education research more rigorous, Hedges and Hanis show that randomized controlled trials are not the only way to rigorously assess causal relationships about education. They illustrate the usefulness of regression discontinuity models for assessing causality in conditions in which random assignment is not possible.

Blending Quality and Utility: Lessons Learned From the Quality Debates . . . . 260

Sheri Ranis

Ranis shows that the debates about the quality of education research have been propelled by and conflated with debates about the utility of education research in ways often unacknowledged. She demonstrates that research utility became a "resonant problematic" that provided a powerful justification for the movement to improve the quality of education research.

Part IV: Toward a More Comprehensive Understanding of Science . . . . . . . . . . . . . . 286

Narrow Questions, Narrow Answers: The Need to Broaden the

Methodological Scope of Education Research . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Annette Lareau

Lareau argues that the education sciences movement has misapplied the medical model to education research. She suggests there is a need for more attention to a broader array of questions about meaning, process, and interactional dynamics and greater attention to issues of implementation.

A Quixotic Quest? Philosophical Issues in Assessing the Quality of

Education Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Denis C. Phillips

Philips demonstrates that the current drive to establish a single model of scientific research in education takes an overly-simplistic view of the nature of "science," in the process ignoring the complexities inherent in studying the intrinsically social and cultural dynamics of schooling. He calls the search for a single model of scientific research a "quixotic quest."