Author Interview: Peter John

Peter John offers some insight on his latest work:
Field Experiments in Political Science and Public Policy

1. What experience(s) led you to write this book?
A few years ago I was invited to give a short, three-day course on field experiments at the University of Essex. I prepped for it assiduously so as to cover all aspects of experiments. When I turned up to teach the course, I found the students were less interested in hearing me explain the use of statistics to analyse experiments, but they wanted me to give practical tips on how to carry them out. One economics PhD student said to me, ‘I know all the statistics, but I am going to Africa soon and I have no idea about how to actually do an experiment – help me please’. The reaction of the students influenced my thinking about how to write about experiments. The use of field experiments in political science and public policy is a relatively new development, which is linked to the stress scholars now place on causal inference, that is in establishing authoritatively the causes of many phenomena, in particular political and policy interventions. The claim for causal inference is a great advance, but many researchers do not talk about the other distinguishing feature of field experiments, which is the need to deliver a practical intervention alongside the measurement and analysis, which is a different task and where researchers are often very dependent on outside agencies to implement the experiment on their behalf. However, not many researchers talk about how to work with agencies and to ensure the experiment goes to plan so they can make a causal inference they can trust. I saw it as my job is to explain more about the practical side of experiments because the real world imposes all sorts of constrains on what is indeed possible to do. I wanted to shine a light on the world of practical design and the experience of implementation which people do not usually write about.

2. Are there any key messages or themes that you’d like to highlight for potential readers?
The key message is that effective planning should be sensitive to the interlocking nature of experimental design, so that if one aspect changes, so does another. For that reason, I set out ten steps of experimental design for researchers to follow as they think through the linkages. It is important to note that the design of the experiment needs to be sensitive to what may happen in the world once the experiment starts up. Contingencies need to be thought about, and so I have set out nine threats to experiments in chapter three for researchers to watch out for. The assumption is that all sort of events happen during an experiment and the researcher needs a design that anticipates a worst-case scenario.

3. How is this book different from other books in the field?
Most books on experiments focus on statistics, which is essential, so such a book should be the main course book for any module. My book is about the practical side and should be read alongside, not instead of, other texts. I would recommend Alan Gerber and Donald Green’s book, Field Experiments: Design, Analysis, and Interpretation as the core reading, with my book as the second text to be consulted.
4. Did any findings in writing/researching the book surprise you?
What was interesting was finding out about the differences between different kinds of experiment in terms of the amount of cooperation needed from partners and the difference between large-scale policy experiments and more small-scale political science experiments. Although some experiments need a large amount of funding and a long-term relationship with a range of partners, it was possible to build partnerships for small-scale experiments. Lone researchers using their ingenuity and ability to cultivate good personal relationships can get really great experiments off the ground. This opportunity suits PhD or research students who must usually work on a smaller project. The message is that anyone can do an experiment, and first-time researchers have an advantage in being able to work with a smaller organization and are able to find like-minded people to work with in crafting the intervention.

5. Can you tell us about the instructor materials/pedagogical tools available with this book, and why students will find the book so useful in a Research Methods course?
The core of the book is chapters two and three, which walk the reader respectively through the ten steps needed in an experimental design and the nine threats that can come up. These steps, and an awareness of the threats, can be used for the students to design their own experiment and can be deployed in teaching as a class exercise. The students can find examples of how researchers design experiments when they read the substantive chapters, four to eight, on fields within political science, such as experiments with elites, and fields within public policy, such as behavioral or nudge experiments.

6. What additional research do scholars in the field need to be reading?
There is a huge amount of research coming down the line as scholars get more interested in experiment and PhD students become fully-fledged researchers. Readers can look at the contents page of the leading journals in political science to find examples of experiments, such as the American Journal of Political Science, or the specialist journal, Journal of Experimental Political Science. There is a similar story with public policy and public administration, with experiments often reported in Journal of Policy Analysis and Management and in the Journal of Public Administration Research and Theory.

Peter John is Professor of Political Science and Public Policy in the Department of Political Science, University College London. He is an expert in the study of public policy, and has pioneered the use of experiments in the study of civic participation in the UK.

Featured Title