What does it mean when experiments fail?

Description

What does it mean when experiments fail? Attending to positionality in the “replication crisis”

The idea of replication—that one scientist should be able to get the same results as another if they follow the same methods—is central to the scientific method. While replication is straightforward in theory, it is complicated in practice because an unexpected result can have many sources: The second scientist may not have as much skill in the technique as the first, the second experiment may be different in some small but important way, a reagent might have an impurity, or the original results may have been incorrect.

This uncertainty is especially deep in cases where the experiments are performed by students, who are by definition still learning to conduct experiments. Drawing on a set of interviews with 37 graduate students about their experiences of failing to replicate a result, we show how students tend to resolve this uncertainty by blaming themselves rather than questioning prior findings. These attributions of error are shaped by their identities as students but intersect with other identities, and we explore how race, class, and gender also appear in students’ narratives.

This default towards dismissal has important implications for understanding knowledge production in the biomedical sciences, where students generate most experimental results in many fields. Recent reports of low rates of replication in areas such as cancer biology have taken many by surprise, but the interpretive framework we describe here helps explain how high rates of irreproducibility may have been hiding in plain sight for some time, disguised as student error. This framework also helps explain the historical specificities of the emergence of the “reproducibility crisis,” which was catalyzed by reports of failures to replicate from industry scientists rather than academic laboratories.

 

Keynote:

Nicole Nelson is Associate Professor in the Department of Medical History and Bioethics, University of Wisconsin–Madison. Her research examines scientists’ assumptions about the natural world and how these assumptions shape scientific practice. In her book, Model Behavior, she explores how animal behavior geneticists’ beliefs about the complexity of gene action and of psychiatric disorders are reflected in their research with mouse models. Her current project focuses on the “reproducibility crisis,” a recent phenomenon where scientists have discovered that many established findings are difficult or impossible to replicate on subsequent investigation.

 

Organizers:

Ignacia Arteaga, Debangana Bose, Katarzyna Cieslik, Juan Del Nido