Replicability
Since different visiting teachers had promoted contradictory philosophies, the villagers asked the Buddha whom they should believe. The Buddha advised: “When you know for yourselves ... these things, when performed and undertaken, conduce to well-being and happiness — then live and act accordingly.” Such empirical advice might sound surprising coming from a religious leader, but not from a scientist.
“See for yourself” is an unspoken credo of science. It is not enough to run an experiment and report the findings. Others who repeat that experiment must find the same thing. Repeatable experiments are called “replicable.” Although scientists implicitly respect replicability, they do not typically explicitly reward it.
To some extent, ignoring replicability comes naturally. Human nervous systems are designed to respond to rapid changes, ranging from subtle visual flickers to pounding rushes of ecstasy. Fixating on fast change makes adaptive sense — why spend limited energy on opportunities or threats that have already passed? But in the face of slowly growing problems, “change fixation” can prove disastrous (think of lobsters in the cooking pot or people under greenhouse gases).
Cultures can also promote change fixation. In science, some high profile journals and even entire fields emphasize novelty, consigning replications to the dustbin of the unremarkable and unpublishable. More formally, scientists are often judged based on their work’s novelty rather than replicability. The increasingly popular “h-index” quantifies impact by assigning a number (h) which indicates that an investigator has published h papers that have been cited h or more times (so, Joe Blow has an h-index of 5 if he has published 5 papers, each of which others have cited 5 or more times). While impact factors correlate with eminence in some fields (e.g., physics), problems can arise. For instance, Doctor Blow might boost his impact factor by publishing controversial (thus, cited) but unreplicable findings.
Why not construct a replicability (or “r”) index to complement impact factors? As with h, r could indicate that a scientist has originally documented r separate effects that independently replicate r or more times (so, Susie Sharp has an r-index of 5 if she has published 5 independent effects, each of which others have replicated 5 or more times). Replication indices would necessarily be lower than citation indices, since effects have to first be published before they can be replicated, but might provide distinct information about research quality. As with citation indices, replication indices might even apply to journals and fields, providing a measure that can combat biases against publishing and publicizing replications.
A replicability index might prove even more useful to nonscientists. Most investigators who have spent significant time in the salt mines of the laboratory already intuit that most ideas don’t pan out, and those that do sometimes result from chance or charitable interpretations. Conversely, they also recognize that replicability means they’re really on to something. Not so for the general public, who instead encounter scientific advances one cataclysmic media-filtered study at a time. As a result, laypeople and journalists are repeatedly surprised to find the latest counterintuitive finding overturned by new results. Measures of replicability could help channel attention towards cumulative contributions. Along these lines, it is interesting to consider applying replicability criteria to public policy interventions designed to improve health, enhance education, or curb violence. Individuals might even benefit from using replicability criteria to optimize their personal habits (e.g., more effectively dieting, exercising, working, etc.).
Replication should be celebrated rather than denigrated. Often taken for granted, replicability may be the exception rather than the rule. As running water resolves rock from mud, so can replicability highlight the most reliable findings, investigators, journals, and even fields. More broadly, replicability may provide an indispensable tool for evaluating both personal and public policies. As suggested in the Kalama Sutta, replicability might even help us decide whom to believe.
