Blind Analysis
When we measure a value that is consistent with the prediction our tendency is to trust the result. When we get an unexpected answer we apply greater scrutiny, trying to determine if we’ve made an error before believing the surprise. This tilts us toward revealing errors in one category of measurements and leaving them unexposed in another. It is particularly dangerous when our assumptions are flawed—and if there is anything we should bet on, given the history of progress within physics, it’s that our underlying assumptions are in some way flawed. Bias can creep into the scientific process in predictable and unpredictable ways.
Blind analyses are employed as a protection against bias. The idea is to fully establish procedures for a measurement before we look at the data so we can’t be swayed by intermediate results. They require rigorous tests along the way to convince ourselves that the procedures we develop are robust and that we understand our equipment and techniques. We can’t “unsee” the data once we’ve taken a look.
There are options when it comes to performing a blind analysis. If you are measuring a particular number, you can apply a random offset to the number that is stored but not revealed to the analyzers. You complete the full analysis and reveal the offset and true result only when the work is done. Another method is to designate a sensitive segment of the data, the “signal,” as off limits. You don’t look at the signal until you’re convinced that you understand the remaining data, the “background.” You can fully develop your analysis using the background and a simulated fake signal. Only when the analysis is fully developed do you look at the signal and obtain the result, a process known as “opening the box.” Another flavor of blind analysis was employed by the LIGO experiment in the discovery of gravitational waves. Fake signals were periodically inserted into the data so that full analyses were undertaken without analyzers knowing if the signals they were seeing were real. They carried the analyses all the way to the point of preparing the corresponding publication before they learned if they were analyzing real or fake data.
Blind analyses force scientists to approach their work with humility, acknowledging the potential for bias to influence the process. They require creativity and rigor as we establish an understanding of the data without direct access to it. They enforce good stewardship of the data, which can represent significant investments into experiments that are not easily repeatable. They highlight the mystery and anticipation inherent in the discovery process where opening the box has the potential to reveal a surprise. Humility, rigor, stewardship, and mystery are the essential ingredients of blind analyses and represent the best that science has to offer.
