Wrapping up the neutrino saga with musings on the scientific value of big mistakes

Back in September, I wrote a post on the faster than light neutrino discovery:

Neutrino physics tells the story of a number and science as a whole

Now, six months later, I think we are close to the closing chapter on this particular story. For the full story, I point you to this excellent summary written by theoretical physicist Matt Strassler:

OPERA: What Went Wrong

So no, this experiment didn’t “prove Einstein wrong”, and reporters with a real understanding of science avoid “It’s only a theory” and “prove scientist X wrong” framing anyway. Instead, the initial finding that neutrinos travel faster than light did not hold up to the careful scrutiny and investigation that follows such a fantastic claim.

But, as Strassler says “The process of finding a mistake is itself often an interesting and instructive story.”

Taken most glibly, the problem was reported as:

But I’d like to go deeper, and see how headlines like this miss the real story and the lessons that can be learned from it.

Strassler’s entire account is worth reading, but here are some highlights I’d like to point out.

• In the verification/error checking process, it took scientists quite some time to come to checking the low level equipment errors that might have caused an anomalous measurement. Stressler posits that it might be due to overconfidence.
• When the scientists found the first hints of the error in an anomalous timing measurement in December, it was the first time they had repeated this particular measurement since 2007.
• Armed with this finding, scientists tracked down the an optical fiber that wasn’t fully screwed into an input. This is more than just a loose wire, or a bad cable.

The problematic fiber interconnect. (From http://profmattstrassler.com/articles-and-posts/particle-physics-basics/neutrinos/neutrinos-faster-than-light/opera-what-went-wrong/)

• Just as scientists discovered this error, they discovered another possible timing error, that might lead to the neutrino signal drifting in the other direction.
• As the investigation progressed, scientists found that this loose fiber causes a small delay in master clock timing pulse by tens of nanoseconds. Further investigation of the delay issue found that it caused neutrinos to appear to late, but not enough to cancel out the error caused by the loose cable.
• Now that scientists had a working hypothesis, they set out to test it. To do this they need to know exactly how the cable was mis-connected in the first place. Scientists were, astonishigly, able to find a photo from October showing the mis-connected cable. Still, this one photo wasn’t enough to tell scientists exactly when the fiber first became unscrewed, so they needed to find another method to test their unscrewed cable hypothesis.
• The second way scientists tested their loose cable hypothesis was by using data from a nearby experiment, LVD, located 300 meters away from the Opera experiment. It so happens that cosmic muons are bombarding us all the time, traveling at near the speed of light, and every so ofen (about 300 times in the past five years), a muon will be oriented in just the right way to pass through both the LVD and OPERA experiment, allowing for a very precise measurement of the time it takes for the muon to pass between the two detectors.

• When this and a few more checks were performed, OPERA scientists came away with a measurement of , essentially a measurement of zero.

Stressler fleshes out more of the details and provides some very insightful commentary on the handling of these errors worth reading.

Thoughts on the role of mistakes in the scientific process

Here are a few reflections I have on this story and the role of mistakes in the scientific process.

• It is hard to question or most basic assumptions. The scientists took months before they checked some of the most basic elements of their experiment, like checking that all of the cables were completely screwed in.
• Mistakes expand our understanding when we explore them deeply. When scientists did finally find their mistake, they were able to develop a testable hypothesis to rigorously verify the cause, and learn much more about their experiment in the process.
• Scientists will go to extraordinary lengths to track down their mistakes. It wasn’t enough to find a photo from 2008 showing an loose cable, scientists required verifiable data, and were willing to devote the time resources to find it.
• Bad consequences come not from the mistakes themselves, but from how we handle them. Thoughout this process, the OPERA team seemed to take steps to overplay their discovery, minimize their uncertainties, and overlook many potential errors. Utlimately, it was these mistakes in handeling the errors, rather than the errors themselves that had dramatic consequences, like the resignation of OPERA’s leader, Antonio Ereditato.

Overall, this is a vivid and edifying story of the scientific process, and the central role that mistakes play in that process. It also leaves me wondering how we can encourage our students to do the kind of deep and rigorous exploration of mistakes that lead to real learning, and handle their mistakes with the openness and forthrightness that allows for learning, without the nasty consequences that come from ignoring and minimizing mistakes.