Software faults raise questions about the validity of brain studies – Ars Technica

It’s not an exaggeration to say that functional MRI has revolutionized the field of neuroscience. Neuroscientists use MRI machines to pick up changes in blood flow that occur when different areas of the brain become more or less active. This allows them to noninvasively figure out which areas of the brain get used when performing different tasks, from playing economic games to reading words.

But the approach and its users have had their share of critics, including some who worry about over-hyped claims about our ability to read minds. Others point out that improper analysis of fMRI data can produce misleading results, such as finding areas of brain activity in a dead salmon. While that was the result of poor statistical techniques, a new study in PNAS suggests that the problem runs significantly deeper, with some of the basic algorithms involved in fMRI analysis producing false positive “signals” with an alarming frequency.

Frankly, I’m not surprised. Scientists liked this stuff because it looks sexy. Journalists love it because it gives them pretty pictures to include in articles that over-glorify studies that, at best, showed a possible correlation in thought and brain areas.

So 40,000 papers have been published without the scientists ever really ensuring that their method of measurement was valid…

Here’s the abstract of the study that found these errors:

Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.

That’s pretty damning stuff. An entire scientific field is built on incorrect data.

Source: Software faults raise questions about the validity of brain studies – Ars Technica

2 thoughts on “Software faults raise questions about the validity of brain studies – Ars Technica

  1. My father was a neurologist for many years before retiring, and still keeps up on research. The one thing he always stressed to me about it was how much we don’t know about how the brain works – and how many clueless charlatans there are who think they know much more than we do.

    Science is very difficult specifically because it requires humility. Every new discovery shows us that we have much more to learn, and new data can shatter our worldview at an instant. The moment you think that your current state of knowledge is incontrovertibly true, you stop being a scientist.

  2. My father was a neurologist for many years before retiring, and still keeps up on research. The one thing he always stressed to me about it was how much we don’t know about how the brain works – and how many clueless charlatans there are who think they know much more than we do.

    Science is very difficult specifically because it requires humility. Every new discovery shows us that we have much more to learn, and new data can shatter our worldview at an instant. The moment you think that your current state of knowledge is incontrovertibly true, you stop being a scientist.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.