The Problem with Selective Selection

tumblr_mghj3tlmYY1qbpwkro30_r2_250tumblr_mghj3tlmYY1qbpwkro31_r1_250

tumblr_mghj3tlmYY1qbpwkro8_r1_250tumblr_mghj3tlmYY1qbpwkro11_r2_250

Back in January, Peta Pixel shared a series of photos from Infinity Imagined that compared images of cities at night as seen from the International Space Station with neurons imaged with fluorescence microscopy. The similarities are striking and upon first blush, quite compelling.

But if you think about it, these are an excellent example of confirmation bias: finding correlation where you look for it. For every “match” between a city and a neuron, there must be many non-matches that were deliberately set aside; pictures of cities that didn’t fit the hypothesis. This kind of selective thinking occurs in many areas of life – astrologers and numerologists, for example, count on their audience remembering the “predictions” that came true while conveniently forgetting the vast majority that did not.

Lately there has been an uptick in negative press surrounding researchers in the fields of neuroscience, psychology and sociology. They are accused of regularly engaging in a form of selective thinking by bending data to reach desired conclusions. Some are even calling it a “mini crisis.” The accusations range from double-dipping data, to using too-small sample sizes, to outright fraud.

Here is a sampling of the recent bad press:

  • According to an analysis of 49 meta-analyses, the field of neuroscience produces a lot of small, low-powered studies, which leads to a lot of false and/or misleading conclusions (a.k.a. “discoveries”).
  • A scathing take-down of a study linking fist-clenching with memory. In the comments section, no less! (Another take-down was just posted on The Neurocritic).
  • A 2009 study about the practice of double-dipping (using overlapping data) in the field of neuroscience was recently making the rounds.
  • A lengthy profile in the Sunday New York Times Magazine of the eminently unlikeable Diederik Stapel, the infamous Dutch social sociologist who perpetrated perhaps the biggest academic fraud in the field of sociology (retractions to date: 53 articles and counting). (Film studies side note: Stapel’s account of going back to the sites of some his faked experiments and trying to make the actual setting fit his fabricated descriptions is strikingly similar to the sequence in Shattered Glass when Chuck Lane goes back to the locations Stephen Glass describes in his fabricated stories and tries to reconstruct the truth).

But there may be some good news. A recent editorial in The New Yorker by Gary Marcus says that this is all a tempest in a teapot. Marcus doesn’t claim that the accusations are wrong per se, but rather that the field of psychology is well prepared to address the problems. He also assures us we’ll be better off for the effort, even if we have to suffer through a “lost decade” of dubious research.

Finally, if you want to keep track of scientists keeping track of themselves, I recommend the blog Retraction Watch. It reads like a gossip column about all manner of scientific bad behavior. It’s the TMZ of science!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s