The Problem with Selective Selection

Back in January, Peta Pixel shared a series of photos from Infinity Imagined that compared images of cities at night as seen from the International Space Station with neurons imaged with fluorescence microscopy. The similarities are striking and upon first blush, quite compelling.

But if you think about it, these are an excellent example of confirmation bias: finding correlation where you look for it. For every “match” between a city and a neuron, there must be many non-matches that were deliberately set aside; pictures of cities that didn’t fit the hypothesis. This kind of selective thinking occurs in many areas of life – astrologers and numerologists, for example, count on their audience remembering the “predictions” that came true while conveniently forgetting the vast majority that did not.

Lately there has been an uptick in negative press surrounding researchers in the fields of neuroscience, psychology and sociology. They are accused of regularly engaging in a form of selective thinking by bending data to reach desired conclusions. Some are even calling it a “mini crisis.” The accusations range from double-dipping data, to using too-small sample sizes, to outright fraud.

Here is a sampling of the recent bad press:

• According to an analysis of 49 meta-analyses, the field of neuroscience produces a lot of small, low-powered studies, which leads to a lot of false and/or misleading conclusions (a.k.a. “discoveries”).
  
• A scathing take-down of a study linking fist-clenching with memory. In the comments section, no less! (Another take-down was just posted on The Neurocritic).
  
• A 2009 study about the practice of double-dipping (using overlapping data) in the field of neuroscience was recently making the rounds.
• A lengthy profile in the Sunday New York Times Magazine of the eminently unlikeable Diederik Stapel, the infamous Dutch social sociologist who perpetrated perhaps the biggest academic fraud in the field of sociology (retractions to date: 53 articles and counting). (Film studies side note: Stapel’s account of going back to the sites of some his faked experiments and trying to make the actual setting fit his fabricated descriptions is strikingly similar to the sequence in Shattered Glass when Chuck Lane goes back to the locations Stephen Glass describes in his fabricated stories and tries to reconstruct the truth).

But there may be some good news. A recent editorial in The New Yorker by Gary Marcus says that this is all a tempest in a teapot. Marcus doesn’t claim that the accusations are wrong per se, but rather that the field of psychology is well prepared to address the problems. He also assures us we’ll be better off for the effort, even if we have to suffer through a “lost decade” of dubious research.

Finally, if you want to keep track of scientists keeping track of themselves, I recommend the blog Retraction Watch. It reads like a gossip column about all manner of scientific bad behavior. It’s the TMZ of science!

Illustrated Timeline of the History of Schizophrenia Through the Ages

Scientific American just published an interactive timeline of the history of schizophrenia, from Ancient Egypt to the present. It’s very interesting, and comes on the heels of important discussions taking place surrounding the release of the DSM-5. From the article:

“Less than two hundred years ago, schizophrenia emerged from a tangle of mental disorders known simply as madness. Yet its diagnosis remains shrouded in ambiguity. Only now is the Diagnostics and Statistical Manual of Mental Disorders, psychiatrists’ primary guidebook, shedding the outdated, nineteenth-century descriptions that have characterized schizophrenia to this day.

“There is substantial dissatisfaction with schizophrenia treated as a disease entity, its symptoms are like a fever—something is wrong but we don’t know what,” says William Carpenter, a psychiatrist at the University of Maryland and chair of the manual’s Psychotic Disorder Workgroup. Psychiatrists may discover that this disorder is not a single syndrome after all but a bundle of overlapping conditions.”

Further Reading:
Vaughan Bell’s latest article, about new research on schizophrenia.

The Trouble With Pre-Crime Assessment

Still from “Minority Report” (2002).

In the wake of the tragedy at Sandy Hook, there has been increased interest in looking at ways to prevent such devastating crimes. Variations on the theme of gun control is obviously at the forefront of that discussion, judging by the national debate at the moment. But others are talking about psychological screening tools to asses future risk of violence in criminals.

With echoes of the fantastical PreCrime Unit in the sci-fi thriller Minority Report, the desire for a pre-screener for violence is nothing new. But a recent article published in the journal Behavioral Sciences & The Law details what the authors – preeminent forensic psychologists – conclude is a fatal flaw in actuarial risk assessment instruments used to predict recidivism rates. From the abstract:

“Consistent with past research, ARAI scores were moderately and significantly predictive of failure in the aggregate, but group probability estimates had substantial margins of error and individual probability estimates had very large margins of error.”

Described another way,

“[T]he researchers established through a traditional statistical procedure, logistic regression, that the margins of error around individual scores were so large as to make risk distinctions between individuals “virtually impossible.” In only one out of 90 cases was it possible to say that a subject’s predicted risk of failure was significantly higher than the overall baseline of 18 percent.”

At issue is whether or not such actuarial surveys can be admissible in court. Researchers Stephen Hart and David Cooke say ‘no,’ and declare they have definitively proven that any accuracy in predictive ability is a statistical artifact resulting in “fundamental uncertainty,” whereas the massive margin of error is “reality.”

For a very readable summary and analysis of the research, I suggest this blog post by forensic psychologist Karen Franklin, Ph.D. Some highlights:

• The APA ethics code requires psychologists to inform clients of “the strengths and limitations of test results and interpretation” and to “indicate any significant limitations of their interpretations.”
• The fundamental uncertainty of actuarial risk assessment “cannot be overcome,” therefore Hart and Cooke recommend use of such statistical algorithms be stopped.
• The “image of certitude” projected by actuarial risk assessments is misleading and can result in cognitive biases, therefore their admissibility in court should be seriously questioned.
• Courts should not rely on any one assessment of an individual’s supposed traits or characteristics, but instead must look at all information in context.

As I see it, the implications for school psychologists are twofold. First, school psychologists should be wary of any assessment instrument purporting to determine a student’s risk of future violence. Just the legal implications alone should give pause to anyone considering administering – and interpreting – such an assessment in the context of a school setting.

Second, Hart and Cooke’s final recommendation that legal professionals and the courts “recognize that their decisions ultimately require consideration of the totality of circumstances – not just the items of a particular test” is exactly in line with best practices for school psychologists. We are never supposed to make an academic or clinical decision based on the results of only one test.

The pressure is on right now, and in such a climate schools may be tempted to change or adopt practices quickly in order to ease political tension. Research like this reminds us of the importance of ethics codes and best practices guides: they provide a steady rudder to guide us through periodic storms.

Further reading:

Link to original journal article.
Link to Karen Franklin’s blog post about the article.
The Minority Report and Other Classic Stories by Philip K. Dick

Oh, The Humanities!

New research shows that reading difficult prose and poetry “lights up” the brain more than reading dumbed-down versions of the same text. Specifically, researchers gave subjects passages by Shakespeare, Wordsworth, and T. S. Eliot (among others). The readers’ brains were then monitored as they read, and the results seem to be two-fold.

First, there was more electrical activity in the brain while subjects read the difficult (original) passages. Second, when subjects read poetry, regions in the brain associated with memory and self-reflection became active.

“The [original] version caused a greater degree of brain activity, lighting up not only the left part of the brain concerned with language, but also the right hemisphere that relates to autobiographical memory and emotion.”

The study’s authors suggest that this pattern of brain activity indicates a deeper engagement with the text that leads to meaningful self reflection. Philip Davis, an English professor who was part of the research team, said this:

“Poetry is not just a matter of style. It is a matter of deep versions of experience that add the emotional and biographical to the cognitive[…]This is the argument for serious language in serious literature for serious human situations, instead of self-help books or the easy reads that merely reinforce predictable opinions and conventional self-images.”

As a longtime Shakespeare disciple, I’m excited to see scientific confirmation of what his fans have known for centuries: Shakespeare really does put the ‘human’ in ‘humanities.’

Interestingly, this comes at a time when recent Common Core guidelines are being interpreted as calling for fiction and poetry to be gradually tapered from classroom instruction in order to make room for “harder” reading, such as technical manuals and non-fiction writing. Others say teachers are misinterpreting the guidelines and that Shakespeare still should be taught.

States are just beginning the long journey to the Common Core, so time will tell what it will ultimately look like. Let’s just hope that by the time it’s all sorted out, when we ask a student what he’s reading, rather than answering “Words, words, words,” he’ll say “How many goodly creatures are there here! How beauteous mankind is! O brave new world, That has such people in’t!”

Link to another news story about the study, with a slightly different angle.
Link to Science Daily report on the study.
Link to LA Times editorial about the implications of Common Core.

JSTOR Opens Some Archives to the Public

A couple of days ago, the online library JSTOR announced that they are opening part of their archives and making them available for free. Over 1,200 journals will now be available to anyone who signs up for a JSTOR account. Usually, unless you are affiliated with a university library, most JSTOR content is not freely available to the public. Most public libraries can’t afford to pay for a JSTOR account, so their archives have been notoriously unavailable to lay people. While this is a nice gesture, the access is still somewhat limited: once you sign up for an account, you can read “up to three articles for free every two weeks.”

Many people feel that all publicly funded research should be freely available to…the public. Critics argue that commercial entities have made huge profits on the results of research funded by taxpayers. The United Kingdom seems to be a pioneer in making taxpayer-funded research freely available, thanks in large part to the work of Dame Janet Fitch. In July of last year, the British government accepted her recommendations.

The U.S. is still arguing with itself about how to approach the issue, and many people are not optimistic about a resolution. But, as professor Michael B. Eisen wrote in a New York Times op-ed piece last year, “Rather than rolling back public access, Congress should move to enshrine a simple principle in United States law: if taxpayers paid for it, they own it.”

*****

In related news, Reddit co-founder and online activist Aaron Swartz died yesterday, the result of suicide. He had battled depression for years and wrote publicly and eloquently about it on his blog. But he was also facing trial after being accused of illegally downloading millions of articles from  MIT and JSTOR archives. The case was controversial, with some claiming Swartz broke no laws. I guess we’ll never know what the legal outcome would have been, but it’s a sad coincidence that his death came just days after JSTOR made their announcement.

A Tale of Two News Stories

If you were in a hurry and only had time to read the first half of the #5 most e-mailed article on the New York Times website today, titled “Study Questions Effectiveness of Therapy for Suicidal Teenagers,” you probably would have come away pondering the following important points:

1. “55 percent of suicidal teenagers had received some therapy before they thought about suicide, planned it or tried to kill themselves, contradicting the widely held belief that suicide is due in part to a lack of access to treatment.”
2. “…[T]he new study is the first to suggest, in a large nationwide sample, that access to treatment does not make a big difference.”
3. “The study suggests that effective treatment for severely suicidal teenagers must address not just mood disorders, but also behavior problems that can lead to impulsive acts, experts said.”

But, if you read the first half of a Reuters article posted on the NBC News website titled, “1 in 25 U.S. teens attempts suicide, national study finds,” you probably would have come away pondering these important points:

1. “About one in 25 U.S. teens has attempted suicide, according to a new national study, and one in eight has thought about it.”
2. “Just over 12 percent of the youth had thought about suicide, and four percent each had made a suicide plan or attempted suicide.”
3. “…[A]lmost all teens who thought about or attempted suicide had a mental disorder, including depression, bipolar disorder, attention deficit hyperactivity disorder (ADHD) or problems with drug or alcohol abuse.”

The difference? The first story initially focuses on the fact that more than half of teens who attempt suicide had already received mental health treatment; the second story places its initial focus on facts and figures and diagnoses. Interestingly, if you continued to read both articles they wind up mirroring one another: The NY Times article ends with a discussion of the facts and figures and diagnoses, while the NBC News article ends with a discussion of the previous treatment issue.

In a continuation of the overall mirroring between articles, both include daunting quotes from mental health professionals:

NY Times: “I think one of the take-aways here is that treatment for depression may be necessary but not sufficient to prevent kids from attempting suicide,” said Dr. David Brent, a professor of psychiatry at the University of Pittsburgh, who was not involved in the study. “We simply do not have empirically validated treatments for recurrent suicidal behavior.”

NBC News: “We know that a lot of the kids who are at risk and thinking about suicide are getting (treatment),” [researcher Matthew Nock] told Reuters Health. However, “We don’t know how to stop them – we don’t have any evidence-based treatments for suicidal behavior.”

So what is the take away from this little exercise? Both articles were written about the same research study. Yet a causal reader would have come away with different impressions depending on which article they read. An important reminder to cast a net far and wide when analyzing data, and always consult the original source when possible.

Speaking of the original source, there was an interesting finding not mentioned in either article. The study found that more than 80% of suicidal adolescents “receive some form of mental health treatment.” The 55% figure quoted in both news stories above applies only to therapy that had begun before the “onset of suicidal behaviors,” yet failed to prevent them.

Link to original JAMA article.

Relevant to the Situation

Earlier this week my child & adolescent therapy class discussed an unfortunate situation involving multiple children being take in by relatives after their mother was arrested. The question was how best to help children in a situation like that, and help them cope with a suddenly chaotic existence infused with grief, loss, anger, and fear. At one point I wondered aloud where the father of the children was in all of this, and a person who was familiar with the family said, “It’s not relevant to this situation.” If that’s true, then I feel even worse for these children.

Of course fathers are relevant, and their absence in a child’s life can be a predictor of a host of negative outcomes including sexual promiscuity, increased risk of incarceration, emotional and behavioral problems, and increased risk of poverty. And even though there is a growing body of research on the importance of fathers, as a society we still tend to marginalize fathers and in some instances assume they have less interest in parenting their children than do mothers.

If you’d like more information, the U.S. Department of Health and Human Services has put together an extensive resource list called Resource Listings of Selected National Organizations Concerned with Fatherhood and Child Maltreatment. There are programs to get fathers involved in schools, information for stay-at-home dads, resources for low income fathers, minority fathers, incarcerated fathers, parenting groups, and more.

And last but not least, a list.

20 Reasons Why Your Child Needs You to be an Active Father:

1. Lets your child know that you love her. Love involves more than saying the words, “I love you.” Fathers who love their children demonstrate their love by spending quality and quantity time together. Children who feel loved are more likely to develop a strong emotional bond with their father and a healthy self-esteem.

2. Provides your child with greater financial resources. Research clearly indicates that families with an active father are “better off” financially. This means that children with active fathers will be more likely to have access to resources that facilitate healthy development (e.g., food, clothing, shelter, quality medical care).

3. Provides your child with a positive male role model. Children, regardless of gender, need positive male and female role models. Children tend to model behavior (positive and negative) that they witness on a consistent basis. Active fathers can promote positive behaviors by setting a proper example for their children.

4. Provides your child with emotional support. In addition to financial support, children also need emotional support from their parents. Active fathers listen and support their children when they experience joy, sadness, anger, fear, and frustration. Fathers who support their children emotionally tend to raise children who are more in-tune with the needs of others.

5. Enhances your child’s self-esteem. Self-esteem refers to how a person feels about himself. Children with high self-esteem tend to be happier and more confident than children with low self esteem. Active fathers promote their children’s self-esteem by being fully involved in their lives and letting them know that they are highly valued.

6. Enhances your child’s intellectual development. Children who are raised with actively involved fathers tend to score higher on measures of verbal and mathematical ability, and also demonstrate greater problem-solving and social skills.

7. Provides your child with guidance and discipline. From infancy, children need proper guidance and discipline. Active fathers play an important role in teaching their children proper behavior by setting and enforcing healthy limits.

8. Gives your child someone to play with. One of the primary ways that fathers bond with their children is through play. According to researchers, there are qualitative differences in the ways fathers and mothers play with their children. Fathers tend to use a more physical style of play (e.g., wrestling) that offers a number of benefits to children, including enhanced cognitive ability.

9. Provides your child with someone to talk to when she has questions. Young children are full of questions. This natural curiosity helps them learn about their environment. Active fathers can be a valuable source of information for children who are seeking answers to life’s important questions.

10. Increases your child’s chances for academic success. Children whose fathers are actively involved in their lives are more likely to achieve academic success than children whose fathers are not actively involved. These academic benefits appear to extend into adulthood.

11. Provides your child with an alternative perspective on life. Research indicates that men and women often differ in their parenting styles; however, one style is not necessarily better than the other. Instead, it can be healthy for children to be exposed to different perspectives on life, such as a father’s.

12. Lowers your child’s chances for early sexual activity. Children with actively involved fathers are less likely to engage in early sexual activity, thus reducing their chances for teen pregnancy and sexually transmitted diseases.

13. Lowers your child’s chances for school failure. Children with actively involved fathers are less likely to drop out of school than children with uninvolved fathers.

14. Lowers your child’s chances for youth suicide. Children with actively involved fathers are less likely to commit suicide than children with uninvolved fathers.

15. Lowers your child’s chances for juvenile delinquency. The benefits of having an active father throughout a child’s early years extend into the teen years as well. Children with active fathers are less likely to commit juvenile crimes than children with inactive fathers.

16. Lowers your child’s chances for adult criminality. The chances that a child will commit crimes as an adult also diminish when he grows up with an actively involved father.

17. Provides your child with a sense of physical and emotional security. One of the major benefits that fathers can provide to their children by being actively involved is a sense of security (physical and emotional). By being actively involved in a child’s life, a father promotes a trusting relationship. The child does not have to worry about being abandoned.

18. Facilitates your child’s moral development. Children need a moral compass to guide them when they face difficult moral choices. Fathers, like mothers, help children to develop a sense of right and wrong that serves as a foundation for establishing moral character.

19. Promotes a healthy gender identity in your child. Boys and girls benefit from having healthy role models from both sexes. Research points to the fact that mothers and fathers socialize their children in different ways. Fathers can help their children, especially boys, to develop a healthy sense of what it means to be a male.

20. Helps your child learn important life skills. Most of the essential life skills that children need to survive are learned within the home. Fathers have a unique opportunity to teach their children valuable skills that will enable them to grow up to be healthy and productive adults.

Fake It ‘Til You Make It

Research has shown that the simple act of sitting up straight can improve confidence. In a recently posted TED talk, social psychologist Amy Cuddy takes this idea a step further and introduces us to the concept of the “power pose”: feet planted firmly on the ground about shoulder-width apart, knees locked, arms akimbo. Power pose! Or, as Cuddy calls it, “A free no-tech life hack.” She goes on to say that our body language is a form of communication:

“…[W]hen we think of nonverbals, we think of how we judge others, how they judge us and what the outcomes are. We tend to forget, though, the other audience that’s influenced by our nonverbals, and that’s ourselves…

We are also influenced by our nonverbals, our thoughts and our feelings and our physiology. So what nonverbals am I talking about? I’m a social psychologist. I study prejudice, and I teach at a competitive business school, so it was inevitable that I would become interested in power dynamics. I became especially interested in nonverbal expressions of power and dominance.

And what are nonverbal expressions of power and dominance? Well, this is what they are. So in the animal kingdom, they are about expanding. So you make yourself big, you stretch out, you take up space, you’re basically opening up. It’s about opening up.”

Cuddy explains that her studies have shown that not only do we change others’ perceptions of us when we power pose, we also alter our brain chemistry such that we actually start to feel better about ourselves. In animal studies, alpha leaders tend to have high levels of testosterone (for dominance) combined with low levels of cortisol (reduced stress; you can’t be an effective alpha if you’re both dominant and overwhelmed by stress). To study whether this hormonal interplay also occurs in humans, Cuddy and her fellow researchers had subjects pose in either strong or weak positions, let them gamble, and had them fill out a questionnaire about how powerful they felt. The researchers then took saliva samples from the subjects. As it turns out, the “high power” group had reduced levels of cortisol and the “low power” group had increased levels of cortisol. Cuddy’s conclusion? “Our nonverbals govern how we think and feel about ourselves.” Something to keep in mind next time you head to a job interview. So maybe rather than ‘fake it ’til you make it,’ the more accurate conclusion is ‘fake it and you WILL make it.’

And while the way in which this research was conducted may be new, judging by this 1953 video, it seems we’ve been concerned about posture for quite a while. What we now understand as cortisol and testosterone and brain science used to simply be called “health.” Sometimes you can’t help but think, the more things change, the more they stay the same. [click on image below to go to video].

Debunking an Urban Myth

For today’s post, I want to address an urban myth that apparently has been making the rounds for years. I heard this last year in one of my classes, and then it came up indirectly again last month in a discussion about the importance of cementing reading skills by third grade. The myth is that states use third grade literacy scores to determine how many prisons to build (or how many prison beds they will need). This is not true, and although it has been cited repeatedly, nobody seems to know the origin of the myth.

One possible source might be a book titled The School-To-Prison Pipeline: Structuring Legal Reform. I haven’t read the book, but last week someone I know was discussing it and cited the third grade literacy quote in such a way that it seemed it came from the book. At any rate, the idea that officials use third grade reading scores to determine how many prison beds they will need has now been debunked and relegated to the dustbin of urban myths. The only problem, of course, is that (almost) nobody knows this. So I’m going to do my part to ensure that truth soldiers on, and urban myths…don’t.

Here are a couple of links to explain how and why the erroneous quote has been proven false, including perhaps the original article in the Oregonian that started the debunkification:

Prisons don’t use reading scores to predict future inmate populations: “The myth probably has survived and circulated for more than a decade because it reflects the more fundamental truth that there is a powerful connection between school failure and crime.”

Cite the Source: “The idea that prison-planning is based on elementary school literacy drop-out rates is a commonly held assumption…The problem is that I can’t find one single piece of state legislation or governmental corroboration for this. Nowhere can I locate official documentation that prisons indeed “use reading scores to predict future inmate populations.” As far as evidence goes, I can find nothing.”

PolitiFact Oregon: “We started with a basic Google search and came across the claim we’d kept hearing: Prison officials use third grade reading scores to predict the number of beds they’ll need. That claim, it seems, is nothing more than an Internet rumor that has been soundly disputed. In fact, The Oregonian happened to refute the adage a couple years back. “This is an urban myth,” California Department of Corrections and Rehabilitation spokeswoman Terry Thornton wrote in an email to Oregonian reporter Bill Graves.”

Monday Giveaway!

Last month I wrote about The New Teacher Project and their research on the irreplaceables: teachers who are “so successful that they are nearly impossible to replace.” I went ahead and ordered two copies of their report, and I’m giving them away to you! If you would like one, email me at [schoolpsychscholar at gmail dot com]. Be sure to include your name and mailing address in the email. The first two people to email me will get a copy. Happy Monday!  Update: all gone!