Monthly Archives: May 2014

Apology – Brent Donnellan (The Trait-State Continuum)

Posted on May 28, 2014 by | Comments Off on Apology – Brent Donnellan (The Trait-State Continuum)

There has been a lot of commentary about the tone of my 11 December 2013 blog post. I’ve tried to keep a relatively low profile during the events of the last week.  It has been one of the strangest weeks of my professional life. However, it seems appropriate to make a formal apology.

1. I apologize for the title.  I intended it as a jokey reference for the need to conduct high power replication studies. It was ill advised.

2. I apologize for the now infamous “epic fail” remark (“We gave it our best shot and pretty much encountered an epic fail as my 10 year would say”). It was poor form and contributed to hurt feelings. I should have been more thoughtful.

Continue reading →

Comments Off on Apology – Brent Donnellan (The Trait-State Continuum)

Posted in The Trait-State Continuum

Notes on Replication from an Un-Tenured Social Psychologist – Michael Kraus (Psych Your Mind)

Posted on May 27, 2014 by | Comments Off on Notes on Replication from an Un-Tenured Social Psychologist – Michael Kraus (Psych Your Mind)
Last week the special issue on replication at the Journal of Social Psychology arrived to an explosion of debate (read the entire issue here and read original author Simone Schnall's commentary on her experience with the project and Chris Fraley's subsequent examination of ceiling effects). The debate has been happening everywhere--on blogs, on twitter, on Facebook, and in the halls of your psychology department (hopefully).
Read More->

Comments Off on Notes on Replication from an Un-Tenured Social Psychologist – Michael Kraus (Psych Your Mind)

Posted in Psych Your Mind

Tagged , , ,

An apology and proposal – Brent Roberts (pigee)

Posted on May 26, 2014 by | Comments Off on An apology and proposal – Brent Roberts (pigee)

Brent W. Roberts

My tweet, “Failure to replicate hurting your career? What about PhDs with no career because they were honest” was taken by some as a personal attack on Dr. Schnall.  It was not and I apologize to Dr. Schnall if it were taken that way. The tweet was in reference to the field as a whole because our current publication and promotion system does not reward the honest design and reporting of research. And this places many young investigators at a disadvantage. Let me explain.

Our publication practices reward the reporting of optimized data—the data that looks the best or that could be dressed up to look nice through whatever means necessary. We have no choice given the way we incentivize our publication system. That system, which punishes null findings and rewards only statistically significant effects means that our published science is not currently an honest portrait of how our science works. The current rash of failures to replicate famous and not so famous studies is simply a symptom of a system that is in dire need of reform. Continue reading →

Comments Off on An apology and proposal – Brent Roberts (pigee)

Posted in pigee

I’m disappointed: A graduate student’s perspective – Carol Tweten (Person X Situation)

Posted on May 25, 2014 by | Comments Off on I’m disappointed: A graduate student’s perspective – Carol Tweten (Person X Situation)

I recently created this blog, and I had no intention of writing my first post about the replication conversation (if you need to catch up, David Johnson just posted links to everything here), especially because all of the points I would make have already been made and I’m getting tired of reading the same arguments over and over.

But, I* do feel the need to say: I am disappointed in our field right now. As a graduate student in training for a career in Social/Personality Psychology, you (i.e., established researchers) are the people I am learning from. I should be able to look up to and model your professionalism, passion, and dedication to science. But instead, I am reading personal attacks, inappropriate references to respected historical figures (i.e., Rosa Parks), and a conversation focused on reputations as opposed to the purpose and pursuit of scientific principles. Of course, not every comment/post meets these criteria, but enough do that, only 2 years into graduate school, I’m already starting to question what it is that we, as scientists, are doing.

SO, I urge that the conversation move forward. Discuss, for example, the potential diversity problem in this conversation. I also want to point out that all of this (at least what I’ve seen most recently…) is about ONE of the fifteen replications published in this special issue. Continue reading →

Comments Off on I’m disappointed: A graduate student’s perspective – Carol Tweten (Person X Situation)

Posted in Person X Situation

Does the replication debate have a diversity problem? – Sanjay Srivastava (The Hardest Science)

Posted on May 25, 2014 by | Comments Off on Does the replication debate have a diversity problem? – Sanjay Srivastava (The Hardest Science)

Folks who do not have a lot of experiences with systems that don’t work well for them find it hard to imagine that a well intentioned system can have ill effects. Not work as advertised for everyone. That is my default because that is my experience.
– Bashir, Advancing How Science is Done

A couple of months ago, a tenured white male professor* from an elite research university wrote a blog post about the importance of replicating priming effects, in which he exhorted priming researchers to “Nut up or shut up.”

Just today, a tenured white male professor* from an elite research university said that a tenured scientist who challenged the interpretation and dissemination of a failed replication is a Rosa Parks, “a powerless woman who decided to risk everything.”

Well then.

The current discussion over replicability and (more broadly) improving scientific integrity and rigor is an absolutely important one. It is, at its core, a discussion about how scientists should do science. It therefore should include everybody who does science or has a stake in science.

Yet over the last year or so I have heard a number of remarks (largely in private) from scientists who are women, racial minorities, and members of other historically disempowered groups that they feel like the protagonists in this debate consist disproportionately of white men with tenure at elite institutions. Continue reading →

Comments Off on Does the replication debate have a diversity problem? – Sanjay Srivastava (The Hardest Science)

Posted in The Hardest Science

Can Cannabis Cause Psychosis? A Hard Question to Answer – Scott McGreal (Unique—Like Everybody Else)

Posted on May 24, 2014 by | Comments Off on Can Cannabis Cause Psychosis? A Hard Question to Answer – Scott McGreal (Unique—Like Everybody Else)
Although a number of long-term studies have linked cannabis use to later risk of mental illness, the question of whether the one causes the other remains unresolved. The possibility that a third factor, such as genetic or personality predispositions, underlies both cannabis use and the development of psychosis needs to be considered more carefully in future research.

read more

Comments Off on Can Cannabis Cause Psychosis? A Hard Question to Answer – Scott McGreal (Unique—Like Everybody Else)

Posted in Unique—Like Everybody Else

Tagged

Additional Reflections on Ceiling Effects in Recent Replication Research – Brent Roberts (pigee)

Posted on May 23, 2014 by | Comments Off on Additional Reflections on Ceiling Effects in Recent Replication Research – Brent Roberts (pigee)

By R. Chris Fraley

In her commentary on the Johnson, Cheung, and Donnellan (2014) replication attempt, Schnall (2014) writes that the analyses reported in the Johnson et al. (2014) paper “are invalid and allow no conclusions about the reproducibility of the original findings” because of “the observed ceiling effect.”

I agree with Schnall that researchers should be concerned with ceiling effects. When there is relatively little room for scores to move around, it is more difficult to demonstrate that experimental manipulations are effective. But are the ratings so high in Johnson et al.’s (2014) Study 1 that the study is incapable of detecting an effect if one is present?

Image

To address this question, I programmed some simulations in R. The details of the simulations are available at http://osf.io/svbtw, but here is a summary of some of the key results:

  • Although there are a large number of scores on the high end of the scale in the Johnson et al. Continue reading →

Comments Off on Additional Reflections on Ceiling Effects in Recent Replication Research – Brent Roberts (pigee)

Posted in pigee

Random Reflections on Ceiling Effects and Replication Studies – Brent Donnellan (The Trait-State Continuum)

Posted on May 21, 2014 by | Comments Off on Random Reflections on Ceiling Effects and Replication Studies – Brent Donnellan (The Trait-State Continuum)

In a blog post from December of 2013, I  described our attempts to replicate two studies testing the claim that priming cleanliness makes participants less judgmental on a series of 6 moral vignettes. My original post has recently received criticism for my timing and my tone. In terms of timing, I blogged about a paper that was accepted for publication and there was no embargo on the work. In terms of tone, I tried to ground everything I wrote with data but I also editorialized a bit.  It can be hard to know what might be taken as offensive when you are describing an unsuccessful replication attempt. The title (“Go Big or Go Home – A Recent Replication Attempt”) might have been off putting in hindsight. In the grand scope of discourse in the real world, however, I think my original blog post was fairly tame.

Most importantly: I was explicit in the original post about the need for more research. I will state again for the record: I don’t think this matter has been settled and more research is needed. We also said this in the Social Psychology paper.  It should be widely understood that no single study is ever definitive.

As noted in the current news article for Science about the special issue of Social Psychology, there is some debate about ceiling effects with our replication studies. We discuss this issue at some length in our rejoinder to the commentary. Continue reading →

Comments Off on Random Reflections on Ceiling Effects and Replication Studies – Brent Donnellan (The Trait-State Continuum)

Posted in The Trait-State Continuum

Big Data, n. A kind of black magic – Tal Yarkoni ([citation needed])

Posted on May 19, 2014 by | Comments Off on Big Data, n. A kind of black magic – Tal Yarkoni ([citation needed])

The annual Association for Psychological Science meeting is coming up in San Francisco this week. One of the cross-cutting themes this year is “Big Data: Understanding Patterns of Human Behavior”. Since I’m giving two Big Data-related talks (1, 2), and serving as discussant on a related symposium, I’ve been spending some time recently trying to come up with a sensible definition of Big Data within the context of psychological science. This has, in turn, led me to ponder the meaning of Big Data more generally.

After a few sleepless nights mulling it over for a while, I’ve concluded that producing a unitary, comprehensive, domain-general definition of Big Data is probably not possible, for the simple reason that different communities have adopted and co-opted the term for decidedly different purposes. For example, in said field of psychology, the very largest datasets that most researchers currently work with contain, at most, tens of thousands of cases and a few hundred variables (there are exceptions, of course). Such datasets fit comfortably into memory on any modern laptop; you’d have a hard time finding (m)any data scientists willing to call a dataset of this scale “Big”. Yet here we are, heading into APS, with multiple sessions focusing on the role of Big Data in psychological science. And psychology’s not unusual in this respect; we’re seeing similar calls for Big Data this and Big Data that in pretty much all branches of science and every area of the business world. I mean, even the humanities are getting in on the action.

You could take a cynical view of this and argue that all this really goes to show is that people like buzzwords. And there’s probably some truth to that. More pragmatically, though, we should acknowledge that language is this flexible kind of thing that likes to reshape itself from time to time. Continue reading →

Comments Off on Big Data, n. A kind of black magic – Tal Yarkoni ([citation needed])

Posted in citation needed

you had me at quintillion – Simine Vazire (sometimes i'm wrong)

Posted on May 12, 2014 by | Comments Off on you had me at quintillion – Simine Vazire (sometimes i'm wrong)

Helix2

i don't mean to pile on,* but want to share two quick thoughts about the jens förster case.

1.

first, i think the evidence is pretty overwhelming that the data in the paper in question are not real.  in addition to the original report and LOWI committee's report, additional analyses by the data colada guys show that the pattern of results could never have happened without data manipulation. 

this case is interesting in part because the evidence for fraud comes from statistical analyses of the published results, rather than from a whistleblower inside the lab, or a confession.  this makes some people uncomfortable.  i agree that concluding fraud based on probabilities could be problematic - this case makes us wonder what we would think if the odds were, say, 1 in 10,000.  how small does the probability have to be for us to conclude fraud?

i don't know, but i know 1 in 508 quintillion is improbable enough.  i agree it is worth thinking about what we should do with borderline cases, but it is also important to recognize that this is not a borderline case. Continue reading →

Comments Off on you had me at quintillion – Simine Vazire (sometimes i'm wrong)

Posted in sometimes i'm wrong