I spent last Thursday and Friday (February 20 and 21) at an NSF workshop concerning the replicability of research results. It was chaired by John Cacioppo and included about 30 participants including such well-known contributors to the discussion as Brian Nosek, Hal Pashler, Eric Eich, and Tony Greenwald, to name a few. Participants also included officials from NIH, NSF, the White House Office on Science and Technology and at least one private foundation. I was invited, I presume, in my capacity as Past-President of SPSP and chair of an SPSP task force on research practices which recently published a report on non-retracted PSPB articles by investigators who retracted articles elsewhere, and a set of recommendations for research and educational practice, which was just published in PSPR.
Committees, task forces and workshops – whatever you call them – about replicability issues have become almost commonplace. The SPSP Task Force was preceded by a meeting and report sponsored by the European Association of Personality Psychology, and other efforts have been led by APS, the Psychonomic Society and other organizations. Two symposia on the subject were held at the SPSP meeting in Austin just the week before. But this discussion was perhaps special, because it is the first (to my knowledge) to be sponsored by the US government, with the explicit purpose of seeking advice about what NSF and other research agencies should do. I think it is fair to say: When replication is discussed in a meeting with representatives from NIH, NSF and the White House, the issue is on the front burner!
The discussion covered several themes, some of which are more familiar than others. Continue reading
We recently read Karg et al (2011) for a local reading group. It is one of the many of attempts to meta-analytically examine the idea that the 5-HTTLPR serotonin transporter polymorphism moderates the effect of stress on depression.
It drove me batty. No, it drove me to apoplectia–a small country in my mind I occupy far too often.
Let’s focus on the worst part. Here’s the write up in the first paragraph of the results:
“We found strong evidence that 5-HTTLPR moderates the relationship between stress and depression, with the s allele associated with an increased risk of developing depression under stress (P = .00002). The significance of the result was robust to sensitivity analysis, with the overall P values remaining significant when each study was individually removed form the analysis (1.0×10-6<P<.00016).”
Wow. Continue reading
Full Disclosure: I am second author on the McDonald et al. (2014) commentary.
Some of you may have seen that Psychological Science published our commentary on the Birtel and Crisp (2012) paper. Essentially we tried to replicate two of their studies with larger sample sizes (29 versus 240 and 32 versus 175, respectively) and obtained much lower effect size estimates. It is exciting that Psychological Science published our work and I think this is a hint of positive changes for the field. Hopefully nothing I write in this post undercuts that overarching message.
I read the Crisp and Birtel response and I had a set of responses (shocking, I know!). I think it is fair that they get the last word in print but I had some reactions that I wanted to share. Thus, I will outlet a few in this blog post. Before diving into issues, I want to reiterate the basic take home message of McDonald et al. (2014):
“Failures to replicate add important information to the literature and should be a normal part of the scientific enterprise. Continue reading
In thinking about the past three years writing for PYM, I just realized that I write a lot of posts about issues that graduate students care about. I've made a list with links to each of these posts below. Now all the "wisdom" I have to offer about graduate school is in one place. I hope this will help you--current and future graduate students in psychology--to navigate the challenges and opportunities that many of us face on our way to a PhD! Good luck in your journey and don't be afraid to leave comments or questions on the post or on twitter (@mwkraus