(Sample) Size Matters – Michael Kraus (Psych Your Mind)

Sample Size Matters
On this blog and others, on twitter (@mwkraus), at conferences, and in the halls of the psychology building at the University of Illinois, I have engaged in a wealth of important discussions about improving research methods in social-personality psychology. Many prominent psychologists have offered several helpful suggestions in this regard (here, here, here, and here).

Among the many suggestions for building a better psychological science, perhaps the simplest and most parsimonious way to improve research methods is to increase sample sizes for all study designs: By increasing sample size researchers can detect smaller real effects and can more accurately measure large effects. There are many trade-offs in choosing appropriate research methods, but sample size, at least for a researcher like me who deals in relatively inexpensive data collection tools, is in many ways the most cost effective way to improve one's science. In essence, I can continue to design the studies I have been designing and ask the same research questions I have been asking (i.e., business-as-usual) with the one exception that each study I run has a larger N than it would have if I were not thinking (more) intelligently about statistical power.

How has my lab been fairing with respect to this goal of collecting large samples? See for yourself:

Read More->

why i study self-knowledge – Simine Vazire (sometimes i'm wrong)

someone recently asked my why i do what i do.  it's easy to come up with just-so stories. for all i know, i could just as easily have ended up inventing new ice cream flavors for tara's ice cream (i still haven't ruled it out).  and if i had, i could probably make up a story about why it was always meant to be.  narratives are deceptively easy to construct, and amazingly convincing after the fact. it would be especially ironic for a self-knowledge researcher to deceive herself about why she studies self-knowledge.  so i mostly try to resist giving an explanation.  but then there's this:

Note

this is a note from my best friend, written when we were 15. (i have honored her request to remain anonymous -- she undoubtedly has much more embarrasing material on me.)

a few excerpts:

'at miriam's birthday party, i got the idea that you wanted an honest evaluation from me about you.'

Continue reading

the self-deception alarm system – Simine Vazire (sometimes i'm wrong)

 

Montaigne

'those who venture to criticize us perform a remarkable act of friendship,
for to undertake to wound and offend a man for his own good
is to have a healthy love for him.'

-michel de montaigne*

(all this time, reviewers were just expressing their healthy love for me!)

today i am going to talk about self-deception. don't worry, i will connect it back to scientific integrity.

being a researcher who studies self-knowledge and self-deception is a little nerve-wracking.  watching other people delude themselves and be entirely convinced by their self-deception, you start to wonder whether all of your own self-beliefs might not also be delusional. it can make a person paranoid. so i started wondering, could there be an internal marker of self-deception? a red flag that, if trained, one could detect and catch oneself in the act of self-deception? Continue reading

buckets of tears – Simine Vazire (sometimes i'm wrong)

Bucket04

i learned a new word the other day. bucketing. it almost made me cry.

one of the most common mistakes i see when reviewing papers is authors who take a continuous variable and, for no good reason, mutilate it by turning it into a categorical variable. our old friend the median split is one example.  (whose idea was it to befriend the median split? and why won’t he stop harassing us?)

bucketing, from what i can tell, is another such technique.  i had a hard time finding a definition, but i think it’s basically creating categories out of multiple response options and grouping the data that way. for example, you can turn the continuous variable ‘age’ into a categorical variable by categorizing people into age ‘buckets’ (e.g, 20-29, 30-39, etc.). Continue reading

Why Do We Take Personality Tests? – Kate Reilly Thorson (Psych Your Mind)

I often get questions from friends and family that they would like answered in a post. This month, my post is inspired by a question from my grandmother. Kudos to my grandma for asking a question about a popular trend on the internet!


Personality tests
Personality tests are not new, but they have recently skyrocketed in popularity on the internet. This week, Buzzfeed published 15 such tests in one 24-hour period. It seems every day on my Facebook news feed, someone has posted new results from one of these quizzes. Online personality tests have expanded beyond the traditional format of telling us certain traits we possess, although those do still exist (try here and here). Now, there are also tests that give us information about ourselves by comparing us to people or characters we know (“Which pop star should you party with?” or “Which children’s book character are you?”) and by comparing specific behaviors or knowledge to others’ (“How many classic horror films have you seen? Continue reading

life after bem – Simine Vazire (sometimes i'm wrong)

Bear_longing

many people have written about how bem's esp paper was one of the major factors that triggered the latest scientific integrity movement in social/personality psychology.  that is an interesting story.  but that is not the bem paper i want to talk about today.

i have come here today to discuss bem's chapter, 'writing the empirical journal article' (2003) that i - and i suspect many others - used to assign to every undergrad and graduate student taking our psychology research methods classes.  there are many extremely wise points in that chapter.  but there are also many pieces of advice that seem entirely antiquated today.  if the bem chapter is no longer the gold standard for how to write an empirical article, what is? (see also: laura king's article for the spsp dialogue (pdf, p. 6)).

i was reminded of the complexity of this question when a historian friend of mine suggested i read 'the question of narrative in contemporary historical theory' by hayden white (1984). i will share a few quotes with you:

'but it is precisely because the narrative mode of representation is so natural to human consciousness, so much an aspect of everyday speech and ordinary discourse, that its use in any field of study aspiring to the status of a science must be suspect. for whatever else a science may be, it is also a practice which must be as critical about the way it describes its objects of study as it is about the way it explains their structures and processes.'

'a discipline that produces narrative accounts of its subject matter as an end in itself seems methodologically unsound; one that investigates its data in the interest of telling a story about them appears theoretically deficient. Continue reading

Things that make me skeptical… – Brent Donnellan (The Trait-State Continuum)

Simine Vazire crafted a thought provoking blog post about how some in the field respond to counter-intuitive findings.  One common reaction among critics of this kind of research is to claim that the results are unbelievable.   This reaction seems to fit with the maxim that extraordinary claims should require extraordinary evidence (AKA the Sagan doctrine).  For example, the standard of evidence needed to support the claim that a high-calorie/low nutrient diet coupled with a sedentary life style is negatively associated with morbidity might be different than the standard of proof needed to support the claim that attending class is positively associated with exam performance.  One claim seems far more extraordinary than the other.  Put another way: Prior subjective beliefs about the truthiness of these claims might differ and thus the research evidence needed to modify these pre-existing beliefs should be different.

I like the Sagan doctrine but I think we can all appreciate the difficulties that arise when trying to determine standards of evidence needed to justify a particular research claim.  There are no easy answers except for the tried and true response that all scientific claims should be thoroughly evaluated by multiple teams using strong methods and multiple operational definitions of the underlying constructs.  But this is a “long term” perspective and provides little guidance when trying to interpret any single study or package of studies.  Except that it does, sort of.  A long term perspective means that most findings should be viewed with a big grain of salt, at least initially.  Skepticism is a virtue (and I think this is one of the overarching themes of Simine’s blog posts thus far).   However, skepticism does not preclude publication and even some initial excitement about an idea.  It simply precludes making bold and definitive statements based on initial results with unknown generality. Continue reading