An innovative research network, co-led by McGill psychology professor Jessica Flake, is tackling ingrained issues of reproducibility, publication bias and a lack of diversity in psychology research with a systematic approach for obtaining data on a global scale.
The Psychological Science Accelerator (PSA) brings together hundreds of psychology researchers around the world to conduct coordinated studies across every populated region on earth.
“The data we typically collect in psychology tends to be from what they call WEIRD demographics – Western, educated, industrialized, rich, democratic countries. The PSA aims to address that shortcoming,” Flake says.
While there is precedent for large-scale research collaborations performing one-off studies in psychology, the PSA goes a step further in being a global standing network to which any researcher can submit a study proposal.
A CERN for psychology
The network began to take shape in 2017 when Chris Chartier, a psychology professor at Ashland University in Ohio and now director of the PSA, sought to answer the call for what some in the field described as a ‘CERN for psychological science’. The PSA was conceived as both a response to the reproducibility crisis plaguing psychology research and a tool to test new hypotheses using large, geographically diverse sample populations.
Two years on, the initiative will soon publish the results of its first study, a re-examination of an influential theory on how we make social judgements about other people’s faces. Making good on its ambition to introduce greater diversity to psychology research, the PSA replicated the methodology underpinning Oosterhof and Todorov’s valence-dominance model across 11 world regions, 41 countries, and 11,481 participants.
Study reveals greater complexity in face perception
According to Oosterhof and Todorov’s theory, two key factors are at play in the first impression we form of another person’s face: valence (how trustworthy the person appears) and dominance (how physically strong they appear). The PSA study sought to determine whether these two factors were as important in other cultures as they appear to be in the West.
While the findings suggest the valence-dominance model is broadly applicable across all the regions studied – spanning the Americas, Europe, Africa, the Middle East, Asia and Australia – additional analytical methods used in the PSA study revealed a greater deal of complexity than the original two-part model might suggest.
“When you open the lid a little bit, you can start to see some other complexities,” Flake explains. “More factors show up, and the relationship between the indicators that make up those factors is more complicated than the original theory.
“There were a few cultures where the trait ratings ‘emotional’ and ‘weird’ were correlating in a way we hadn’t seen before. I think there’s more research to be done to dig in at the region level to understand how, when some cultures look at faces, they might focus on certain things differently.”
Rigorous methodology and review-first publication model
Flake was instrumental in bringing this kind of additional analytical power to the study. A specialist in measurement and quantitative methodology, her involvement in the PSA first came about through exchanges with Chartier, in which she emphasized the importance of involving methodology experts in studies conducted on such a large scale. These discussions eventually led to the creation of a data and methods committee within the PSA, which Flake now oversees as an assistant director for methods, alongside Patrick Forscher, the assistant director for data.
The PSA’s early focus on methodology, placing as much if not more emphasis on how data is gathered and analyzed as on identifying which research questions to investigate, is complemented by the network’s preferred publication model, known as the registered report process.
“The registered report format is a direct response to publication bias and the inflation of false positive results in the literature because people are motivated to get a significant result,” Flake explains. “The idea is that you get reviewed on the idea and the proposed methodology and then you revise and resubmit just like you do with a regular paper.
“But then if the decision is acceptance, it’s an in-principle acceptance, which means that if you adhere to the protocol, your paper can’t be rejected because of the results.”
While Flake says the PSA is still a work in progress, the network’s fundamental principles and the democratic processes by which it selects which studies to pursue have been drawing a positive response from the psychology research community. In a significant endorsement of the rigour and scale of the network’s approach, Alexander Todorov, the Princeton University psychology professor who co-authored the seminal paper on the valence-dominance model, himself signed on as a reviewer for the PSA’s re-evaluation of his theory.
Language barriers
Language can be a significant hurdle in conducting psychology studies on a global scale. The PSA’s most recent study, ‘Moral thinking across the world: Exploring the influence of personal force and intention in moral dilemma judgements’, deals with decisions people make when faced with a moral dilemma. The researchers will rely on a multi-question survey to investigate whether various psychological and situational factors have a consistent impact on this kind of decision-making across different cultures.
Flake has been successful in obtaining a $50,000 grant from McGill’s Faculty of Science through the Wares Science Innovation Prospectors Fund to support the extensive work involved in translating the surveys for this study and ensuring their consistency and reliability across languages.
Flake says the money will go a long way towards supporting the translation work that has largely been a volunteer effort so far. She describes a heartwarming moment as her PSA colleagues around the world expressed their appreciation through the network’s online chat system: “They were over the moon. The whole Slack channel lit up with emojis that continued to trickle in as the news made its way across time zones.”
Taking risks for better science
The support from the Wares fund is all the more meaningful for the fact that the PSA’s approach to research is an unconventional one. Flake acknowledges that she and her colleagues are investing their time and energy in a venture that does not resemble a traditional pathway towards academic career progression, where an individual’s published output tends to count for more than service work in support of collaborative research efforts.
The McGill alumnus behind the Science Innovation Prospectors Fund, Bob Wares, is himself no stranger to high-risk, high-reward ventures, having made his fortune in the mining industry. Bruce Lennox, Dean of Science at McGill, describes Wares’s motivation for establishing the fund in the following terms:
“He has set out to fund researchers – professors and their students – to try out bold, inspiring, difficult ideas – the types of ideas that if you go out in the public space for funding, they say no.
“It’s really inspiring for someone who has been a prospector to allow our researchers to also experience the exhilaration of being a prospector.”
