'Bropen Science'? What Should Critiques of Emerging Psychological Data Look Like?
A recent controversy about data reporting has led to renewed claims that open science movements are exclusionary.
Over the past couple of weeks, #psychtwitter (the specific part of the Twittersphere dedicated to discussing psychological research) has been ablaze with talk about ‘pile-ons’ and critiques of papers, and how respectful criticism can be achieved.
This most recent situation was sparked by science critic Nick Brown who, as he regularly does on the platform, tweeted a screenshot of a results section containing multiple marginally significant results alongside terse criticism.
The critique was about a recent paper published in the high-impact British Journal of Social Psychology, which was lead-authored by PhD student Roxanne Felig.
What follows is a description of the affair, and how it contributes to concerns and debates about the open science movement.
The Paper
Felig’s paper was co-authored with several of her colleagues and centers around the question of whether self-objectification among women can have physical effects in terms of their not noticing conditions that should cause pain or discomfort. Specifically, the researchers investigated how cold women felt when wearing different levels of clothing, and whether any effect of ‘clothes coverage’ was driven by self-objectification.
A free version of the paper is available here.
A major strength of the work was its naturalistic approach. Instead of the usual dry social psychological approach of asking people to rate agreement levels or give judgments on self-report questionnaires, the researchers got out into the street to test people ‘in the wild’.
In a busy area of an American city known for its nightlife, they managed to recruit 185 women to take part in their study. Participation entailed women posing for photographs of their outfits, and completing measures of both self-objectification (i.e., the importance they placed on their own physical appearance) and their current warmth. The researchers also logged the actual temperature at the time of data collection, and asked participants about their alcohol intake during the night. These latter variables were controlled for in the data analysis because they could affect how warm or cold participants felt.
The results were interesting, and supported the hypothesis that women who self-objectify are less prone to ‘feeling the cold’. That is, there was no relationship between ‘feeling cold’ and the amount of clothing being warm by those participants who reported self-objectifying to a higher degree. However, those who did not self-objectify felt colder when they were wearing less covering clothes, and warmer when they were showing less skin.
It other words, feeling like you looked ‘hot’ stopped women feeling cold, even when they were exposing more skin.
The Critiques
Felig and colleagues’ paper received a lot of attention, both in mainstream news outlets and on social media platforms, such as TikTok. The heightened profile led some members of the scientific community to dig a little deeper into the findings.
The most prominent example of this was Nick Brown’s tweet that is linked above. In this, he was perceived as mocking the premise of that study (that “looking hot” keeps women feeling warm), criticizing the data (by highlighting the marginal significance levels), and by encouraging a ‘pile on’ by tagging well-followed colleagues.
Brown’s principle complaint about the paper was the repeated reporting of borderline-significant results in support of the author’s main hypotheses. Without going into statistical details, tests that yield a p-value of less than 0.05 are said to demonstrate a real effect. Felig’s work was peppered with p-values of 0.02-0.05 which, although sometimes ‘significant’, appeared suspicious to some commenters.
This led to a lot of attention on Twitter. The team led by Felig had engaged in some open science practices, which usually indicate good research processes. In this case, the team had made all of their data available, which allowed some of those expressing suspicion to download the numbers and perform some re-analyses. The most comprehensive of these was conducted by Toronto-based psychology professor Yoel Inbar.
To begin with the good news, Inbar was able to replicate the analyses reported by Felig’s team in their paper. This suggests that no errors were made in the reporting of the major findings. However, these were the only circumstances under which he could find ‘significant’ findings. Under multiple differ scenarios, such as retaining all participants or using a range of different data exclusion rules, no other sample configuration was able to reach a conclusion that supported the researchers’ hypotheses.
This might indicate that the data were ‘p-hacked’, meaning that the researchers explored their data to see what was there, and then reported the version of the analysis that best fit the narrative of the paper.
This, if it was actually done (there is no conclusive evidence of this) would relate to ‘researcher degrees of freedom’, whereby researchers and research teams make a set of decisions that impact the results that end up being reported. One way of getting around accusations such as these is to ‘pre-register’ your analysis, which means setting out how data exclusions will take place and what analyses will be run before you actually run them. In doing so, you safeguard against accusations of mishandling data, or of selecting the results that tell a particular story. Felig’s team did not do this, which opened the door to accusations of such cherry-picking.
'Bropen Science' and Gendered Interpretations of Critique
As alluded to earlier, this particular case set #psychtwitter ablaze.
On the one side we saw relatively high-profile open science advocates interrogating open data in response to highly-publicized marginal results that supported and important theory that, in spite of its serious nature, looks ‘fun’ on its face. On the other side we saw push-back against such behaviors from a lot of people who saw the attention directed towards a young female researcher who, while still conducting her PhD, was engaging in some open science practices.
This led to the re-emergence of a phrase that has been used for a little while to describe this form of critique — 'bropen science'.
At its core, the premise behind this label is as follows:
Within the open science movement a bro will often be condescending, forthright, aggressive, overpowering, and lacking kindness and self-awareness (Reagle, 2013). Although they solicit debate on important issues, they tend to resist descriptions of the complexities, nuances, and multiple perspectives on their argument. They often veer into antisocial patterns of dialogue, such as sealioning, the act of intruding on and trying to derail a conversation with disingenuous questions (Kirkham, 2017). You’ve interacted with a bro if you’ve ever had the feeling that what they’re saying makes sense superficially, but would be hard to implement in your own research practices. In general, bros find it hard to understand – or accept – that others will have a different lived experience.
As can be seen here, there does appear to be an attempt to play the man, but not the ball (pun half-intended), and of course there is nothing in the (relatively clear) open science movement that is impenetrable to most who are engaged in research.
In fact, entire infrastructures and open resources have been produced to assist in reducing almost all barriers to entry.
Of course there are some people within the ‘open science community’ who are obnoxious, just as there are in any walk of life. Some of the criticism of the study published by Felig and colleagues followed this pattern, with scorn being directed not towards the study’s design (which was actually quite nice!). This initial approach meant that when critiques of the scientific choices and open science processes took place, this emerged with the appearance of a ‘pile-on’ ostensibly fueled by an undercurrent of misogyny. This has been met with advice on how to engage courteously, with one blog post suggesting:
Reaching out to authors who you might disagree with privately, to give them the opportunity to correct themselves or clarify things without publicity
Support and show solidarity with those facing scrutiny, and ask influential others to speak out in support of those whose work is under the proverbial microscope
Contributing to the shifting of power dynamics by amplifying contrary voices and protecting those being scrutinized from ‘senior’ voices within the open science movement
Leaving social media sites such as Twitter completely, and moving academic discussions to more appropriate mediums
There are obvious downsides, though, of viewing social critiques of research through dichotomous lens of ‘good / evil’ or ‘oppressor / oppressed’. One might even ask whether there is much of a point in posting open data (as Felig and colleagues did) if there is an expectation (or social pressure) to not interrogate this. Open science is about transparency. About checks. About confirmation of scientific claims.
What we need are social norms and expectations about how we engage with empirical research. We need to tap back into the Mertonian norms of organized skepticism and communalism (working together to test each other’s claims), disinterestedness (placing value on truth for its own sake, rather than seeing critique as an evaluation of our personal ‘rightness’ or ‘wrongness’), and universalism (by breaking down traditional barriers to entry and subsequently scrutinizing everybody’s ideas to an equal extent).
Only by engaging in these norms can we break down the tribal shackles in which we currently find ourselves, as we seek to advance our science for the benefit of all in the field of psychology.