This article in Aesthetics for Birds has some interesting statistics on the percentage of papers authored or co-authored by women and minorities in the top print aesthetics journals: Journal of Aesthetics and Art Criticism and British Journal of Aesthetics. About 20% of articles in these journals are written by women in the period from 2010 onwards. When we look at memberships of professional aesthetics organizations, the percentage of female aestheticians is about 32%. So that means women are underrepresented in JAAC and BJA. What can account for this disparity? JAAC keeps a record of gender and geographic location of submissions.

Sherri Irvin finds "It is notable that over the past three years, women authors have submitted to JAAC at a rate substantially higher than the rate at which they are published in JAAC from 2010-2014, and closer to the proportion of women members in the ASA. During 2 of the last 3 years, the acceptance rate for women has been lower than for men. Though the differences seem small (only 2-3 percentage points), another way of putting them is that in 2012-3, men were 21.4% more likely than women to have their manuscripts accepted, while in 2013-4, they were 11.6% more likely." She also writes "US submissions tend to be accepted at a rate slightly over 20%, while submissions from non-English-speaking countries tend to be accepted at far lower rates".

JAAC practices double-anonymous refereeing. I am in the statistics, since I've co-authored an article that was published in JAAC in 2011. My co-author and I were very pleased with thoroughness of our reviewer, who is one of the few experts on the aesthetics of paleolithic art. We could guess who he was, and it turned out (as he later communicated with us) he also had an inkling as to who we were. Aesthetics is a small world. The only time I reviewed for JAAC I didn't know who the author was, so I believe I reached a verdict that was unsullied by considerations of the author's identity. But was it? Thoughts about the identity of an author can play a role in one's decision, even if you don't want to, this is after all how implicit bias works.

There are several clues in a text that might elicit implicit biases. For instance, citation practices: in my experience, early-career philosophers tend to be more thorough in their review and citation of sources than more senior people (I'm not aware of statistics on this, though, so it might be a wrong impression). Also, the tone of a mature author is something that is very hard to emulate – as Marcus Arvan writes, it is hard even to convey what it is, but there is something about the writing style of confident, established, native English-speaking authors that reviewers unconsciously pick up on and respond positively to. People who aren't native speakers find it even harder to write in this way. We are more likely to respond positively to an engagingly-written piece, even if the points it makes are rather banal.

The author might give clues about the person they are in other ways. For instance, giving examples about the geographic location of buildings in New York city, or aspects of American culture might signal that the author is American (e.g., using a Pontiac or the MoMa in thought experiments). The author might use breastfeeding or other female gendered examples, indicating they are female. JAAC only had 22.4% female reviewers.

Sherri Irvin's piece revealed that people outside the English-speaking world have a much harder time to get their papers accepted at JAAC [update: the overall acceptance rate of people from non-English speaking countries over 3 years of 3.5%]. This is obviously a big problem. Some journals offer proofing services, and I know people in France and The Netherlands who employ professional proofreaders, but both are in senior positions and can afford it. However, it seems rather harsh to impose such a cost on non-native speakers with small budgets.

So often we can at least hazard a guess as to who the author is, especially if the field is small. And even if we don't, there are so many clues in the style of the piece, citation patterns, examples, that thoughts about the author's identity unwittingly come to mind. In the Philosophers' Cocoon, I recently wrote about my personal reviewing practices, and in point (3) I raise this point: What strategies can we, as reviewers, employ to mitigate implicit biases? Just thinking that considerations about an author's geographic location, etc. could impact one's decision is a start, but I don't think it's enough.

Posted in , ,

21 responses to “Anonymous reviewing is not enough to counter implicit bias, so what can we do to mitigate it?”

  1. Sherri Irvin Avatar

    Hi Helen,
    Thanks for these comments! I agree that “anonymous” refereeing is far from sufficient to eliminate implicit bias for the reasons you mention.
    I just wanted to point out that in the second table in my piece, there are specific numbers and percentages of authors who submit to JAAC from the US, from other English-speaking countries, and from non-native English speaking countries. There are also acceptance rates, which over the past 3 years have ranged from 0% to 6.7% for authors from non-native English speaking countries, with an overall acceptance rate over 3 years of 3.5%, which is depressingly low.

    Like

  2. Helen De Cruz Avatar

    OK – thanks. I did not see it in the text and missed the table. I’ll update this in my piece. Indeed, depressingly low, especially given that aesthetic judgments seem to have some cross-cultural variability.

    Like

  3. Crimlaw Avatar
    Crimlaw

    How much of the difference in the acceptance rates is explained by implicit bias? Implicit bias is presumably only one of many possible sources of the difference. You can’t cover everything in one post but is there some other place where implicit bias is shown or at least argued to be a significant factor in this case? Both the title and the content of the post suggest that there is.
    Whether implicit bias is a major factor in this instance or not, I agree that the general issue of how to minimize the possible effect of implicit biases in manuscript review deserves attention and appreciate your reflections about this in the link you provided.

    Like

  4. Helen De Cruz Avatar

    Hi Crimlaw, I think it’s at present unclear what explains the lower acceptance rates of women and non-Americans/non-English speakers in the sample of JAAC and BJA. We have some data on what happens when an identical abstract is submitted with a male or a female name on it, or a grant application. But I have no idea how much of a role this plays when papers are anonymized. We’d need to know in how far reviewers are aware of the identity of the author. In a small field like aesthetics, it’s not unlikely you know who the author is (e.g. you saw them present the paper). Perhaps the subject, the examples that are used, might also provide clues.
    It’s also hard to judge to what extent the perceived senior or junior status of an author plays a role. I’m inclined to think, a lot. In my experience as a referee (if my impression is correct), junior authors tend to anonymize as “omitted for review”, whereas senior authors don’t hide who they are. They just put the pubs in brackets (see Jones 1991, 2002, 2005), and it’s not hard to guess that they are Jones.
    So I think that we do get all sorts of implicit ideas about who the author is, and those ideas color our perception of the work. One way to control for it would be to vary some linguistic markers (e.g., people who are lower-status use personal pronouns more; non-native speakers of English use less idiomatic expressions, more experienced authors cite less sources) in an otherwise identical text, and examine how referees evaluate them.

    Like

  5. Tom Hurka Avatar
    Tom Hurka

    I find the idea of “tone” interesting but think it can mean something else. One thing a mature philosophical writer knows is how much space to give the different parts of their argument — what needs going through carefully and what doesn’t and would be a digression to elaborate on. That’s something grad student writing often hasn’t learned, so there’s too much on minutiae and not enough on the crux of the argument. (It’s not just grad students. I just read a book by a well-known senior philosopher that had this flaw in spades.) And I don’t think it’s wrong for reviewers to be influenced by this kind of “tone.” It shows an author who fully understands, or is fully in control of, their material, and that’s a relevant fact. On the other side, an author who spends more time on less important issues doesn’t inspire confidence.
    It took me a while to learn whatever I have of writing with this “tone,” and I think others need time too. Which leads me to ask whether there’s demographic information about submitters to the aesthetics journals, i.e. about their ages and seniority. If — as I hope is the case — the hiring of women has been increasing in recent years, the women members of the aesthetics organizations may be clustered somewhat disproportionately in the junior ranks, and that may explain some — I’m not for a minute suggesting all — of the gap between their percentage membership in the associations and their percentage of acceptances in the journals.

    Like

  6. Allen Hazen Avatar
    Allen Hazen

    One of the minor pleasures of being an anonymous referee (the big pleasure, of course, being when you discover, and are maybe able to make a tiny, stylistic, improvement in, a brilliant and important paper by someone you’ve never heard of!) is trying to guess, on the basis of internal evidence, who the author is. High point: deciding that a paper is probably by a not-too-talented student of Professor X, and then finding out (editors often reveal the author’s identity after you’ve turned in your report) that it was by Professor X in person.
    Not sure what the relevance of this is to the (important) point of the blog post: maybe that referee’s are human, and that mental processes that could lead to bias are probably unavoidable.

    Like

  7. Helen De Cruz Avatar

    It would be great to have more refined information about the demographics (I’m already pretty impressed with Irvin’s stats now). I do not think the correct tone is irrelevant for refereeing. As I’ve commented on Marcus Arvan’s piece (to which I link in the original post), tone is, I believe, an honest because hard-to-fake signal of maturity and expertise. Knowing how much to contextualize, letting your argument breathe, dealing with objections, are all relevant things. I just sometimes worry that a confidently written piece that hits the right tone might get published even if the ideas are rather banal, and that a really good idea written by a grad student who doesn’t yet master the tone (e.g., too much time spent on unimportant ideas) can get a reject decision and thus we miss out. What I’m worried about is that the tone can play too much of a role in our decisions as referees and editors, even if it correlates significantly with the expertise of the author.

    Like

  8. John Schwenkler Avatar

    In addition to implicit bias, another thing that could explain some of these differences is stereotype threat (http://www.apa.org/research/action/stereotype.aspx), which could lead women and others in underrepresented groups simply to do poorer work. Tamar Gendler has a really nice forthcoming paper on this subject, though not in connection with philosophy specifically.

    Like

  9. Tom Avatar
    Tom

    “editors often reveal the author’s identity after you’ve turned in your report”
    I find this shocking. Can others confirm that this is common practice? What journals are known to do this?

    Like

  10. Helen De Cruz Avatar

    John: I’ve been thinking about the effects of this too. If women and other minorities are submitting in roughly the same proportion or similar proportion as they are represented in the profession (as the AFB post suggests, and my earlier poll), but are accepted at significantly lower percentages, then there are several possibilities, none of which mutually exclusive
    1. Women and minorities are a lot more junior and after a while, they will catch up
    2. Women and minorities are subject to implicit biases that anonymous peer review cannot address (either b/c reviewers pick up unconsciously on various cues, or because reviewing isn’t always anonymous, or because the editor knows the identity and plays a non-trivial decision in acceptance/rejection
    3. Women and minorities, for various reasons, turn in work that doesn’t conform to our philosophical expectations (e.g., Buckwalter and Stich’s paper on differing intuitions across men and women, work by Machery, Nichols etc on differing intuitions across ethnicities)
    4. Women and minorities, for various reasons, submit work of poorer quality – I think we need to be careful in further looking at 1-3 before resorting to 4 as an explanation. Tentatively, I did not find a difference in my study on submissions to top-5 journals in reasons for why people didn’t submit, something one would expect if stereotype threat were at work (e.g., the reason “My work doesn’t fit in these journals” might come up more frequently in underrepresented groups.) That didn’t seem to be the case. I ran various tests, all came up with null results.

    Like

  11. Eric Winsberg Avatar
    Eric Winsberg

    Another possibility here is that the data are just noise. I haven’t thought about whether they look statistically significant. But even if they are, (for this particular set of journals) that could just be a result of data fishing.
    In general though, I pretty reluctant, myself, to talk about implicit bias among reviewers who are looking at anonymized papers. In the first place, insofar as we are talking about journals that are only double blind, not triple blind, I’m much more likely to hypothesize that the it is the journal editors that are subject to the bias–because they actually know the gender, geography, rough age, instituational affiliation, etc. of the author. And they could easily be sending work by women to the people they know to be the more skeptical reviewers. Who know? Maybe women make easier reviewers and editors tend not to send papers by women TO women because they subconsciously think that’s intrinsically suspicious. But secondly, the mechanism you propose doesnt make much sense to me. If the way that I determine that this paper is by, e.g., Helen De Cruz is that I know her and I know her work, than my particular opinions about her are bound to trump whatever biases I have about Dutch women. In any case, whether what I just said is true or not, I certainly don’t think there is any scientific evidence to show that implicit bias works in the way you are suggesting it does. All the studies I know involve using the exact same material and then giving it generic male and female sounding names. I don’t really see how those results extend to the kind of mechanism you are talking about–but they extend much more easily to my hypothesis regarding the absence of triple blinding.
    finally: triple blinding such a freakin’ obvious thing that we ought to be doing!

    Like

  12. David Wallace Avatar
    David Wallace

    It’s noise – or at least, there’s no statistical reason to doubt that it’s noise, unless I’m getting something badly wrong.
    Over the three years, there were 446 papers submitted by men, of which 70 were accepted, for an acceptance rate of 15.7%. If we assume that each paper submitted by a woman had that probability of being accepted (and that there are no correlations between acceptances; being technical, if we assume a Poisson distribution) then the expected number of acceptances of women’s papers would be 27.2 and the standard deviation would be 4.8.
    The actual number of papers by women accepted in that period was 25. That’s less than half a standard deviation lower than the expected level. The probability of women’s acceptance rate being at least that different (in one direction or another) from the expected value is about 2/3 (using the normal approximation to the Poisson distribution).

    Like

  13. David Wallace Avatar
    David Wallace

    Just for interest, here’s a toy Bayesian version. Suppose there are two hypotheses:
    (NoDiff) the probability of a woman’s paper being accepted is 0.157, the same as the average acceptance rate for men;
    (Diff) the probability of a woman’s paper being accepted is 0.145, the actual average acceptance rate for women.
    Let E be “25 papers by women were accepted”. The Poisson distribution gives
    Pr(E|NoDiff)= 0.0728
    Pr(E|Diff) = 0.0795
    Taking (say) Pr(NoDiff)=Pr(Diff)=0.5, we’d get
    Pr(Diff|E)=0.522
    So – of course – this should increase your confidence that there’s some kind of systematic bias, but only by a tiny amount.

    Like

  14. Eric Winsberg Avatar
    Eric Winsberg

    Thanks for running those numbers. I didn’t even look at the raw data, but my gut feeling was that the kinds of numbers involved wouldn’t support much confidence.

    Like

  15. Christy Mag Uidhir Avatar

    I’m with Eric here. There’s simply no evidence that suggests implicit bias triggers sensitive to anything other than explicit and robust gender cues let alone unconscious indirect subtleties such as authorial tone. So for implicit bias to play any role at this stage requires some pretty substantial professional failures on the part of referees and editors (referees actively seeking to learn the author’s identity, inconsistent double anon review, not moving to triple anon, etc.). Of course, to the extent that’s the case is the extent to which it’s no longer specifically a problem of gender bias but one of bias run amok where solving for the latter effectively solves for the former.
    In fact, I think placing so much focus on implicit gender bias at the journal stage runs the risk of ignoring or just pushing aside the far more insidious sorts of problems occurring at the undergrad/grad/early career stage (ones I think much better suited to explain those problems seen to occur at the journal submission stage). For example, from the simple presence of non-negligible gender-based gaps in graduate advising, early career mentorship, and peer-networking we should expect there to be gender-based gaps at the journal submission/acceptance stage because the training, skill sets, and support groups crucial for productivity are precisely what such relationships tend to promote.
    This suggests what I think many folks are reluctant to admit: namely, that at least superficially, what explains the submission/acceptance gap between men and women is perhaps nothing more than an ordinary gap in productivity/work-quality. Of course, what explains that productivity/quality gap is an earlier gender-based resource gap in which the ready availability of and easy access to invaluable resources like advising, mentoring, networking, etc. systematically enjoyed by men are not similarly afforded to women.

    Like

  16. Filippo Contesi Avatar

    For non-native speakers the obvious way seems to me to be anonymization with labelling. Something like this is already done with students’ essay marking in the case of dyslexia and other conditions/disabilities. There will be problems with certifying the information provided by the submitting author with respect to their labelling in this respect, but there may well be ways to overcome these problems.

    Like

  17. Helen De Cruz Avatar

    Hi everyone: Thanks, I’ve just done 2×2 contingency tables, and based on the 3 year period, the percentage of women who get papers accepted at JAAC isn’t significantly lower (it’s, averaged over the 3-year period, 16% for men and 14% for women, NS). It would be good to have more data on this (e.g., if there were a small gap between acceptance rates between genders, but it holds true across many years, it could turn out to be significant. 3 years is a relatively short period).
    I agree differences in mentorship and other features in early career philosophers needs to be addressed. But regardless, we need to move to triple reviewing, and editors might need to emphasize that reviewers should not attempt to find out the identity of the authors they review (there is anecdotal, albeit not systematic evidence this is happening).
    For the acceptance of non-English speakers, averaged over three years, acceptance is substantially lower (p < .0001) than for Americans and other English speakers. This is an enormous difference! – e.g., American average acceptance rates are 22%, non-English speakers, 3% . I’ve been wondering what could account for this disparity. It could, in part, be that the training and mentorship at American and other English-language departments is superior, so that people at these departments turn out better work. But it might also be that reviewers are picking up on the style of the prose. Since quality of writing isn’t irrelevant for a paper’s overall quality, I don’t know the solution here.
    The online journal Contemporary Aesthetics, mentioned in Sherri Irvin’s analysis, has an editorial policy of encouraging submission of such work, “specifically tries very hard … to encourage works from non-English speaking authors whose language may need a lot of editorial work. As long as the content is there, we do not automatically reject submissions because of poor English. [Berleant] spends an inordinate amount of time and effort working with the authors to improve their writing. I think this adds to the cultural diversity of our collection.” –

    Like

  18. Tom Avatar
    Tom

    I repeat being shocked at the above comment that “editors often reveal the author’s identity after you’ve turned in your report”. To my mind, that seems like a breach of confidence and trust: journal editors ought not reveal the author’s identity to referees without the author’s consent. I have naively relied on this being standard practice – am I wrong? Do journal editors frequently reveal the author’s identity after the referee has turned in her/his report?

    Like

  19. Filippo Contesi Avatar

    (1) From Susan Feagin’s “Reflections on Being Editor of JAAC, 2003-2013” ( http://www.aesthetics-online.org/articles/index.php?articles_id=68 ): “Roughly 17% of submissions in 2011-2012 were from countries outside the English-speaking world or Europe — a passable percentage, I suppose, but we did not accept a single one. The challenge is to sustain JAAC’s main mission as to provide a venue for Anglo-American aesthetics and philosophy of art, while being sufficiently ecumenical about what else it can do that will be of philosophical interest to its past and future readership.
    Current statistics involving women are in some ways a little more encouraging, in some ways not. Only 24% of submissions during 2011-2012 were from women, but the acceptance rate was almost exactly that as for men, about 11%.”
    (2) Although double blind refereeing may be fine, and triple blind refereeing may even improve things WRT gender, ethnicity and other minorities, the situation WRT non-native speakers of English is much harder to address. One way to address it is anonymization with labelling, of the kind that is done in UK university for student marking and conditions such as dyslexia. Self-certification from submitting authors of their status as non-native speakers may well not be enough to solve all technical problems with this fix, but I am confident that a good enough solution can be found for such technical problems.
    (3) Of course, a lot else does and would still remain to be done to make things ideal both WRT to gender and ethnicity and WRT to non-native speakers minorities, as well as WRT to lots if not all other minorities, including disability/ies and class. This includes things such as fair access to mentoring etc., the broadening of stylistic requirements for what is considered as good, publishable philosophy, etc.

    Like

  20. Helen De Cruz Avatar

    Dear Tom: I didn’t hear about this before. I’ve reviewed for over 20 journals, most in philosophy, but also in cognitive science, archaeology, anthropology and religious studies, and never did I encounter this practice except once: After turning in a revise & resubmit recommendation, the editor wrote his letter to the author and sent it to the 2 reviewers (of which I was once). The letter was addressed “Dear Dr. xxx” – xxx being the name of the author, so I knew who he was at that point. It didn’t seem like a good idea to reveal his identity at this point, given that the decision was R&R. But as far as I know, this isn’t general policy, and it wasn’t a philosophy journal. I’ve refereed for several journals (e.g., BBS, Current Anthropology, Cambridge Archaeological Journal) that don’t practice anonymous reviewing – in my experience, philosophy is actually quite good in that respect to several other humanities disciplines.

    Like

  21. Neil Avatar
    Neil

    Cognitive science journals often reveal the names of the authors with the invitation to review. Double blind may be unacceptable, but many journals continue to practice single blind reviewing.

    Like

Leave a reply to Filippo Contesi Cancel reply