New APPS

Art, Politics, Philosophy, Science. Mostly Philosophy.

recent posts

about

    An open letter from Jonathan Ichikawa to Berit Brogaard regarding her invitation to be a PGR evaluator

    Posted on Facebook and reposted here with his consent and encouragement (if I understand correctly)

     

    Dear Berit,
    Thanks for this invitation. While the interest in my opinion about my colleagues is flattering, I don't feel able to participate, for three reasons.
     
    First, as I think you know, I have signed on to the 'September Statement', which is a commitment to declining to provide volunteer work for the PGR while under Brian Leiter's control. While I'm aware that he's appointed you as a co-editor and announced an intention to step down after this year's PGR, it is unclear to me how much control he retains in his position as co-editor. And since the reason his control was relevant in the first place was his social power as a result of the general perception of his control, and the current unclarity will allow that perception to continue, I don't consider it consistent with the September Statement to participate. (Other signatories may well make contrary decisions for themselves about this point.)
     
    Second, I remain deeply ambivalent about the PGR itself. While it unquestionably does good for many prospective philosophy students, it also, in my opinion, unquestionably does harm to the structure of the discipline of philosophy on the whole. I just haven't been able to make up my mind about whether I would support a PGR, even if totally divorced from Brian Leiter. I really don't know whether it would count as a service or as a harm to the profession; under such moral uncertainty, I'm inclined to decline to participate.
     
    Third, I very much doubt that I would be able to provide anything like reliable judgments of philosophical quality based on the names of individuals in faculties, without spending an enormous amount of time reading people's work. Although I've been in professional philosophy for nearly ten years, and have gained at least some familiarity with a large number of philosophers, I think I know next to nothing about a large majority of the philosophers whose departments you'd be asking me to rate. At a minimum, I might be able to find and skim the CV of every member of a department in half an hour or so, but to employ any of my own philosophical skill to give anything like an expert opinion, I'd have to read and engage with people's work. Since we're talking about roughly a hundred departments, this represents a daunting task to say the least. (Even just skimming CVs, at 30 minutes per department, would take some 50 hours.) Or I could skip most departments, limiting my attention to those containing my friends and those colleagues I've interacted with enough to have an opinion about already; but I'd worry about selection bias, and even those departments are made up mostly of people I don't know. I'm just not comfortable contributing a meaningful opinion about the quality of someone's work without spending orders of magnitude more time than I could offer if I wanted to.
     
    I'd be happy to discuss any of these considerations with you further if you're interested.
    Best wishes,
    Jonathan
    Posted in

    26 responses to “An open letter from Jonathan Ichikawa to Berit Brogaard regarding her invitation to be a PGR evaluator”

    1. Eric Winsberg Avatar
      Eric Winsberg

      My own view is that these are all excellent points and I certainly hope other prospective evaluators, (especially those who signed the September Statement, or the letter from the advisory board), will follow Professor Ichikawa’s lead.

      Like

      Reply
    2. Ed Kazarian Avatar
      Ed Kazarian

      What Eric Winsberg said @1, with added emphasis on the absolutely crucial importance of Professor Ichikawa’s third point, which seems to me to encapsulate everything that is almost necessarily wrong with the evaluations upon which the PGR is based.

      Like

      Reply
    3. Dylan Avatar
      Dylan

      Add to these excellent points the selection bias in who does and does not get invited to be an evaluator in the first place.

      Like

      Reply
    4. Christopher Gauker Avatar
      Christopher Gauker

      Echoing Ed Kazarian, thank you, Jonathan Ichikawa, for stressing this point: You don’t have enough first-hand acquaintance with the work of the people you would be evaluating to evaluate them competently. There cannot be many people who would be able to do that competently. What Leiter has said to this objection in the past is that an accumulation of evaluations from a lot of people with limited information can produce a reliable ranking. There is no reason to believe that, given that many of them may share significant biases. But in light of your own observation about the flaws in the method, I do not understand why you think that the PGR “unquestionably does good for many prospective graduate students”. How can a ranking that fails to do what it claims to do, namely, measure faculty quality, do anybody any good?

      Like

      Reply
    5. Jim Maffie Avatar
      Jim Maffie

      Well said, Chris.

      Like

      Reply
    6. Rebecca Kukla Avatar
      Rebecca Kukla

      I think that the PGR measures exactly what Brian has always claimed it measures, namely faculty reputation (as opposed to faculty quality, or quality of the grad program, both of which many people seem to take it as measuring or purporting to measure). What you can tell when you glance at people’s cv’s is whether they are shiny in the way that makes you a Big Deal in the field. When you rank on the basis of whether you’ve vaguely heard that someone is a Big Deal, you are picking up on that shininess, and I do think that for this, the accumulation of evaluations from a lot of people with limited information works pretty well. And the concern that selection bias enters into whose name we recognize seems to me to be not so much a bug as a design feature – yeah, reputation depends on many things other than quality.
      So in Brian’s defense, a lot of the criitiques of PGR seem to me to miss the mark a bit as they are complaining about it not measuring something he never claimed it measured. But on the other hand there is a huge question as to why THAT is a thing we want to measure.
      Interestingly, and separately, I was not asked to evaluate this year even though I have done it in the past. I figured that was because I signed the September Statement but apparently that is not the reason, given Jonathan’s letter. So now I am very curious who they decided to reinvite and how. Although I would have declined anyway, I wonder whether their principles for reinviting themselves embedded any selection biases.

      Like

      Reply
    7. anon Avatar
      anon

      To Rebecca Kukla:
      It is not exactly true that BL says the PGR measures only reputation. See for example the opening lines of his “What the Rankings Mean” part of the report:
      “The rankings are primarily measures of faculty quality and reputation. Faculty quality and reputation correlates quite well with job placement…”
      In other words, BL is putting “quality” (or how talented/good/insightful/etc. the philosophers are) in lockstep with “reputation” (or the shiny, “have you heard of this person or not?” quality you’re talking about).
      Indeed, putting the word “quality” first suggests that the PGR overall rankings primarily track quality!
      The implication that quality moves in lockstep with how well known you are in the field is one of the many drawbacks of the report.

      Like

      Reply
    8. Rebecca Kukla Avatar
      Rebecca Kukla

      OK that is fair, although in public discussions he has emphasized reputation, and he regularly describes it as a ‘reputational survey.’
      In any case, regardless of his intentions, I think faculty reputation is pretty much what it measures. But I don’t know why we would want to measure such a thing.
      I do think that the specialty rankings have been substantially more useful and track quality much more closely. I know that as an evaluator, when I did speciality rankings, more often than not I knew at least something about the work of the people I was evaluating. For the general rankings I mostly had to go on cv scans and shininess. I hope that if there is some sort of rebirth or reboot of a philosophy ranking system that it will focus on particular specialties, though there are problems there too (including that every possible way of dividing up specialties is going to embed problematic assumptions).

      Like

      Reply
    9. Michael Kremer Avatar
      Michael Kremer

      I agree with you (Rebecca) about the quality rankings; when I tried to do the evaluations I had roughly the experience described by Jonathan in his third point, and that is why I quit, but I think I could reasonably judge quality in my own fields of specialization.
      As to why measure reputation, I would guess it is reasonably well correlated with job placement. But PGR measures only partial reputation — reputation within a certain circle. At least that is what I think — the fields in which strength raises PGR ranking are also over represented among the evaluators.

      Like

      Reply
    10. Christopher Gauker Avatar
      Christopher Gauker

      Dear Rebecca: It is not unlikely that the evaluators share many biases. They may share a conception of the long-term reputation of a department, they may share a conception of the university to which it belongs, and they certainly know where the department has ranked in the PGR in the past. There is no reason to think that the PGR is a measure of faculty quality rather than being mainly an expression of these shared biases. The first sentence in the methods section of the report is the following: “This report ranks graduate programs primarily on the basis of the quality of faculty.” So you are mistaken to say that Leiter takes his report to be a measure only of reputation and not of quality. You write, “The accumulation of evaluations from a lot of people with limited information works pretty well.” I cannot imagine what such an assertion could be based on. I see that in your next post you concede that the PGR is described as measuring quality. The sentence that I quoted is even more explicit on this point than the sentence quoted by anon.

      Like

      Reply
    11. Anon Avatar
      Anon

      @Rebecca Kukla Regarding why some people want to measure reputation, I think that Jonathan’s insightful analogy at http://blog.jonathanichikawa.net/2014/10/high-school-popularity-modest-proposal.html is actually quite revealing on this front. The analogy neatly captures why some would want such rankings, and also the kind of large-scale consequences that they have.

      Like

      Reply
    12. Kelly James Clark Avatar
      Kelly James Clark

      Since the job market is so awful, how about a ranking based on placement of candidates in TT jobs? That’s the best service one could provide to prospective students. Maybe also provide information about real time to graduation, what percentage of students get funding and how much, and how much debt students leave with. Maybe also info on how much time graduate students actually spend teaching and grading. Or how about, does famous Professor X teach any courses (or care about the courses he/she is teaching) or take on advising of dissertations (and if so, does Professor X actually direct or meet with you or even reply to emails)? You know, stuff like that.

      Like

      Reply
    13. Zara Avatar
      Zara

      If you think the specialty rankings but not the general rankings are a benefit to people, my advice is to just do the specialty rankings. If everyone does that, then there will simply be no general rankings.

      Like

      Reply
    14. J Avatar
      J

      The first point is fair enough, though it only applies to evaluators who signed the statement. The second point is not really a point. The third point strikes me as unjustified.
      First of all, if I understand correctly (and this is based purely on the description given on the website), the survey contains the instruction “Do not check any box if you lack sufficient information to make an informed judgment about faculty quality.” So one need not—and should not—evaluate departments about which one is insufficiently informed.
      Second (again, based purely on the description given on the website), the ‘judgements’ being asked for here are pretty coarse-grained. One need only be able to say whether a department is distinguished, strong, good, adequate, marginal, or inadequate for the purposes of undertaking a PhD in philosophy (either overall or in a particular subfield).
      Are we supposed to believe that when Ichikawa is asked by a student whether Department X is strong, good or adequate in subfield Y, he professes agnosticism for all X and Y? Of course not. We all make these judgements, all the time. We all give our professional opinions to students when they ask for our advice. It’s part of the job. The PGR simply aggregates these opinions for the benefit of prospective graduate students.

      Like

      Reply
    15. Clement Loo Avatar
      Clement Loo

      Rebecca,
      I’ve noticed a number of people suggesting that the specialty rankings are more substantial and useful because they better track quality. But, I have a concern in regards to just those rankings that I hope you can address.
      It strikes me that even when the evaluators are well-informed regarding a particular sub-discipline, if the number of evaluators is not very large and the evaluators are not randomly selected, rankings are likely to reflect the biases of the evaluators regarding what research topics are most centrally important and interesting rather than provide an actual objective measure of the quality of the faculty.
      That said, I’m not sure what would count as an objective measure of quality for either a faculty member or a department in regards to a specialty area – so perhaps that is the more fundamental basis for my worry.

      Like

      Reply
    16. Gordon Avatar
      Gordon

      “The rankings are primarily measures of faculty quality and reputation”
      Alternate but compatible hypothesis: the PGR encourages a world in which quality and reputation are taken to be synonymous. The difficulty in being in a good epistemic position to measure quality (perfectly described in #3) who therefore has to go on shininess (as RK suggests) demonstrates the problem, since that person’s attempt to rate based on shininess gets laundered by the “quality and reputation” language into a judgment of quality.

      Like

      Reply
    17. Rebecca Kukla Avatar
      Rebecca Kukla

      Clement Loo in particular, and others: I guess I just don’t believe that there are these objective fine-grained distinctions of quality to be made, if only people were unbiased and informed enough to make them. A rough cut is all we are going to get. I’d have to be pretty insanely biased not to be able to tell that a department with Sally Haslanger in it is a better place to do feminist philosophy than a department with no one doing anything I have ever read or seen discussed in feminist philosophy (or whatever).

      Like

      Reply
    18. Jonathan Jenkins Ichikawa Avatar
      Jonathan Jenkins Ichikawa

      Thanks everyone for the interest and thoughts.
      Christopher Gauker asks me why I think the PGR does prospective students any good. Basically I agree with Eric Schwitzgebel’s new post: the PGR does at least identify departments that are perceived as prestigious within the discipline, and it is pragmatically useful for prospective students to have access to that knowledge. Of course, as I said in my letter, I have serious concerns about whether that benefit justifies its deleterious effects. (I really do think the high school popularity analogy is somewhat clarifying on both these points. It is socially advantageous to know who is and isn’t popular.)
      J asks with some skepticism: “Are we supposed to believe that when Ichikawa is asked by a student whether Department X is strong, good or adequate in subfield Y, he professes agnosticism for all X and Y?” I’m not sure why J considers that question relevant; I never suggested that there are no such X and Y for which I know the answer. I’m comfortable saying that Rutgers and Edinburgh are strong in epistemology. (I’d be disqualified from saying the former for the PGR.) But for the vast majority of instances, if a student asked me that question, no, I wouldn’t pretend to know the answer. I could and would mention individuals at departments I know things about. But without doing hours of research on the members of a given department — something I don’t do for the students who ask my advice about graduate programs — with a quite small handful of exceptions involving departments I know well, I wouldn’t be at all comfortable making assertions of the form “Department X is marginal for doing a PhD in area Y.” I’m surprised that J finds this surprising.

      Like

      Reply
    19. Margaret Atherton Avatar
      Margaret Atherton

      But Rebecca, that’s not what people are asked to do. They have to say which department with which feminist philosophet in it would be a better department than that one with Hadlanger in it to study feminist philosophy, and then which philosopher with some one other than Haslanger would be a slightly worse place to go to for feminist philosophy. I know this kind of point unleashes cries of, we do it all the time, but I am not convinced this is something to be proud of.

      Like

      Reply
    20. J Avatar
      J

      “I never suggested that there are no such X and Y for which I know the answer.”
      … in which case, you could have usefully contributed to the survey! As far as I know, no one is forcing evaluators to provide ratings for departments and subfields about which they don’t feel sufficiently informed. Indeed, if the website is any guide, evaluators are actively discouraged from doing this. The survey is only asking for your professional opinions about those X and Y for which you do feel sufficiently informed.
      You clearly have your reasons for not participating. My point was simply that an inability “to provide anything like reliable judgments” is not a great reason, given that you can undoubtedly form reliable professional opinions of the coarse-grained, mostly subfield-specific sort being requested.

      Like

      Reply
    21. Shane Wahl Avatar
      Shane Wahl

      Does teaching matter, at all? To anyone?

      Like

      Reply
    22. Ed Kazarian Avatar
      Ed Kazarian

      What Gordon said @ 16. Every bit of it.

      Like

      Reply
    23. Brit Avatar
      Brit

      You were invited but the email bounced back. It was re-sent this afternoon.

      Like

      Reply
    24. P2 Avatar
      P2

      And once again Brogaard, like Leiter before her, completely fails to acknowledge or respond.

      Like

      Reply
    25. Carl Sachs Avatar
      Carl Sachs

      To re-emphasize Rebecca’s point above: the PGR measures prestige, and it would function as a measurement of merit only if we had good reasons to believe the perceived prestige is a reliable indicator of merit. But we don’t have good reasons to believe that, and plenty of good reasons to believe otherwise — for all the reasons that Iris Marion Young outlines in “The Myth of Merit”.
      The question, as I see it, is whether a measurement of prestige is something we want. And I can see one reason why the answer might well be “yes”: because the prestige of one’s Ph.D.-granting institution (and the prestige of the people one works with while there) does play a substantial role in whether one gets a tenure-track job upon graduating. No doubt it ought not play such a role, but there can’t be much doubt that it does. Eliminating a ranking system like the PGR is not going, all by itself, bring about an end to the cognitive biases that search committees must rely upon to winnow a pool of 500 candidates to the manageable pool of 15 for first-round interviews.

      Like

      Reply
    26. ck Avatar
      ck

      It’s a plain (an undefended) assumption (made by many above) that ‘prestige’ tracks ‘placement and hiring’. But why assume it? It’s an empirical claim, after all.
      At this point in time, it is painfully obvious that good data on actual placement rates (which can be measured in multiple ways) is needed. The fact that Leiter himself openly (and rather viciously) criticized some of the first attempts at this (earlier on this blog) is telling about his opinions about what needs to be measured and why.
      Indeed what’s the point of a reputational survey if it doesn’t track placement (or some other affirmed value it is a proxy for)? Is reputation good for its own sake?
      Shouldn’t Leiter and all the PGR evaluators be spending their 50+ hours on collecting good data on placement? Why are all you evaluators spending your precious time on reproducing data (it’s only been a few years since the last one after all) when there is other more valuable data we (including your future students!) all need produced? How would you justify the opportunity cost?

      Like

      Reply

    Leave a comment