Two of my students just put together a philosophical climate survey. It can be found here. It's primarily intended for philosophy graduate students in PhD programs.
If you are a PhD student in philosophy, it would be great if you could complete it.
Hopefully this, together with The Philosophical Gourmet Report and other resources, will help students make informed decisions about which PhD programs to apply to.
“An anonymous survey has its drawbacks. For one, there is no way to ensure that all reports are made by actual students of the department.”
One way to read this remark is that, although there are going to be efforts to verify that the self-reports of institutional affiliation are accurate, there is no way that they can guarantee that no mischief will occur. Another way is that there will be no such efforts, and the authors of the survey will just take the self-reported affiliation as is. Here’s hoping it is the first.
LikeLike
Good point. They are very open to comments on how to make the results more trustworthy. It is also v possible that there are better ways to do it. But they thought they would give it try.
LikeLike
I think a survey like this is a wonderful initiative and resource. However, I want to point out that though the survey description states that “the survey will not only ask questions about the atmosphere for women, but also the atmosphere for racial minorities, the disabled, and those in the LGBTIQ community,” no independent questions are asked about the atmosphere for the LGBTIQ community. One question asks whether “Sexism (or cissexism) is not a problem in our department.” While these are certainly related phenomena, I think they warrant individual treatment, and so separate questions, because they often empirically diverge, and because a disambiguated report on each would be necessary for a non-cisgender prospective graduate student to assess the climate of a department. If the question asking whether “I would feel comfortable reporting an instance of sexual discrimination in the department” is also meant to ask about LGBTIQ discrimination, I think it would also benefit from separation into two disambiguated questions. Many LGBTIQ students effectively make a shot in the dark with respect to climate when choosing programs, as such information is difficult to come across anecdotally or in a brief department visit, and this survey would be a wonderful way to remedy that opacity.
Thanks for setting this up!
LikeLike
That’s a very good point. Thanks! I will pass on this and any further feedback. All this will also come in handy in discussing limitations of the survey later.
LikeLike
I have to be honest, I wouldn’t fill out this survey and I wouldn’t recommend others do so either (at this point), nor would I recommend people rely too much on the results. Here’s why:
1. The survey doesn’t note whether or not those who run it are capable of tracking IP addresses. Unless they specifically claim that they can’t, I would assume that they can. This is problematic for those who are in a vulnerable position and truly do need for such surveys to be anonymous in order to safely fill them out.
2. The survey asks students to identify their gender, their program, and their year of study. I’m in an especially large program, and even filling out that much information would make it possible for outsiders to make educated guesses about which students filled it out, and make it almost certain knowledge for those inside my program to know who filled it out (depending on the answers provided to other questions and the year of study reported).
3. The above two considerations makes it likely that those who are in the most vulnerable positions will be last to be in a position to respond to the questions, and that means even if everyone who does respond to the survey does so honestly to every question (which is itself unlikely), the results are likely to not be especially informative regarding the climate of a program over all.
LikeLike
Do you know whether the survey makes any of those questions optional?
(I obviously haven’t taken the survey)
LikeLike
I’m not sure–though, even if some of those questions were optional, I still it’s a problem. Say you have a program with only very few women. There’s usually even fewer women who are known to be unhappy. They will be likely suspects if anyone from that institution fills out a response that’s negative with respect to the gender climate. There are even fewer LGBTIQ philosophy students, students of color, etc., than there are students who are women given that so many women students are cis, straight, and white, and so the potential problems with other underrepresented social identities are even more acute.
These kinds of surveys are just extremely hard to do right, and harder still when they’re public. You have all the usual problems plus a greater risk of retaliation (and a greater risk of retaliation being directed at people who may not have even filled out the survey just because people will make assumptions about who did).
LikeLike
I worry that without any measures to ensure that respondents are who they say they are, or that the respondents are representative of the department, any results about a department will be about as reliable as entries in Ratemyprofessors.com. You’ll get responses from people who happen to hear about the survey and want to put in their $.02, maybe sometimes their $.02 about departments they don’t attend (or departments they ‘attend’ as faculty members).
It would be a lot more work, and still would have limitations, but what I’d suggest–if you’re going to move forward with this sort of thing–is to collect lists of e-mail addresses of current graduate students at all of the departments. Most departments list this. Then for each department set up a separate and tine-limited surveymonkey.com (or the like) questionnaire page, with the graduate students from each department receiving invitations to click on a link and give (anonymous) answers to a survey about their departmental climate.
Then, at least, you’d have information about the response rate (plus you’d be more likely to have a greater response rate), and it would be more representative–not perfect, as who would bother to respond might be skewed. But it would be better. And bogus identity answers, while maybe not entirely eliminated–people could, I guess forward on the link to unauthorized parties)–would be substantially reduced.
LikeLike
Yes, I can see that. I will ask them whether the questions are optional. But perhaps this will make the results unusable.
LikeLike
They will have information about response rates. They told me that they are running a statistics program on it. But I don’t know the details.
Your alternative suggestion, of course, has the advantage that the survey is sent to actual philosophy students.
They can probably call this a “beta-survey.” They are full-time students, so they have limited time on their hands. And I don’t personally have time to do it.
LikeLike
The respondents select themselves. They do not constitute a representative sample of the population. Each respondent rates only one object, i.e., makes no comparisons. There is nothing to prevent someone from submitting multiple responses from different locations. The results of this survey cannot provide any legitimate basis for drawing any comparative conclusions about departments. Individual narratives could persuade the reader that something was wrong in a given department or that department was in a good state, but this survey provides no opportunity for that. It cannot effectively serve the purpose of promoting inclusive departmental climates. It can only create confusion. Please suspend it.
LikeLike
That’s right. I think they should treat it as beta-survey. That’s kind of what they are for.
I can’t suspend it. I am not in charge of the survey, and I did not initiate it. I simply publicized it. (Unless you are talking about suspending the blog post)
LikeLike
Sorry, Berit, I did not mean to ask you in particular to suspend it. My “please suspend it” was a plea to the world of philosophy. Let’s please stop embarrassing our discipline by promoting illegitimate statistical methodologies. But in case the creators of this survey are looking in, I do also ask with them to suspend this survey.
LikeLike
I agree with Chris. And it’s not just about using dicey statistical methodologies period, but using dicey statistical methodologies to assess the climate of graduate departments including the climate for women, racial minorities, etc. If results from this survey are released, even with all sorts of disclaimers about its being ‘beta’ appended, any departments that fare poorly in it are going to be apoplectic. And they should be.
LikeLike
That’s never going to happen. They are not planning to do a ranking of departments. We already have a Gourmet report. For all I know, they are not going to release the information about departments together with the comments.
Perhaps they will leave the comments out for everyone to see without being linked to individual departments (just like the blog that allows women to post their stories).
So, what’s the point (you might say)? Well, just as the blog can have fake stories, there could be fake comments here. But it will help give people a picture of what it’s like not to be, say, a white, abled, straight man in philosophy.
And the authors are very reasonable people. So, I very much doubt that anyone has anything to fear.
LikeLike
Berit, they may not be planning on doing a ranking of departments, but they have already very strongly implied that they were planing to use this to make public information about particular departments.
http://forum.thegradcafe.com/topic/50741-philosophy-climate-survey/
LikeLike
Yes, they did, but they are listening to commenters’ concerns and are thinking about possibly redoing it with some other people in the social sciences.
The authors are well-meaning, very intelligent, and very reasonable people. They don’t work in the social sciences but in philosophy. But they never meant any harm. They just want to help improve the profession.
They have learned something from this experience.
And I prefer to treat my students as independent individuals who can make their own decisions. I don’t decide on their behalf whether or not to suspend any surveys or whether to initiate them. I don’t censor my students. I try to advice them. But I cannot and should not advice them on everything.
The reasonable decision that they have come to is their own.
LikeLike
Thank you all for the feedback. This was done with the intention of giving graduate students more information about the climate of graduate departments, but these concerns do seem to raise very serious worries about the methodology and the usefulness of any potential results. We tried to balance the concerns of retaliation from departments with the benefits of having an understanding for how graduate students feel about their departments, but many of the concerns raised here seem valid.
For what it’s worth, we were not intending to rank departments and not intending to compare them to each other. We weren’t trying to villify certain departments, or anything of the sort. We’ve suspended the survey accepting results and will likely take it down.
LikeLike
I have a worry from a different direction. How much do grad students know about a department? I certainly know stories about very bad behavior which deeply affected students while the students were unaware of the actions that produced the effects. It is quite possible for some faculty members to lie about, e.g., letters of reference without the students finding out, or finding out at all easily. Some members of a department might then decide to nix some of the students on letters in order to promote a favorite one.
For the record, I do not know of anyone doing that. I have heard candidates’ suspicions that it was done, but it is hard to tell what’s going on in the craziness of the job market. I think my beliefs about implicit bias suggest something like that can happen, but if so, it can happen anywhere. Let’s suppose the example is about a regular practice that is quite intentional.
LikeLike
I thank you, Matt DeStefano, with all sincerity!
LikeLike
Thanks Matt.
LikeLike
That’s also a very good point. And it raises another potential problem (letter writing).
LikeLike
Yes, thanks Matt–I think the motivation is really wonderful, but I think it’s equally wonderful that you suspended the project given the practical problems.
LikeLike
Although I agree that there are practical issues that should be addressed, I hope that the students who put this together don’t give up on the idea. I think that practically everyone agrees that academic philosophy as a whole has some major problems with discrimination and harassment, and so there is a real need for information. Unfortunately, it’s been very easy for people to paper over the problem by simply denying that it exists. So even some very rough and coarse-grained information about perceptions of discrimination could be of real value — even if it was as simple as reporting that a particular percentage of respondents claim to have witnessed a specific kind of behavior. Despite the inevitable caveats about methodology, self-selection of respondents, and so on, that kind of data would help push some departments into at least acknowledging that there is a real issue here. Furthermore, it would help justify spending additional resources to do a more rigorous survey later. Even if a survey had serious flaws, it could still lay the foundation for future studies — and so long as specific people/departments weren’t harmed, that would be a very good thing and a significant advance.
LikeLike
Yes, some information would be good. I don’t know what their plans are. But I know of some other people elsewhere who are thinking about putting this kind of evaluation together.
LikeLike
New APPS 
Leave a reply to anonymous Cancel reply