Note from Jeremy: this is a guest post from evolutionary ecologist Ruth Hufbauer. Thanks Ruth!
In helping a friend with some academic job applications, I recounted my experience on search committees. As it was eye opening to my friend, I thought it might be useful to others who are applying to academic jobs. The approach taken at my university is not taken everywhere, but my impression is that it is fairly common.
The excellent and thorough post written by Jeremy Fox a couple of years ago covers how search committees for tenure track positions work from start to finish in North America, particularly (given our fields) for Ecology/Biology positions. I will not reiterate what he wrote. It is a useful post even if you are not a biologist
My experience differs from him in how members of the search committee come up with a short list. At my R1 state school (Colorado State University, ~32,000 students, has a vet school, no med school), part of the process of conducting a search is creating a score sheet to rank applicants.
The job ad is critical because it provides the framework that the search committee uses to create the score sheet. Each of the minimum and preferred qualifications listed in the advertisement can be incorporated into the score sheet. If something is not in the ad at all, it cannot, per Office of Equal Opportunity regulations, be listed in the score sheet. That leads often to words being individually debated in writing the ad, as Fox noted in his post.
The upshot is that we do not have individualized rankings of candidates. The score sheet, like the example from an actual search linked to above, structures how the members of the search committee rate each candidate. The rating is done on a set scale that can be fairly coarse (e.g. out of 20 maximum) or it can attempt finder gradation (e.g. out of 100). Members of the committee score each candidate in each of the different areas: research, teaching, grant writing, postdoc experience, collegiality etc. These areas are often awarded different point values, according to what the position entails (e.g. a position with more teaching would allot more points toward evidence of teaching experience and ability). There can be heated debate on the search committee about what should be emphasized on the score sheet.
Using a set score sheet is not unique to my university. A colleague who is a chair at a major west coast research university system confirms that they use something similar there, and that they use it system wide across all disciplines.
For something like publications, the search committee typically takes time since PhD into account, and indeed, I have even seen committees create graphs of the productivity of the top 20 candidates by time since PhD (separated by first authored vs. total publications).
If a job attracts many applicants (e.g. >150) then often there will be a first cull of candidates who do not meet the minimum qualifications stated in the advertisement. This is typically done by just two members of the search committee. Not meeting the minimum qualifications can be things like not quite yet having a PhD, or having a PhD in an area different from that stipulated in the ad.
I haven’t personally served on a search that I would consider very large (with more than 200 minimally qualified applicants). In the last big search I was on (~90 applicants who met the minimum qualifications) every member of the search committee scored every single file.
The goal with this type of ranking system is to look at each applicant’s qualifications more objectively and holistically – not focusing solely on publications, and considering more completely what applicants bring to the table rather than, for example, their academic pedigree. It is by no means entirely successful in that effort, but it is a start. I find that it forces me, even when tired, to really evaluate each candidate in all the different areas, and not skip over aspects of an application.
A scoring system like this means, however, that applicants are reduced to a number on a spreadsheet. Major differences in individual scores given by members of the search committee are discussed in detail, but really only for the upper echelon of candidates. The majority of the applicant pool is not discussed. Only applications in the top ~10-20 are individually discussed. That group often is divided into a ranked list of potential interviewees, and a cut-off of people not acceptable to the department. The short list called for interviews is taken from the top 3 or top 5 (after discussions, during which some scores might have been adjusted if another member of the search committee points out something others hadn’t noticed). The upshot is that whether or not you make the top 20 is not at all a judgment of you personally. It is done by the numbers.
I’m curious –For those searching for an academic post – have you heard of this type of rating system before?
Among those who serve on search committees – Do you use this approach, or a more individualized/personalized ranking like at Jeremy Fox’s university?
Thanks to out to Josh Drew and Jeremy Fox whose suggestions and questions improved this post.