If you didn’t know, in economics and political science, people are hired for faculty positions based in large part on their “job market paper”. As in, one paper, ordinarily from their Ph.D. work and often not even published yet. Number of publications matters relatively little (though apparently it matters more in political science than in economics). Economics even has a centralized repository of job market papers; that’s how much they matter.
I am curious to hear what you think of this, and whether you think this approach or something like it could be an improvement on current practices in ecology. Personally, I think current faculty hiring practices in ecology are mostly pretty reasonable (see also), and so don’t think this would be a net improvement on current practices in ecology. But I think it’s not so obviously a bad idea as to be uninteresting to think about. I find it useful to think about the practices of other fields and whether they’d transfer to ecology. It helps me look at standard practice in ecology with fresh eyes. A few thoughts to get the ball rolling:
- If you think that quality ought to matter more than quantity and that it would be best for science as a whole if scientists had incentives to publish fewer, better papers, well, here’s an actually-existing model of what it might look like to strongly favor quality over quantity!
- I’m wondering if this is a good way to identify the job candidate who will have the best long-term research program? Or if it might skew hiring towards people who do certain kinds of (good) research over people who do other kinds of (also good) research. This procedure would seem to favor people who, by dint of choice of question and/or research approach, can produce outstanding standalone papers as grad students (e.g., the sort of papers that win the Mercer Award). Which isn’t necessarily the same people who will have the strongest long-term research programs, though it might not be a bad predictor either (or even a pretty good predictor?) Of course, it’s already the case that you can’t expect to get a faculty position in ecology at a research university without doing significant research that only took you and your collaborators a few years to do. But possibly, relying on job market papers would select for people who do short studies of tractable problems, to a greater extent than current practices select for such people. (important aside: “short study” here means “study that can be completed in a few years or less”, which isn’t necessarily the same thing as “study based on only a few years or generations worth of data”. A “short study” might be based on many years or generations worth of previously-collected data, or based on many generations of data from short-lived organisms.)
- Economics and political science are not collaborative fields, though that’s slowly changing. Many papers in those fields are sole-authored, or at most dual-authored. I’ve noted in the past that current hiring, tenure, and promotion practices in ecology do not disfavor collaborative work. But relying heavily on job market papers probably would.
- In economics and political science, it’s my impression that job market papers get read, carefully, by everyone on the search committee, at least for the many candidates whose research area makes them a decent fit for the position. Whereas in ecology, I suspect it’s common for the search to be cut down pretty far before most search committee members thoroughly read any of the candidates’ papers, at which point they might well read up to three papers from each remaining candidate (or maybe that’s just me?) Which raises the question: is it better for search committee members to carefully read one paper from each of many candidates, or read several papers from each of fewer candidates? Or maybe that’s a poor framing of the choice here?
- I think there’s a big difference between evaluating reforms like this in the short- vs. the long-term. To use a macroeconomic analogy, it’s like the difference between partial equilibrium and full equilibrium analysis. To use an ecological analogy, it’s like the difference between the short-term response to a perturbation, and the long-term response after feedbacks kick in. If tomorrow every research university hiring an ecologist announced that they were going to start focusing heavily on each candidate’s one best paper, I think that’d be a bad idea. There’d be a serious mismatch between how faculty job candidates conduct and publish their research, and how their work is evaluated by search committees. And I don’t know how you’d get from the current “locally stable equilibrium” in ecology to the econ/polisci “locally stable equilibrium” without going through a transition phase that would be quite awkward. But I haven’t thought about it much.
I know not everyone likes to comment, so just for fun here’s a little poll:
Dynamic Ecology 
Via Twitter, Laura Dee shares various other tidbits about the way hiring works in econ (of which I was aware but others may not be). Some of which seem like they’d be improvements on current practice in ecology, others of which not. Which raises the difficult question of whether ecology could pick and choose ideas from economics if it wanted to, or if there’s some reason why current practices in any given field are a ‘package deal’ (meaning any one practice wouldn’t work without the others, or would be difficult to adopt without also adopting some of the others). A sampling below, see her Twitter feed for the full thread:
As Laura Dee points out on Twitter, job market papers in economics often contain much more than the typical ecology paper. In particular, they often contain both a theoretical model the author developed, and then data testing it, as well as extensive literature review. So arguably, going to the econ model of “job market papers” might not make as much difference in practice in ecology as you might think. Because instead of evaluating job candidates based on several papers, as is currently the case, we’d be evaluating them based on one big paper that reports the same work that would have been reported in ~3 papers. Compared to economists, ecologists arguably are salami-slicers.
Having said that, an expectation that the best papers will include both theory and data would be a pretty major change in ecological practice. Those papers, and the ecologists who can write them, exist in ecology, but they’re rare. But I don’t think you’d get more such papers, or more ecologists who can write such papers, by changing publication or hiring incentives. You’d get more such papers and more such ecologists through long-term and fairly radical changes in how ecologists are trained. Basically, you’d have to do as economists do and require all grad students to take classes in both theory and data analysis. Which probably would not fly (?) because of another big difference between economics and ecology: most economists don’t collect their own data.
Going to the job market paper model definitely selects against dissertations that comprise multiple unrelated projects, and the people who by dint of choice and/or circumstance end up writing such dissertations:
https://dynamicecology.wordpress.com/2015/10/20/can-a-thesis-or-dissertation-comprise-multiple-unrelated-projects/
In ecology, I’d advocate for an intermediate approach: ask candidates to pick their top 3(+/-?) papers, tell what they did for them, and give a short description of why they’d pick them. That way a candidate can highlight what they think of as their key research program, and show the threads connecting their work.
Also: I really like how econ allows pre-prints to be counted for hiring. You can control when pre-prints get released, whereas peer-reviewed papers can take up to a couple years to come out, stochasticity which hurts early career researchers more.
That intermediate approach is sort of what ecology already does. It’s not uncommon for application packets to include up to three papers. And any application for a position with some expectation of research is going to include a description of the applicant’s research program. Having sat on a search committee myself, I confess I find the usual statement of research interests and plans more useful than I think I’d find the answer to the question “what are your three best papers, and why?”
Re: preprints, that’s an econ-ecology difference with deep roots. Preprints have been important in economics for decades; the internet only changed how they circulate and somewhat democratized the process by which they’re circulated and get attention. Journals are still very important in economics, but mostly to validate and enshrine the results of informal discussions that have been taking place for years and that are based in large part on preprints.
🙂 well, then I’d say you just answered your own question. If you don’t find the top three papers useful, how could switching to the top one paper be better?
I realize that econ has had pre-prints much longer than we do. However, I think that incorporating pre-prints into hiring would be much easier to do than switching to the one-paper system, given that econ hiring is structured entirely around it, whereas counting pre-prints could be easily done by a single search committee. It would also help encourage people to put more up as pre-prints, which I at least see as a strong positive.
@Eric: Just to clarify, when serving on a search committee, I personally do find it useful to read a couple of papers from each of the top candidates, in addition to their research statements.
Can you clarify your suggestion re: incorporating pre-prints into the search process in ecology? You mean, you’d like to see search committees put the same weight on number of preprints as they do on number of publications? I confess I can’t see that happening and don’t think it would be a good thing if it did. I don’t think incentivizing people to (self-)publish even more stuff is a good thing for science as a whole. Progress of science is not publication-limited, I don’t think. I think you want to find incentives for people to post as preprints work that they would’ve written anyway, and that is polished enough to be well worth other people’s time to read. Or else find some other way to achieve the benefits that posting of preprints is intended to achieve. But I would say this, since I’m an old fogey and at least some of my filtering methods would cease to work in a world in which everybody was cranking out bunches of preprints for readers to sort through. 🙂
Further, would the job market paper select against dissertations in small, not so well funded labs and favor dissertations in “big shot”-labs? As this may increase the likelihood of getting a job market paper.
I think the right question to ask is whether it would do so to any greater extent than current hiring procedures in ecology. I’m not sure it would. After all, being in a big lab with lots of other good people and lots of resources really does help you do more and better science than you otherwise could (while perhaps also sometimes making it less clear to outsiders where “your” science stops and someone else’s begins.) Whether you write up that science as a single “job market paper” or as a bunch of separate papers.
On the other hand, if this were the way grads were evaluated, there’d be a lot stronger incentive to have a cohesive dissertation. My experience, which would probably have been different:
Me to advisor: “So… My first dissertation project failed. And my second dissertation project failed. So now I’m working on several unrelated projects. They could all become dissertation chapters, I suppose, but isn’t a dissertation supposed to be cohesive?”
Advisor: “Eh, don’t worry about it. You just have to write an introduction that ties everything together in some way. Many students have had at least one chapter that’s not closely related to the others.”
Yes, relying on job market papers would select for coherent dissertations, I think. Though perhaps not as strongly as it might seem at first glance. Because apparently the “job market paper” is usually just one chapter of someone’s dissertation, not the whole thing. So it’s probably wrong to think of the ecology equivalent of an econ job market paper as “my entire, coherent dissertation, crammed into one paper”.
In diverse departments there might not be a large number of people that would be comfortable in detailed analysis of a single paper. They can count number of pubs and impact factors and grant money. Probably one of the marks of a good candidate is the ability to do an oral presentation of the big research idea in way that engages a broader audience than the single awesome journal article (of course a few ecologists are good writers 🙂
Good point. Though my impression of economics is that the rationale for relying heavily on a job market paper is not primarily that they want to hire whoever’s best at getting all their technical analytical details right. They do want somebody who has important/interesting ideas and is good at communicating them to a broad audience–they just feel that a single paper is a good test of that ability.
Imagine a single paper that reports everything an ecologist would put in his or her job seminar, minus the “future research plans” bit. I think that’s roughly economists’ “search image” for their job market papers.
That would be very interesting. Do you imagine dissertations/post docs guided to this kind of format or, asking for a separate effort to better communicate the research story?
So far my Twitter prediction of “no clear consensus” in the poll is holding up. “Horrible” leads, but is garnering only 45% of the vote. Significant support for “awsorrible” (which I interpret as “mixed feelings”) at 33%, and “awesome” at 16%.
Surprised that there are only a few votes for me being strange. 🙂
That number might have gone up if you’d let us choose multiple options. 😉
Great, now I have to add “think of witty retort to Meg” to my to-do list. Like I don’t have enough to do. 😉
Via Twitter, I am informed that some social science fields are moving to a “both/and” model:
Worth noting this post from an economist who describes the job market paper more as a chance to show off your skills, without necessarily needing to obtain an earthshattering result: http://noahpinionblog.blogspot.ca/2012/02/how-i-survived-economics-job-market.html
Econ and political science do not have the same sort of postdoc tradition as ecology. What I have had explained to me is that, as a consequence, people get hired before they have had a chance to establish a research portfolio and reputation. Hiring committees therefore have very little signal so tend to rely on indirect measures, especially pedigree (and a single, often unpublished, dissertation paper). This is reflected in the data. The vast majority of hires come from the top 10 programs. My PhD advisor was an ecologist on a search committee for a joint hire with economics. He told me that the economists on the committee would not even look at the applications from anyone not from a top five econ program – regardless of their research topic or skill. However, my political science friends tell me that a really strong publication record from a second tier school can make one competitive with people from top schools with mediocre publications.
Aha! I was aware of the lack of postdocs in those fields, but hadn’t connected the dots to the higher reliance on pedigree (and preprints).
I think the current system is not great – I’ve heard people thinking about going on the job market in ecology/biology say they think they need something like 10 publications to have a shot at a faculty job. And this number is usually used without any acknowledgement of quality, length, etc. This rule of thumb, regardless of whether or not it’s true (not true for me, but I’m not in an ecology department now), means people who haven’t been crazy productive or who haven’t been coached to salami slice or who didn’t start publishing when they were an undergrad or who’ve had bumps and delays in their academic trajectory think it’s not even worth trying for a faculty position. If the mentality was ‘I just need one really great, thoughtful paper’ and hopefully some other collaborations, etc, that might change the views of students/postdocs to focus on quality and ideas, not papers papers papers. And maybe some of the great people who want to stay in academia but think they don’t have a shot at a faculty job would instead give it a try.
By “faculty position” I assume you mean “faculty position at an R1 university”. Because while I don’t know if that number is true or not for R1s, it’s definitely not true for most other sorts of places. The sorts that, collectively, hire more ecologists each year than do R1s.
Further, even if that number is approximately true as an average for R1s, I don’t know that that average is all *that* useful on its own to anyone seeking a job at an R1. Whether or not you’re competitive for any given job depends on all sorts of factors besides publication count. Quality of publications being one big one. I started getting interviews in the second year of my postdoc, a point at which I had 4 or 5 papers. But they were all first authored (several sole-authored), all in good journals, all substantive standalone papers that weren’t salami-sliced.
Having said all that, I agree with you that search committees ought to care about quality more than quantity. I say that as someone who’s never been that productive in terms of number of papers.
Jeremy, I’m not sure that you’re right about the number of papers ‘needed’ to be considered in the current job market. I’m a postdoc and everyone that I know personally who has gotten a tenure track job in the last ~2 years has had >10 papers and often much >> (about 15 people who I could think of off the top of my head). That includes a few people taking jobs at quite small teaching colleges. If you look at the anonymous qualifications tab on the Eco/Evo wiki for this year there are 22 people who have been invited for at least one campus interview and only 2 have <10 publications (and most have many more than 10). Again, that includes all kinds of schools, not just R1s (and admittedly it is likely not a very representative list). I agree that quality is obviously important too, but unless you have a few REALLY high quality papers it seems difficult to make up for the lack of volume in the eyes of search committees.
You could be right. I don’t know. I’d want to see more systematic data than are provided by any one person’s own knowledge or experience, or than are provided by the relatively small number of people who decide to provide their qualifications to the ecology jobs spreadsheet. It’s not that I trust my own instincts or anecdotal experience any more than yours on this–I don’t trust anyone’s. As we’ll see soon from a post I’m currently drafting, systematic and fairly comprehensive data on the job market often paints a rather different picture than what lots of people are *very* sure is true based on their own anecdotal experiences.
The other difficulty here is separating correlation from causation, in a situation in which there may be fairly high positive correlations between different variables. That most people who get interviews have attribute X doesn’t show that attribute X is effectively a necessary condition for getting an interview. Again, I got 10 interviews as a postdoc, the first when I had 4 or 5 papers, and all of which when I had <10. I'm not saying that many other people could or should do as I did. It could well be that I was an outlier for some reason or other. But insofar as I was an outlier, well, that's kind of the point. Outliers illustrate that the range of possibilities is larger than we might otherwise think. Perhaps I had fewer papers than other people who get many interviews–but presumably I made up for that by having more of other desirable achievements and attributes than other people who get many interviews.
I tend to harp a bit on this because I think widely-circulated "rules of thumb" about what it takes to be "competitive" have some truth to them–but less truth than many people seem to think. We have an old "myths of academia" post that's relevant here: https://dynamicecology.wordpress.com/2014/05/07/what-myths-and-urban-legends-of-academia-need-debunking/
I agree with Anonymous below for what it’s worth. With less than 10 pubs (though all but one first authored), I’m not even looking for faculty positions at any sort of school (US/Canada, R1/not, teaching/research). I applied early in my post-doc in basically your situation and got 0 interviews. No applications to R1 school either.
Also, having been exposed to 3+ hiring committees at different schools, I can definitely say there is a cut-off for number of papers, sometimes unspoken, but it’s there. And that’s before quality gets considered. About 10 seems right.
@Anonymous, ATM:
Here are data on the h indices of over 200 recent ecology asst. profs. hired in N. America, at the time they were hired. h index of course places a lower bound on # of publications:
https://dynamicecology.wordpress.com/2017/10/17/a-crude-statistical-profile-of-the-research-productivity-of-recently-hired-tt-n-american-ecology-faculty/
You’ll see that h indices of recent hires vary very widely, even just considering recent hires at R1 unis (though they do tend to run higher on average among recent hires at research universities). There are many recent ecology hires at R1s (and other research unis, and comprehensives, and SLACs, and teaching colleges) who had low h indices at the time they were hired.
A family member is an economics professor, so I sent this post on to him. In addition to what he had to say (below), I have noticed a few relevant differences between economics and ecology. As was mentioned above, economists often post a pre-print before actually submitting a manuscript. Once an article is accepted by a journal, the time to publication (in print form, at least) can be several years for economics. I was shocked to hear how much longer it is for them than us. (Said family member was an associate editor of a journal for a few years.) Also, as was alluded to by Laura Dee, in econ they have a central repository for applications and there is no to little customizing applications for each position. The applicant just puts their one set of documents into the system, then selects where they want to apply. (I may be over-simplifying the process a bit.)
From the economist:
“most economics PhD students will not have published anything when they go on the job market–and if they have published something, it will most likely be a collaborative effort with an advisor, which makes it difficult to ascertain the student’s contribution to the joint effort. (Remember that in economics it typically takes several years for papers to complete the review process.) The job market paper is intended to showcase the student’s best individual work, so it gets a lot of attention.
It is not true, though, that the job market paper is the only thing a search committee looks at. A new PhD will usually have several papers coming out of the dissertation; the committee will look at abstracts of all of them to get a rough sense of the student’s overall potential. And you may recall that the economics job market is highly organized: A large number of candidates get short (20 minute) screening interviews at the [American Economic Association] meeting in January. During those interviews, candidates are asked about their dissertation/job market paper but also about other current papers and their research plans for the next few years, teaching interests, etc.
Students who get a campus interview will almost always present their job market paper in their interview seminar.”
Thanks for this additional context and information, this is very interesting.
Surprised someone hasn’t pointed out that “ecological economics” is a field, and there is a journal of that name. Sounds like a recipe for confusion to me…. 🙂
Pingback: More faculty hiring practices from economics that ecologists might (not) want to consider | Dynamic Ecology
Pingback: Where did recently-hired N. American tenure-track asst. professors of ecology get their PhDs? | Dynamic Ecology
Pingback: How many first-authored papers in “leading” journals does an ecologist need to be hired as a tenure-track asst. prof at an R1 university? Not nearly as many as most ecologists think. | Dynamic Ecology
Pingback: Did you know that there are fields in which everybody talks up their favorite faculty job candidates on social media? | Dynamic Ecology