tl;dr: Betteridge’s Law of Headlines is alive and well. 🙂
**********
Here’s an example of a influential theoretical result in economics that turns out to be wrong. More precisely, it’s invalid: it doesn’t actually follow from the models from which it was originally derived (as least, I assume it doesn’t; that is, I assume the linked paper now has the derivation correct!).
Which got me thinking: do we need to worry about this sort of thing in theoretical ecology? What fraction of theoretical results in ecology do you think are incorrect in the sense that they don’t actually follow from the assumptions from which they’re purportedly derived?
Off the top of my head, I can think of some serious errors related to the intermediate disturbance hypothesis. See Chesson & Huntly 1997 for the correct derivations, and Fox 2013 for an attempt to make those derivations intuitive. But those errors were all verbal “derivations”, or other non-mathematical mistakes. How common and important are errors in actual mathematical derivations in ecology? Do we have any reason to think that they’re common and important enough that some kind of systematic replication effort would be worth it?
Casual googling* turns up this blog post linking to data showing that minor errors in math papers are common, but major ones are rare. Same for coding errors. Major errors are especially rare in math papers and code that are widely used and relied upon by many people–think of theorems in undergrad textbooks, or the most popular R packages. See also, and also here. All of which suggests to me that we probably shouldn’t worry much about the validity of the theoretical ecology literature, and that a systematic “Many Labs“-style replication effort for theoretical ecology wouldn’t be worth it.
What do you think?
*You get the background research you pay for on this blog.
Dynamic Ecology 
I think I agree that there are (from what I can tell) fewer technical errors in theoretical and mathematical papers, for a variety of reasons. Though I do think there is definitely room for “redoing” classical things and questioning basic modelling assumptions, or simplifying assumptions which were used to make progress. In some sense this is using different operationalizations of verbal models, and seeing if one gets the same results.
For example, I’m having a student consider a very different form of temperature-dependent sex determination in crocodilia from what appears in James Murray’s book and papers in the area. I suspect we’ll find similar results, and that the specific mathematical details won’t matter much, but it’s at least somewhat worth checking I think.
As one of the tweeters brought up (re Gardner & Ashby simulations), anything that is programming *definitely* needs replication and that is an increasingly large fraction of theory these days.
To my mind for the equation flavor of theory, replication is mostly not for detecting errors in equations. I believe that these errors are usually minor (although it is interesting that many reviewers will not bother to independently re-derive say a long calculus derivation – its a lot of work so equations are not the most reviewed things out there). To me the value of replication in theory is implementing the same concept using different mathematical methods (leslie matrix and Mckendrick-vonForrrestor) to see if they get qualitatively similar results. This is a strong test of the robustness or brittleness of a result to implementation approaches. One high profile example is the MacArthur and Wilson simple deriviation of a probabilstic model of colonization/extinction. Which soem people didn’t completley buy (very MacArthurian in its simplification). Then several other people implemented much more sophisticated formal stochastic process models and got the same result.
Beyond mathematical tools, I’d add that replication is also valuable in testing robustness of results to the many simplifying assumptions we make translating biology into math. My 1st PhD chapter was all about the idea that “realized” dispersal often isn’t just diffusion: low competitive ability or reliance on facilitation of larvae/seeds (easy to model at least indirectly/phonologically) leads to remarkably different ecosystem patterning than with diffusion alone.
Of course, lower priority (?) of replication in theory accompanies more widespread plug-and-play approaches in building models, so we more often rely on previous authors making “good” simplifying assumptions.
Another good example of testing robustness (and the fact that assumptions just carried forward is the MacArthur & May model of limiting similarity. Which was shown by Abrams & Roughgarden to fail if you changed literally almost any of the assumptions. Hasn’t stopped people from continuing to model niches as bell curves, etc though.
I have a colleague in fisheries who claims that a single constant widely used for fecundity in fisheries models for 50 years, when traced back through the literature, was just a wild guess pulled out of thin air. But once it was cited 100s of times it had a gravitas that nobody questioned.
“Another good example of testing robustness (and the fact that assumptions just carried forward) is the MacArthur & May model of limiting similarity. Which was shown by Abrams & Roughgarden to fail if you changed literally almost any of the assumptions. ”
Ha, I was just thinking of the same example just now! 🙂
Fortunately, I do think there are a lot of conclusions in theoretical ecology and evolution that are pretty robust to changes in modeling assumptions. For instance, metapopulation persistence time peaks at intermediate dispersal rates. That’s qualitative claim is true in at least a dozen different models (Yaari et al. 2012 Ecology).
Of course, those simple metapopulation models don’t differ in *every* respect. For instance, they all have density-independent dispersal. When making a robustness argument, it’s always good to be clear about what features of the model are being varied, and which are being held constant. William Wimsatt has an old paper critiquing the robustness of the 1970s theoretical claim that group selection was implausible. Wimsatt argued that the various group selection models of the time differed in various ways but yet shared a key assumption. It was that key shared assumption that led to the conclusion that group selection wasn’t a plausible evolutionary mechanism. The risk with robustness arguments is that you get lulled into a false sense of security if all your various models share some key assumption and you don’t recognize that fact.
“I have a colleague in fisheries who claims that a single constant widely used for fecundity in fisheries models for 50 years, when traced back through the literature, was just a wild guess pulled out of thin air. But once it was cited 100s of times it had a gravitas that nobody questioned.”
Wait, what? I’m gonna have to ask my fisheries ecology colleague about that one.
Probably not, since major errors seem to be rare and so the benefit gained from the task would not really cover the cost.
But I would ask a related question: Is it a worthwhile task to build mathematical models for popular/influential verbal models/hypotheses to test whether these are valid? I would argue that it wouldn’t be, however verbal models should still be checked thoroughly. Their assumptions are often implicit or unclear, and the conclusion does not follow from the premises. One way to make a verbal model very explicit is to build it in a step by step way, by stating each term and its definition, and stating premises upon premises in an exact way such that the conclusion is immediately obvious once one puts the sequence of premises together. This kind of framework will also easily allow one to test assumptions (as you have written about before) rather than just predictions of the model. I would argue that doing this process for currently popular/influential verbal models is a worthwhile task. However, I don’t know whether there ARE any major popular verbal models in ecology currently.
The reason I think that mathematical models shouldn’t be made for verbal models is because I think that it is more difficult to change assumptions in a mathematical model and see how the model would work under a different set of conditions, especially if one wanted to do this for a wide variety of conditions. This limits their scope. Hence, one could argue that making a mathematical model of a verbal model, while making it more precise, actually limits the application of it to only a very limited set of conditions. Strictly speaking, this would be true of a single verbal model as well (basically a series of premises) but it to tweak premises and see the effect is easier.
However, I confess that I am biased because I do not understand mathematics very intuitively and easily, and the reason I stated against mathematical models may certainly not be true for many others. Which brings another question: should the depth, intuition and ease with which the scientific community (say, on average) understands your science be a factor in deciding how one does their science?
“One way to make a verbal model very explicit is to build it in a step by step way, by stating each term and its definition, and stating premises upon premises in an exact way such that the conclusion is immediately obvious once one puts the sequence of premises together. ”
I guess that sounds to me like building a mathematical model. And if it doesn’t amount to building a mathematical model, I guess I question whether it serves its intended purpose. For instance, I question whether anyone could’ve used the approach you suggest to catch the mistakes in verbal formulations of the IDH that Chesson and Huntly (and others) caught by using formal mathematics.
Re: the concern that mathematical models are limited in scope because of the specificity of their assumptions, I’d note that it’s often possible to do mathematics on *classes* of models. For instance, that’s why Chesson & Huntly’s derivations are compelling. They express verbal arguments about the IDH in the form of a *class* of models. Thereby ruling out the possibility that “this specific mathematical model doesn’t validate verbal arguments about the IDH, but some other, as-yet-undiscovered specific mathematical model might validate the verbal argument”. By analyzing a class of models, you analyze every possible specific model within that class at a stroke.
“should the depth, intuition and ease with which the scientific community (say, on average) understands your science be a factor in deciding how one does their science?”
I think one important role for mathematical models is to replace our pre-mathematical intuitions with new, better intuitions that we could not have acquired without first doing the math. The (well, a) role of math isn’t to shoot down our pre-mathematical intuitions and replace them with inscrutable math no one can understand. It’s to replace bad intuitions with better ones.
Your first point, that what I am describing is a mathematical model: I guess I would just say in response that it is possible to be mathematical without using mathematics. My hesitancy with accepting your argument is because there are so many fields where logic has to be used, but it is not really very feasible to carry out the entire argument using maths. So, if maths is the final test of validity (as I believe you are stating, please correct me if I’m wrong), then the arguments in fields where it’s not very sensible to apply maths cannot be tested to the same degree as those where they can. However, we use these mathematical-but-not-mathematics arguments in fields outside science all the time, and we base beliefs about many aspects of the world on such arguments. Should we then not, at least not to the same degree as we base beliefs in science? (Not asking this rhetorically, but a genuine question that I arrive at if I extend your argument)
Certainly, in this case i.e. that of the IDH it makes sense to make a mathematical model to test this, but does that necessarily apply to all verbal models in ecology? For example, borrowing from evolution, Darwin’s theory of natural selection and common descent. Mathematical formulations of natural selection came only in the twentieth century, but the theory of natural selection still made sense before that. Yes, mathematical models increased and do increase our understanding of evolution by a HUGE amount, but they did not fundamentally change the predictions of natural selection as made by Darwin (though that is debatable).
I didn’t know about the class of models bit. That’s very cool. Interestingly, this was exactly one of the criticisms that I got when I was discussing your arguments about IDH with some people, and it’s very interesting that you can analyse a class of models.
Your last point about the role of mathematical models is interesting, and I have never considered it like that. I can definitely think of successes (again in the context of evolution) such as neutral theory of molecular evolution, and can’t think of obvious failures (though detection rate would also be low for them). However, placing mathematical models in this context is actually quite interesting and useful. But again, I do feel slightly skeptical of the success with which this occurs – maybe this is the implicit goal, but perhaps it doesn’t come out clearly enough.
“So, if maths is the final test of validity (as I believe you are stating, please correct me if I’m wrong), then the arguments in fields where it’s not very sensible to apply maths cannot be tested to the same degree as those where they can.”
Agreed. When our picture of how some bit of the world works can be mathematized, we ought to mathematize it. Not doing so means not using all the tools in our toolbox. But when our picture of how some bit of the world works can’t be mathematized, then yeah, trying to mathematize it is going to be unhelpful at best.
The other point to make about trying to mathematize verbal models is that, even if you do come up with a mathematical model that makes some of the same predictions as the verbal model, that doesn’t necessarily mean the verbal model was validated. Though there’s often room for argument on this. Because it’s often unclear what the verbal model assumed and predicted in the first place (that’s why we need the mathematization!), and so it’s often unclear if the mathematization actually validates the verbal model.
The example I’ve discussed in old posts, and in my TREE paper on the IDH, is the “flip flop competition” model of Chris Klausmeier. That’s a perfectly valid mathematical model of resource competition between two species, with the identity of the competitively dominant species switching back and forth as the environment switches back and forth between two different states. The identity of the dominant species switches because the species’ per-capita feeding rates depend on the state of the environment. In that model, it turns out that intermediate frequencies of switching promote stable coexistence of both species, by generating a storage effect. This is of course the same prediction as Hutchinson’s famous verbal argument about the same scenario–intermediate frequencies of environmental change that switch the identity of the competitive dominant lead to coexistence. In Chris’ view, his mathematical model validates Hutchinson’s verbal argument. Not that he thinks Hutchinson verbally intuited the storage effect, of course. But he thinks Hutchinson’s instincts were basically right, he was just wrong about some of the details of the argument, details that the math allows us to fill in. I disagree with Chris, because I don’t think those details are mere details, they’re the whole argument! The coexistence mechanism at work in Chris’ model has nothing to do with the one in Hutchinson’s verbal argument. Hutchinson thought that intermediate frequencies of environmental switching promoted coexistence by interrupting competitive exclusion “just in time”. Which is just wrong, that’s not at all what’s going on in Chris’ flip flop competition model (see here for an analogy: https://dynamicecology.wordpress.com/2012/04/10/zombie-ideas-about-disturbance-a-dialogue/). Probably the best illustration that Chris’ model is totally different than Hutchinson’s is that, if you modify Chris’ model by making species’ per-capita mortality rates environment-dependent instead of their feeding rates, you still switch the identity of the competitive dominant when the environment changes states, but you no longer get stable coexistence because you don’t get a storage effect.
In summary, if you make a completely incorrect verbal argument that, by sheer luck, happens to nevertheless arrive at the correct conclusion, I don’t think a correct mathematical argument that arrives at the same conclusion “validates” or “confirms” your verbal intuitions.
An analogy: imagine I set out from point A with the goal of arriving at point B, and I travel by turning in a randomly chosen direction at each intersection. By sheer chance, I arrive at point B. You also set out from point A with the goal of arriving at point B, but you follow accurate directions provided by Google Maps. The fact that Google Maps got you from point A to point B doesn’t somehow confirm that I basically knew the way from point A to point B, or validate my method of getting from point A to point B.
I don’t imagine that this analogy will convince many people. 🙂 In general, I find that ecologists are firm believers in their own intuitions. Which is understandable. After all, when you (or me!) try to reason your way through a problem verbally, it doesn’t *feel* to you like you’re making mistakes. It doesn’t feel like you’re doing the equivalent of trying to drive from A to B by making randomly-chosen turns. Your reasoning process feels logical and plausible to you. But that’s precisely the problem–your reasoning process will *always* feel logical and plausible to you, even when you have made a mistake.
Now I’m wondering if it would be interesting to do a compilation of “incorrect scientific arguments leading to correct scientific conclusions”. Might be an interesting comparative exercise.
For some reason, I can’t reply directly to your last comment so I will reply here.
I completely agree with your argument and your analogy. Just because the math model makes the same prediction as the verbal model doesn’t mean the verbal model is validated, as the fundamental assumptions and premises were never stated in the verbal model or are unclear (often, these premises themselves are of a deeply mathematical nature so it’s not even possible to think of them without doing math – for example the storage effect, which involves statistical theory about products of averages). To say that the intuition was correct because the conclusion was correct is just not right. Also, it is cherry picking. Only in some models is this particular conclusion correct. If you change some aspects of the model, like as you said you make the per capita mortality rate dependent on environment instead of the feeding rate dependent on environment, or if you change the model itself, like Chesson and Huntly described one of their models, then this no longer works. In the latter example, disturbance may speed up or delay the time to extinction, depending upon the particular parameters of the system. So you’re picking the one mathematical model which validates the verbal model but ignoring the ones that don’t – and they all make different assumptions than the verbal model! It reminds me a bit of the predictions of sci-fi authors in and around the 1950s (vaguely). They made so many predictions, some of them were bound to be true. But if you look at the average accuracy of their predictions, it’s definitely less than 50 percent. So you can’t take something they said which resembles our life now and say they were incredibly good at forecasting their future. In the case of Hutchinson and fluctuating environments, sort of the opposite is happening. The proportion of models that validate his hypothesis is not very large, so you can’t take the ones that do and argue that his intuitions were correct.
I just read Chesson and Huntly’s 1997 paper, and I now see your point that you can’t really get there without doing the math.
“Incorrect scientific arguments leading to correct scientific conclusions” sounds like an interesting idea. I think there’s definitely at least a few examples of this. I have one to offer (though it’s likely you already know this) – but it is less of a prediction and more of an explanation. Altruism was though to exist because of group selection, where an individual sacrifices its fitness for the benefit of the group. Later, though, from work by Hamilton and others, it turns out this is better explained by kin selection. An incorrect scientific argument leading to correct conclusions. Here, though, I’m not sure whether the argument was incorrect because it was logically invalid, or whether the premises were incorrect. But nevertheless, it ostensibly explained altruism.
“Altruism was though to exist because of group selection, where an individual sacrifices its fitness for the benefit of the group. Later, though, from work by Hamilton and others, it turns out this is better explained by kin selection. ”
Of course, there’s a school of thought that says that group and kin selection are mathematically-equivalent ways to say the same thing. Which is perhaps another illustration of the value of doing the math (when it’s possible to do the math): it can reveal if two seemingly-different verbal hypotheses are just two different ways of saying the same thing. Darwin’s brief remarks in the Origin on the evolution of sterile castes in social insects are a good example. Darwin’s brief remarks have been interpreted as the first statements of *both* kin selection and group selection. Which illustrates the limitations of words, unaided by math, as a tool for teasing apart subtly-different ideas. Especially if the differences are so subtle that they arguably don’t exist.
(Against this, one could note that the existence of the math doesn’t actually seem to have settled the issue of whether or not group and kin selection are “really” the same thing. In response, I guess I’d say that clarifying an issue, and settling an issue, are two different things. Math is good for the former.
In pure mathematics, people don’t tend to believe proofs of big important theorems if they involve a bunch of new untested tools. Usually, a careful mathematician would use the new tools they develop to prove some statements that are already known to be true. Then once they establish the tools are consistent with known mathematical theorems, they will use the tools to prove the unknown big result. I’m not sure if this is a random tangent or if it is related. I thought I initially had some related point, but forgot, so I’ll just leave this here 😉
There are some caveats here too. Shinichi Mochizuki presented a claimed proof of the ABC Conjecture which relied on thousands of pages developing new tools (really an entirely new theory). The original claim appeared around 2012 if I remember correctly. Even three years ago, the community had grown extremely weary of trying to check the proof. More recently, I believe Peter Scholze and others found gaps in it which are insurmountable. This is a dated but apt sociological commentary which I found captivating.
https://galoisrepresentations.wordpress.com/2017/12/17/the-abc-conjecture-has-still-not-been-proved/
This sort of thing does happen from time to time. Definitely very rarely does one get new progress in pure mathematics from applying the standard techniques of the area. But I think most big advances come with small bits of novelty in the field across many people (for instance, Perelman’s work on the Poincare conjecture built heavily from work by both Hamilton and Thurston decades earlier).
“I’m not sure if this is a random tangent or if it is related. ”
Pretty sure it’s a random tangent. 🙂 Proofs in ecology hardly ever involve new mathematical tools.
Are there examples of important proofs in ecological theory that require “exotic” math? (i.e. beyond, say, a year of linear algebra and diff eq) This might not be e well-posed question, in that it requires judging what’s “important” and “exotic” (and “ecological”), but I’m just curious if any examples come to mind.
Interesting question! Related question: what’s the most ‘exotic’ math used in *any* ecology theory paper?
There are applied math papers about ecologically-inspired models, that are written for mathematicians. Interest centers on the mathematics, not its real-world ecological implications. I have a colleague here at Calgary in the math dept who does that sort of work. For instance, studying the behavior of a delay-differential equation model of density-dependent population growth with an *infinite number of delays* (!) I don’t know anything about the math you need to analyze the behavior of that sort of model, but I can imagine that it might be more exotic than anything you might see in, say, Theoretical Ecology or TPB or Ecological Modelling or Am Nat.
Closer to home, it’s my outsider’s impression that the math required to solve integro-differential equations and partial differential equations is more difficult than that required to solve regular ‘ol ordinary differential equations. So I’d guess that some of the more “exotic” math in theoretical ecology is in areas where such equations naturally arise. Some areas of spatial ecology for instance–think of work from folks like Mark Kot and Mark Lewis and many others. And also some structured population modeling–think work from Andre de Roos and many others.
Nonlinear stochastic processes might be another area that involves more advanced math than the typical theoretical ecology paper?
I’m just guessing, really, hopefully actual theoreticians will chime in!
In evolutionary biology, Alan Grafen has drawn on measure theory as part of his “formal Darwinism” project. That might be another candidate for most exotic math used in mainstream EEB?
The importance of Grafen’s project is debated. Grafen himself views it as fundamentally important, of course. My impression is that few share his view. But my casual impression might be very wrong or outdated; it’s not an area I’ve looked at for many years.
Yep, I think that’s right. Essentially when there is a messy stochastic process (e. g. with complicated possible correlations), or a differential or difference equation arises that needs to account for more than one independent variable, then more sophisticated mathematics is used. The biggest applications are spatial models, but also size or phenotype structured ones too.
I’m currently reading (to write a review of) a book on integro-difference equations in spatial ecology by Frithjof Lutscher. It’s definitely on the theoretical side, containing many results involving essentially new mathematics. But it does give the impression that such models are ideal for things like seasonal seed dispersal in plants.
I think the only other area where sophisticated mathematics appears is in studying stability of more complicated attractors than steady states (so stability of oscillating populations, for instance), and relationships to things like chaos. There are still some open mathematical questions there, though I think most people doing theoretical ecology would work closer to applications than anyone interested in those questions directly.