Back in the fall I wrote a post on statistical machismo in ecology, arguing that ecology is prone to use increasingly complex statistics without necessarily stopping to weigh the costs and the benefits. I singled out four specific techniques: phylogenetic regression, spatial regression, Bayesian methods and detection probabilities. I at no point said these techniques were bad or should never be used. But I did say that we had in many cases reached a point where the techniques had become sine qua non of publishing – reviewers wouldn’t let papers pass if these techniques weren’t applied, even if applying them was very costly and unlikely to change the results. Most of the comments were on the Bayesian (which I do NOT want to reignite here) and the two GLS (phylogenetic and spatial) regressions which lead to this follow-on post.
I got only one comment on detection probabilities. However a new paper published today in PLOS One called “Fitting and interpreting occupancy models” by Alan Welsh, David Lindenmayer and Christine Donnelly made me very excited and wanting to revisit detection probabilities.
Now, if you are based in a wildlife department you already know what detection probabilities are. Indeed, most of the committees of students I sit on in the wildlife department mention detection probabilities with a groan and a roll of their eyes but then go ahead and modify their design, at great cost – namely halving or more the amount of data they can collect – to address detection probabilities. You see, in wildlife journals detection probabilities have become a no publish line – you can’t publish without detection probabilities.
Although many in basic biology and EEB departments remain blissfully unaware of detection probabilities, the expectation is starting to creep into reviews on papers in basic research as well. As somebody who frequnetly publishes papers using the North American Breeding Bird Survey, I have now had three papers rejected a total of six times for the “sin” of not using detection probabilities (never mind that I couldn’t, didn’t need to, and it wouldn’t change the answer). So beware, this issue is coming to your population biology papers soon!
Detection probabilities are a statistic model/method designed to deal with one simple obvious fact. When you are censusing mobile organisms like birds or mammals or butterflies or … (really almost anything except plants and maybe snails), you miss organisms. You are not censusing the whole population. This has long been recognized by reporting such counts as an index of abundance rather than abundance per se (and if you need total abundance you have to use a method like mark-recpature). Detection probabilities got their start when people reported occupancy (presence/absence rates) instead of abundance. The idea was a claim of absence was pretty shaky when you’re not counting all the individuals yet the data was presented as a binary and hence large difference (presence or not). So far reasonable enough.
This paragraph has the heavy math – try to read it – but do keep going on to the paragraphs after and don’t just give up! The proposed solution was a two step model: let Oi be the true occupancy (1 if a species present, 0 if absent at site i). We can assume the true underlying occupancy rate is Ψi (a simple way of saying let Oi be distributed Bernoulli(Ψi). So far this is just a probability model of occupancy (in the simplest case where Ψ is constant across sites the occupancy rate is just Ψ). Now comes the fancy part. Let Di,j be what is actually observed at site i for observation repetition j (again D=1 if present, 0 if absent). Now Di,j can be different from Oi and we have added detection to our model (although not yet fully specified it). To fully specify we need a few things:
- Multiple observations of D giving the subscript j
- Assume Oi doesn’t change across the multiple observations j (i.e. a site doesn’t flip from really occupied to really unoccupied or vice versa between visits)
- Assume P(Di,j=1|Oi=0)=0 (we never mistakenly observe something at a site when it is not really there
- Define pi,j=P(Di,j=1|Oi=1) (i.e. the probability it is detected if it is there) – aka the detection probability
- Assume pi,j is constant across observations (so we can drop the subscript j giving pi)
Now you don’t need me to tell you that assumptions #2, #3, and #5 are all whoppers. But lets give the method a chance. Under these conditions it is not too hard to write down the maximum likelihood estimates (MLE) and solve for pi and Ψi which are both unobservable using the observations Di,j. It is also quite common to let pi and Ψi be functions of covariates like land cover type, elevation, etc (using logistic regression) – this more advanced model is also fairly directly solvable.
If you think about it should be obvious you cannot estimate a detection probability and an occupancy separately if you only have one observation of the site. Thus #1 (repeated observations of the same site) is critical. Right here is the nub of detection probabilities – you can only use them if you make REPEATED observations of the same data point. If you make three repeated observations, then you will in a fixed amount of time only be able to observe one third as many sites as you otherwise would have been able. This is why wildlife ecologists hate having to address detection probabilities. It has a very real cost – it is not just more computations – it is more data collection (or equivalently less independent points for all ensuing analyses). But wildlife ecologists have buckled down and done the more observations while losing power/df inherent in detection probabilities because they can’t get their paper published any other way. Now you understand the eye rolling. This also is a serious problem for people like me using historical datasets like the breeding bird survey that were never designed with detection probabilities in mind. They “only” have one observation per point in space and time.There is no way to go back and add repeated observations thus demanding detection probabilities is tantamount to throwing away historical monitoring datasets – ouch!
Well, a couple of Aussies (the aforementioned Lindenmayer and Donnelly) were doing a nice study on the effect of monoculture pine plantations on bird communities (abundances). I don’t know the full story but judging by the paper I linked to above they must have gotten told by reviewers at least once that their paper was unpublishable and that they should: a) abandon abundances and only do occupancy so that they could b) use detection probability methods. The whole idea of throwing out abundance information and reducing it to occupancy just becaue “its more statistically proper” makes my stomach turn and apparently it did theirs too. They went out and got a clever statistician to work with them, Alan Welsh, resulting in the paper I am discussing.
It is quite a technical read, so let me boil down the main findings:
- In the version of detection probabilities where pi and Ψi have covariates modelled through logistic regression, the solving of the MLE equations is a lot harder than people have given it credit for. They found many cases where there were multiple solutions to the MLE (i.e. the answer depended on the initial guess you gave the solver) or where the solutions converged on the boundary (where pi and/or Ψi are either 0 or 1 which are theoretically impossible). The core issue is that there is a lot of freedom to move the solution back between high pi and low Ψi vs low pi and high Ψi , both of which would give the same observed result. When you throw in logistic regression which can have its own convergence problems when the data is very noisy, you get a mess. To be more exact, you get a lot of real-world wrong answers spit out by the computer even to the point of estimating slopes of the wrong sign.
- This problem is compounded when the data is sparse – i.e. either pi or Ψi is low (by which I mean say 10% which is not at all uncommon to have an occupancy of 10%).
- It has been fashionable recently to notice that detection probability depends on abundance (gee really – a species is more likely to be noticed=detected when there are 30 of them then when there is one of them?). But this is a major violation of #5 above (detection probability constant across sites under the very likely scenario that abundance varies across sites). There are ways to try and deal with this, but as the Welsh et al paper show, all of them have problems, leaving the detection model nearly always in violation of a core assumption of constant detection probability other than for modelled covariates.
So where does this leave us? Every model is imperfect and has assumptions violated. What are the consequences of #1, #2 and #3 for detection models? Welsh et al found (in an extremely rigorous paper where eveyr point was supported by analysis of real-world data, analysis of simulated data and analytical results) that:
- Frequently the estimated relationship of detection and occupancy to covariates is very wrong. So for example in the original study which looked at how maturation of the pines influenced detection probability (old bigger forests should have lower detection probabilities) it was often estimated that detection *increased* with size/age of forest.
- The estimates of occupancy and detection are biased and have high variances. In fact have the same amount of bias and high variance as if you just ignored the detection probabilities and went back to the old way of doing things!!! (and this was on simulation data where detection issues were built into the data).
Bottom line – ignoring detection issues often gives misleading/wrong answers. But at exactly the same rate as if you were modelling detection which also often gives misleading/wrong answers. When you combine this with the real world fact that often times only half or one third of the data (by which I mean independent observations) is collected that would have been collected if we ignored detection probabilities, one really starts to question the appropriateness of demanding detection probabilities.
I claimed at the start of my post that I wasn’t saying any technique wasn’t inherently bad and should never be used. And I’m not saying that about detection probabilities either.
One of the most sensible thinkers on detection probabilities I know is Steve Buckland who has been a leader in the development of detection probabilities. In chapter 3 of the edited book by myself and Anne Magurran (sorry shameless self promotion), Buckland says “Ignoring detectability might not be a major problem if the bias is consistent across time or space.” But then goes on to demonstrate quite clearly that results can be misleading if detection is ignored in other scenarios. He clearly is not black-and-white about the need to use detection probabilities. Buckland also developed a nice method where instead of repeated measures of the same site, one only needs to estimate the distance to observed individuals which can calibrate a detection decay curve. Estimating distances is not cost-free compared to just counting, but it is much less costly than repeated visits to sites and thus is a great benefit to wildlife ecologist who have to worry about detection probabilities. The distance-based detection method seems not to have made it over “the pond” to the US as well as it should have.
Here are my recommendations.
- In light of Welsh et al’s findings it is flat out wrong for reviewers to insist that detection anaysis is a requirement for publication.
- It is more important to address detection if you are actually studying occupancy (presence absence) and less important when you are studying other factors like community structure, abundance etc.
- It is more important to address detection if detection probabilities are likely to vary across species (e.g. different detectabilities by species which is common enough) or space (e.g. varying amounts of brush) or survey points (e.g. varying effort levels) and that comparison (across species or sites) is what is important to you but it is less important to address detection when things are fairly constant across your axis of comparison – e.g. looking at just one species (so no issue of differing detectabilities between species) across space when there is not a reason to expect habitat to vary much (so no reason to expect varying detectabilities across space)
- If you do have to address detection probabilities (because of your question and experimental context, hopefully, not because of reviewers), then: a) consider using Buckland et al’s distance methods, and b) consider getting serious and doing more than just two or three repetitions of each site – if you really are interested in occupancy and detection then you need real replication along that dimension just like for any other variable of interest.
I think the main theme of my post on statistical machismo is there is no such thing as cookbook or one right way to do things in statistics. You have to know what you’re doing and think things out. Sometimes one way is appropriate. Sometimes an alternative way is appropriate. And these have to be weighed against real-world costs in data collection and loss to science of interesting studies. Detection probabilities are no exception. So if you’re a reviewer or editor, please stop telling poor authors you “have” to do detection probabilities because “its the only right way” or “gold standard” for how to do it. Its not – it very likely introduces as much error as it fixes and whether you should do it depends on the question and the data and requires thinking.