Very interesting news article in Science this week on how physicists trying to detect the extremely subtle signal of gravity waves use intentionally-faked data in order to learn how to detect real signals. Once or twice per year, there is a “blind injection”: without the knowledge of the physicists actually doing the data collection and analysis, the gravity wave detector is subtly reconfigured to make it produce fake data that appear to contain a signal.
I was struck by the lengths to which these blind injections are taken. The physicists working on the data analysis are allowed to spend months chasing down fake signals, checking for errors, trying to confirm the signal via independent means, etc. They’ve gone so far as to have observatories look for non-existent events in space that might have produced the fake signals, and in one case even wrote a paper and were ready to submit it when the deception was revealed. All of which perhaps just illustrates why fundamental physics, perhaps alone among major fields of science, doesn’t need to worry about whether it has a systematic problem with its statistical tests. Other fields, quite likely including ecology, have a lot of catching up to do in terms of bringing their analytical practices closer to the textbook ideal.
I was also struck by how the fake data were produced–by actually physically manipulating the detector so that it produces exactly the sort of data that would be expected if a gravity wave actually did occur. It’s not that unusual in ecology for people to test out new analytical approaches to see if they can detect a signal in simulated (i.e. fake) data. But the problem with this approach is that it’s often not at all clear that the simulated “signals” in the simulated data actually resemble the signals that would be seen in real data. For instance, techniques for testing for non-random patterns of species co-occurrence in species x site matrices have been validated by testing them on simulated matrices known to contain “checkerboard” patterns (a given site contains species A, or species B, but never both). But it’s unclear when, if ever, interspecific competition actually would be expected to produce such a checkerboard pattern (and if you say “it’s obvious that it would”, sorry, no it’s not, as can be easily demonstrated with a simple competition model). Which is a problem since we only care about “checkerboards” in the first place because they’re a putative “signal” of interspecific competition. Which perhaps just goes to show that, in order to properly test your signal detection abilities with fake signals, you first have to know what constitutes a “signal”, and what exactly it’s a signal of.