Citizen science and data quality (guest post)

Note from Jeremy: This is a guest post by Margaret Kosmala, a postdoc in Organismic and Evolutionary Biology at Harvard University and co-founder of citizen science projects Snapshot Serengeti and Season Spotter.

---

There’s no doubt about it: citizen science is a growing field. In the past two years, three major citizen science associations have been founded, an international citizen science conference was held, a new citizen science journal is on the horizon, and a new cross-disciplinary online citizen science journal has launched. Aggregator SciStarter and citizen science platform Zooniverse have recorded a linear – or faster than linear – increase in the number of citizen science projects and participants.

But before we go any further, a little pre-survey, if you don’t mind:

I ask these questions because while those of us directly involved in citizen science are excited about the potential of citizen science for conducting science, the broader ecology field seems a bit skeptical. And I wonder how extensive this skepticism is and how it varies with career stage. (With the caveat, of course, that readers of Dynamic Ecology are not guaranteed to be a representative sample of any group.)

A couple years ago, I blogged here that a project I was involved with had gotten the following feedback from a proposal looking for continued funding:

The use of citizen-scientists to provide meaningful and accurate data will depend on their training. It’s unclear how quality of data generated from the citizen-identified animals will be ensured.

Since then, I have heard these skepticisms over and over – from colleagues and from strangers who have contacted me to ask how to launch a citizen science project. An NSF officer is on the record for saying citizen science is more valuable as an education tool than for actually doing science. I heard the frustrations of citizen science project managers again most recently at this summer’s ESA meeting. And it’s starting to get a bit old. Why are reviewers so against citizen science as method for doing science?

I think the proximate cause is a distrust of citizen science data. But I think the ultimate cause is unconscious biases held by scientists about who we (scientists) are and what we can do and who they (the “general public”) are and what they can do. Here are some things that I think cause professional scientists to distrust citizen science data:

Belief: Professional ecologists collect high-quality data. After all, we’ve all had to go through many years of education, have had the experience of conducting research under the watchful eye of established scientists, and are deeply invested in our data.

Truth: On average, I think professional ecologists do collect high-quality data. But, let’s not confuse high-quality with perfect. Like all humans, scientists are error-prone. Sometimes we get tired, sometimes we lose focus, and sometimes we simply press the wrong button. Additionally, many types of ecological data require a fair amount of judgment that varies among data-collectors. Visually conducting percentage cover estimates is one example. In wildlife studies, using a pair of observers to count wildlife over transects is the norm, as it’s recognized that there will be variation among observers. And the statistics we use in ecology accepts that there will be some degree of measurement error, because expert-collected data is variable and imperfect.

Belief: Data in scientific studies is collected by professional scientists. Think about it: you agree to review a paper, and the methods state that “we measured diameter at breast height”. Do you question who “we” is? Probably not. Do you ask what training the measurers had? Almost certainly not.

Truth: A lot of ecological data used in scientific studies is collected by undergrads, graduate students, and technicians. These data collectors have some amount of experience and training, but if the assumption is that ecological data quality is high because professional scientists have collected it, then this assumption is violated on a regular basis.

Belief: Members of the general public (hereafter “volunteers”) cannot collect data as well as professional scientists can. Well, maybe they can come close if they have lots and lots of training. But most don’t have that training. Also, they aren’t invested in the project they way the professionals are. And gosh darn it, I spent 7 years getting my Ph.D. so that has to be worth something, right?

Truth: Ecological data collection often involves very simple measurements that can be accurately performed by schoolchildren. Things like counting and using a tape measure are not expert-level skills. There are dozens of citizen science case studies that measure volunteer data accuracy against expert data accuracy. And for simple tasks, volunteers perform as well as experts (and neither is perfect). Importantly, volunteers may need training on how to record their observations in the context of a particular project. But there is no skills training needed for many types of data collection and analysis. There is a point where training starts to matter, though. Asking untrained volunteers to differentiate among well-known organisms (e.g. elephant vs. giraffe; needle-leaf tree vs. broadleaf tree; bee vs. butterfly) yields good data. Asking volunteers to differentiate among less-well-known or cryptic organisms requires some training to get good data. And yet this training need not be extensive. A targeted guide to the differences between, say Grant’s gazelle vs. Thomson’s gazelle or red maple vs. sugar maple or honey bee vs. carpenter bee may be all that is required. That is, much ecological research involves collecting simple measurements, and much of the rest involves collecting easily-learned measurements.

Belief: Volunteers are members of the general public.

Truth: Citizen science volunteers come from a highly skewed fraction of the general public. They tend to be younger (because many projects target K12), better-educated, and well off (they have free time) than the general public. Except for kids trapped in classrooms, citizen scientists already have a predilection for science. And although they are technically laymen, many of them are expert non-professionals. Consider the many highly skilled birders who take part in avian citizen science projects. We have also found a hard-core following of African mammal enthusiasts in our Snapshot Serengeti project, many of whom have been to the Serengeti in real life or have grown up in eastern Africa and find the animals in our project to be as familiar as many North Americans find squirrels and raccoons.

Belief: Volunteers are not as motivated to pay attention to data quality as scientists and so are sloppy.

Truth: Volunteers are often highly motivated. After all, if you are doing something in your spare time, you probably care about it. A survey of volunteer motivations in an astronomy project found that contribution was the main motivator for people to do the project. That is, volunteers want to work on citizen science projects because they want to contribute to scientific knowledge. The awesome eBird project out of the Cornell Lab of Ornithology allows birders to collect data in two ways: recording a few target species or completing entire checklists. They found that when they informed birders that the checklists method yielded better and more usable data than recording target species, a large number of birders switched to the better protocol. The also-awesome Monarch Larva Monitoring Project based out of the University of Minnesota found that highly trained and engaged volunteers collected better data than paid technicians. There is additional data from the social sciences supporting the idea that paying people to perform a charitable task reduces people’s interest in doing that task. Our best data gatherers may, in fact, be unpaid volunteers devoted to the progression of science.

Belief: Because of all the above reasons, data collected by volunteers has high variability and bias. So we can’t use it for science.

Truth: While some projects do, in fact, collect uncontrolled data, many citizen science projects have procedures in place to record factors that influence accuracy and bias. The fact is that a lot of ecological data is riddled with unexplained variation and bias, no matter who collects it. This has led to a great toolbox of statistical techniques that allow us to measure and correct for such variation and bias. Problems such as non-standard effort, unbalanced designs, under-detection of organisms, and so forth are not unique to citizen science and can all be addressed statistically. Tomas Bird and colleagues have a great paper on this if you want to read more.

So next time you are reviewing a citizen science proposal or manuscript, pause a moment and consider that the volunteers creating the data may be quite skilled. Or perhaps the tasks the volunteers perform are easy and therefore there is unlikely to be much error in the data. Or perhaps the tasks are not so simple and the volunteers were trained. If so, look to see what training the volunteers got; there’s no reason to think that trained volunteers will do a worse job than trained student interns or paid technicians. Or perhaps the project managers are using statistical techniques to control for quality and account for bias in the data. There are many ways to ensure the integrity of data collected by volunteers. A little more respect for citizen science data overall could go a long way towards realizing the potential of this field beyond education and outreach.