This is a guest post by Margaret Kosmala, a postdoc in Organismic and Evolutionary Biology at Harvard University and co-founder of citizen science projects Season Spotter and Snapshot Serengeti. She blogs regularly at Ecology Bits.
Back in December, I wrote a post here on Dynamic Ecology about citizen science data quality. I was in the midst of drafting a paper about the same topic (that I hope will be published soon-ish in a publication near you), and it was nice to explore some less-quantitative ideas in blog format.
You may recall that I had a brief survey at the beginning. It asked about career stage, level of involvement in citizen science, and one’s opinion about the primary purpose of citizen science. With the caveat that Dynamic Ecology readers do not form a representative subset of anything (and the caveat that that particular post attracted a disproportionate number of people involved in citizen science), I’m going to tell you about the results. I tried to capture the couple dimensions that I thought might most matter in influencing people’s opinions of citizen science. So I think the survey is actually a reasonable representation of what ecologists – or at least web-savvy ecologists – think of citizen science.
The survey asked people to choose one of the following statements (presented to readers in random order):
- Citizen science is primarily a way to gather otherwise unobtainable data
- Citizen science is primarily a method for conducting scientific research
- Citizen science is primarily a method for science outreach and education
- Citizen science is primarily a way to increase stakeholder engagement in conservation issues
- Citizen science is primarily a way to meet ‘broader impacts’ criteria on grant proposals
I consider the first two statements to be about science and data. In other words, these statements reflect a belief that the purpose of citizen science is to broaden our toolbox of methods to address scientific questions. And I consider the last three to be about outreach. That is, these statements reflect a belief that the purpose of citizen science is to engage non-scientists in the scientific process in order to promote things like scientific literacy, positive attitudes towards science, and buy-in for conservation objectives.
Just so you know my bias: My motivation for working in citizen science is highly science-skewed. Citizen science opens data doors that would otherwise be closed. I do also believe that citizen science can be successfully used for outreach purposes. And in both domains, project managers must carefully design and execute the project for their desired objectives. However, I believe that a citizen science project that is science-oriented can be successful without targeting outreach objectives. By contrast, I don’t believe that an outreach-oriented project is successful unless it actually contributes to science. (It’s just a glorified lab or field exercise otherwise.)
Having been involved in citizen science for several years now, I have discovered that there is a skepticism of citizen science as science in ecology as a whole. I’ve seen it directly in proposals that I’ve been a part of, and I’ve heard over and over from others others engaged in running different citizen science projects. Reviewer #1: “We can’t be sure the data will be any good.” Reviewer #3: “How do we know the volunteers are providing accurate data?” Summarized panel review: “The use of citizen science gives us hesitation.” Even, from a funding officer: “you shouldn’t propose using citizen science for data collection. It will won’t be received well.” These sort of comments appear even when detailed methods for ensuring data validity are included or when publications are cited that show a likelihood of high data quality in the proposed project. The establishment attitude has seemed to me to be “citizen science is great for outreach and we will support its educational objectives, but we’re not so sure it’s useful for actually doing high-quality science.” And this attitude seems especially deep-rooted at (national) funding agencies.
So my main hypothesis prior to this survey was that most ecologists see citizen science as an outreach activity, and not one whose main aim is science knowledge. With the recent rapid rise in interest in citizen science, I also hypothesized that later career people would be more likely to see citizen science as outreach, and that earlier career people would be more likely to see its scientific possibilities. Finally, I also hypothesized that the more involvement an ecologist had in citizen science, the more likely that ecologist would view it as a science tool, and not just outreach.
Here are all the results in a single figure (click it for a bigger interactive version): 1
I’ve grouped the five opinion options into two groups: “science” in blue and “outreach” in orange. Career stage is ordered from “undergrad student” to “10+ years permanent position” along the x-axis and involvement with citizen science from most involved to least involved top-to-bottom. Each pie shows the fraction of people in that career and involvement group who chose a “science” or “outreach” statement. The size of the pie tells you how many people are in that group. Marginal totals are shown along the right and bottom. 2
The result that surprised me the most was that more than half the 363 respondents saw citizen science as “science” rather than “outreach”.
I also saw the opposite of the trend I expected for career stage. Later career folks are more likely to see the science potential of citizen science than early career researchers. In fact, the only marginal group to lean heavily “outreach” are the group that reported that they were unfamiliar with citizen science – and that group had a small sample size and was heavily biased towards graduate students and postdocs.
Just one of my hypotheses appears to be valid: that the more involved with citizen science you are, the more likely you are to see it as a “science” pursuit, rather than an “outreach” one.
If I wanted to be cynical, I could explain away the results by saying that Dynamic Ecology readers are a weird sampling of the general ecology researcher pool – more likely to be tech-savvy and more likely to be open to alternative ideas, including ways of doing science. But if I listen to the (admittedly not perfect) data, they’re telling me that a bit more than half of ecologists see citizen science as a science tool – and the proportion is even higher than that for those who are likely reviewers for funding proposals.
So maybe that means that most scientists are open to the idea of citizen science as science, as long as proposers convince reviewers that their methods are sound. Maybe it means that it just takes one skeptic to down a proposal — and with, say, a third of all reviewers thinking of citizen science as “outreach,” that’s what explains all the anecdotes I’ve witnessed and heard. Maybe funding agencies have seen a large proportion of citizen science proposals crash and burn, and so are reluctant to fund them. After all, the rest of us mostly only see the positive results of citizen science due to publication bias.
In any case, I’m actually rather heartened by the survey results. Citizen science as a broadly used science tool (as opposed to one used by only a few of the mega-successful projects) is still a relatively new idea. There’s a lot more work that can be done to provide guidelines for better meeting both science and outreach objectives. And I think that the increasing awareness of data quality standards for large datasets and the importance of data management in ecological science generally are only going to help facilitate the production and use of high quality citizen science data going forward.
End note: I’m not planning on doing anything further with these data. They’re openly available if someone else wants them. Note that there was no standardized sampling and no human subjects vetting, so they’re not useful for publishing directly. They might be useful to someone as preliminary fodder for a proposal to do something more formal, though.
1. I’ve been playing around with Tableau Public and Plotly for quick visualization and exploratory analysis. I like them (in different ways for different purposes). While I’m a firm believer in scripted figure-making for publications, being able to make a figure like this quick-and-dirty is super time-saving. This figure was made with Tableau Public. ^
2. Most of the time, I’m adamantly against the use of pie charts. But in this case, I think the pies work well for conveying the general patterns of science vs. outreach. And I like having the number of respondents in each category easily viewable, as any conclusions could easily have been derided with a “but you have unbalanced data” comment. With this table of pie charts, I’ve plotted FOUR dimensions in a small space! I can address my hypotheses by looking at the totals on the right and at the bottom, as well as look for additional patterns in the middle. ^
Dynamic Ecology 
Interesting. Like you, I’m surprised that it’s so common to see citizen science as science, and that it’s more common among more senior ecologists.
Good point that you don’t need that many people to see citizen science as outreach in order for everybody proposing citizen science projects to get negative scientific reviews fairly often. I think the same phenomenon happens in many areas of science (and life). People make the unwarranted leap from “I regularly encounter view X or behavior Y” or “Everyone I know has encountered view X or behavior Y” to “Most people hold view X or engage in behavior Y”.
I’m actually less sanguine about these results. First off, unless you actively run a citizen science project, half of folk view it as outreach primarily. That means half of your reviewers, if they have any familiarity with citizen science (less so if not) are going to view this as a big outreach effort primarily, not science. That’s a lot! And it’s quite likely that even more folk on panels aren’t familiar with the quality of citizen science, and hence are likely to be down on it.
Taking the optimistic view – that as career stage advanced, folk begin to see the scientific purpose more and more, that’s still 1/4 of your reviewers on average who will see this as an outreach, not science, project. That’s not good odds in today’s funding climate when one needs to have stellar reviews across the board to do well.
Granted, I’m starting to ascribe continuous sentiment to a dichotomous exploration, but, this still makes me wary that a high percentage of folk see this as outreach first, and science second (and likely second class).
Then again, perhaps I’m just a pessimist.
Not sure if this will make you feel better, but I bet for any given research approach or question 25% of ecologists would be down on it, or at least not sufficiently excited about it to want to fund it. Microcosms. Data synthesis. Small-scale field experiments. Observational work. Fancy statistical modeling. Theory. Interdisciplinary work. You name it, there are some ecologists who aren’t fans of it. 🙂 😦
“That’s not good odds in today’s funding climate when one needs to have stellar reviews across the board to do well.”
Agreed. But I am heartened because I thought it was so much worse than half. And because the climate is getting better (I think). So we have less of a gap to cover than I had previously thought. (Not sure if that makes me more of a pessimist or less of one.)
“And because the climate is getting better”
This is something that’s come up in other posts. On just about any issue, I think you can divide people into those who focus on how things are, and those who focus on the rate of change in how things are. 🙂
Great post! I’m not surprised by the career trend. As someone using citizen science data for a part of my dissertation, my fellow graduate students are more concerned about data quality and my lack of statistical machismo while my more career-established mentors are excited about exploring the possibilities of citizen science. When I’ve presented this work, questions tend to follow that same trend.
Nice post! Thanks for the effort Margaret. I ve just been reading a couple of very recent citizen science related papers during the last days (e.g. Biggs et al Biol Cons http://dx.doi.org/10.1016/j.biocon.2014.11.029 or Amano et al 2016 doi: 10.1093/biosci/biw022) and for me it seems really hard to ignore the high potential for data collection that this opens. Still, as Jeremy notes, for every approach to ecology (or, science as a whole) you will find many professional people who will dismiss it right from the start. I fear we have to live with this.
Nevertheless, as good papers based on citizen science will keep on cumulating, I think funding agencies will not be able to totally ignore that forever.
Maybe you need to indicate how you validate citizen science data, or test it’s reliability as part of the grant. All data collection efforts include error. Evaluating it is often ovetlooked.
I think you may have missed where this is addressed in the post. Quoting from the post:
“These sort of comments appear even when detailed methods for ensuring data validity are included or when publications are cited that show a likelihood of high data quality in the proposed project. “
Thanks for a really interesting post. From the perspective of someone working at a conservation-science NGO, our answer to the “is it science or outreach” question is “it’s both”. Most of the questions that we need to answer require data collected across large geographic areas, and for this we rely on citizen-scientists. In that sense, citizen-science is all about the science. At the same, though, many of our citizen-scientists become supporters of our organization and its mission through the exposure they get from participating in one of our projects. They also tend to be very good at sharing with others – friends, neighbors, people they meet in the field – the ecological and conservation lessons gleaned from working on the project, so in that sense the citizen-science is really about outreach.
In keeping with the theme of trying to remain optimistic, I’d also add that many of our funding sources actively encourage proposals for projects that engage citizen-scientists. Granted, we aren’t relying on NSF, but if you look beyond the traditional sources of funding for academic research I think you’ll find broader support for projects that rely on citizen-scientists. My guess is that this reflects a growing belief that citizen-science may be a good tool for outreach on conservation issues and the recognition that citizen-scientists can be quite good at carrying out important but chronically under-resourced tasks like large-scale population monitoring.
Thanks for your perspective. I think you’re right that funding for citizen science may be easier to get from non-national funding sources. However, I also think that these sources strongly care about the outreach component. If you really want to do a science-mostly project, I think you’re likely to run into resistance and suggestions to broaden the outreach component (without a concurrent increase in funding).
That said, I think a lot of projects *want* to be both science and outreach. The problem I see is that doing good science and doing good outreach require very different sets of skills and I have yet to see a small project with limited means really get both right. (And some larger projects also fail to do both well.)
Thanks for including info on how you made the figure! I was wondering that. 🙂
Thanks for the interesting post Margaret. One comment I would make is that it seems to me that you see citizen science as being more-or-less entirely driven by scientists; at least this seems to be implicit in the questions you asked, and the way you discuss the issues.
If one takes a broader, and perhaps more historic, view of citizen science, then other possibilties come into view, such as scientists making use of data collected by projects primarily driven by the volunteers themselves, ‘Atlas’ data being an obvious example. Perhaps this is why senior scientists are happier with the CS approach, because they are more aware of the subtleties regarding what might be considered CS. This recent paper discusses the issue to some extent http://onlinelibrary.wiley.com/doi/10.1111/bij.12548/full and this one of mine http://onlinelibrary.wiley.com/doi/10.1111/bij.12548/full
You make a good point. I am coming at this from a frustrated-by-reviewers perspective, which means you’re right that I was only considering scientist-driven citizen science when I created the survey.
When scientists use data from projects driven by volunteers, there are still challenges in publishing. And some of these challenges may be affected by a reviewer bias towards regarding the data as “unallowable” since it was volunteer produced. So I think the results of the survey have implications for scientists using data from volunteer-driven projects, but I hadn’t considered these types of projects explicitly when I made the survey. Thanks for chiming in.
Yes, I see what you mean, although I think a broader conception of citizen science would also help here. For example, most British data on GBIF is collected by amateur naturalists — some of this will come from coordinated Atlas projects or structured surveys (coordinated by specialist societies rather than academics), some will come from more ad hoc recording, so any study collating data from GBIF will, knowingly or unknowingly, be using the fruits of what can be described as citizen science (see the Pocock et al. paper I linked to above). Most of the time such papers will not even mention CS, although of course there is a sizeable literature on biases in Atlas data and the like.
I think there is little appreciation of this fact, and a greater understanding of this amongst proponents and critics of CS would help to clarify the issues (although I appreciate Jeremy’s point that you can never win all people over, all of the time).
“I think there is little appreciation of this fact”
Totally agree. And I think you’re right that making “hidden” CS more apparent might help with its image. Great point.
Is faculty = scientist ^ non-faculty = citizen?
There are scientists who aren’t faculty, and many scientists use research assistants of various types for collecting and processing data. I would consider a volunteer citizen scientist someone who is not being paid for their contribution. See the original post for a better understanding of the context of this one: