Challenges of writing higher order thinking questions in ecology

As I wrote about yesterday, our Intro Bio exams now emphasize higher order thinking skills more than they used to. I think this is really important, and think that the learning gains students achieve during the semester are more likely to stick long-term.

It does lead to some interesting challenges, though. Writing higher order questions is hard — perhaps especially so in ecology. In particular, it can be very hard to come up with several wrong answers that are definitely wrong and can be identified as such by a student who knows and understands the material, but that aren’t trivially wrong. In Software Carpentry instructor training this week, I learned the term “plausible distractors with diagnostic power”. When writing these higher order questions, it can be hard to come up with enough plausible distractors, and really hard to come up with ones with diagnostic power.

As an example: I wanted to write a question last year where I altered the tilt of the Earth on its axis, to see if students could link that with effects on seasonality. (Axial tilt is the reason for the season!) A simple recall question on this topic would be really easy to write. But I wanted to test higher order thinking, so I thought I would write a question where I made the tilt of the Earth more pronounced (because, hey, I can do that on paper!) and see if students could figure out that summers should be hotter and winters colder. But I had a hard time coming up with good distractor answers. For example, I wanted to write a wrong answer related to changing the location of biomes, since that seemed like a distractor that could get at a common source of confusion. But then I thought that it seemed plausible that tilting the Earth differently would alter Hadley cells and therefore alter the locations of biomes. After not coming up with four (or even three – all hail “none of the above”!) plausible but definitely incorrect wrong answers, I gave up on the question.

Lest you think I’m over-thinking how my students will think through the question, consider this example from an exam last fall:

Screen Shot 2015-06-05 at 3.39.04 PM

When I wrote this question, I thought it was a good way of testing whether students knew what net primary productivity really is, and whether they could combine that with information on which biome would have the highest NPP. But, based on approximately 953 student questions* during the exam, it became clear that some students who understood the concept were thinking themselves in circles, thinking that the rain in the rainforest would put out the fire, and maybe the deserts would burn the longest because they’re hot and dry, etc. We had to update the question during the exam to tell students that we were setting the fires in a controlled lab environment (wearing proper personal protective equipment, of course.)

Trying to prevent students from over-thinking a question or going down the wrong path sometimes leads to interesting scenarios where we are, for the most part, obsessed with making sure every last detail on an exam is correct, but then completely make something up because we need to in order to make the question tractable.

A great example of this is a short-answer question from last fall. The question started out with this intro blurb:

Screen Shot 2015-06-05 at 3.44.43 PM

The question is based on this study (and the image is a modification of one in that paper). I wanted to make sure I had all the info in the opening blurb correct, and even wrote a friend who works on African grasslands to make sure my summary was accurate. After giving them that background, we asked some relatively straightforward questions to try to scaffold them towards a harder question:

Screen Shot 2015-06-05 at 3.47.52 PM Screen Shot 2015-06-05 at 3.48.00 PM

Things got more challenging with part C, where we asked them:

Screen Shot 2015-06-05 at 3.49.32 PM

(This is clearly a harder question — we deemed it at Bloom’s level 4 — and is a pretty challenging question to ask freshman.)

I then wanted them to need to make a prediction through multiple trophic levels, so wanted to vary the amount of predation by top predators and ask them to predict what would happen to the thorny plants. In the actual study, predation risk for the herbivores depends on the amount of woody cover. I originally wrote the question with varying woody cover, asking the students to predict thorny plant density in habitats with high woody cover vs. low. But it seemed likely that students would get confused by plants both controlling the amount of predation on herbivores and responding to herbivory. So, my colleague Cindee Giffen came up with the excellent idea to introduce basking rocks that would allow us to vary predation on its own. I thought this was brilliant, though we were worried that some students might not know what a basking rock was. So, I added in a picture of lions with manes blowing in the breeze, basking on a rock, and the question became:

Screen Shot 2015-06-05 at 3.51.38 PM

I think it was a good question, but still find it amusing that we were, for the most part, completely obsessed with making sure everything on the exam was accurate, but then just completely made up basking rocks for the sake of getting a tractable question.

All of which is to say: I think it’s important to write questions that challenge students to think at higher levels. But it’s not easy!


* This might be a slight overestimate. But it felt like this during the exam.

16 thoughts on “Challenges of writing higher order thinking questions in ecology

  1. I’ll admit that I don’t understand how NPP matters to your fire-duration question. To me the obvious answer is rainforest, but not because of NPP – because of growth form (lots of very thick logs that will take a long time to burn to the centre; vs. grassland that will go up very quickly). But my logic here has little to do with ecology. The only way I can think to connect this to NPP is to add a specification that you feed the plants into the fire at 1 g/sec, which you don’t do.

    What am I missing? What am I overthinking? Would I fail your exam?🙂

    • The general idea was that there’d be the most production in the rainforest, so there would be more plant material there to burn, so the fire would burn longer. But I will admit that I wasn’t considering differences in the way particular materials would burn. We do have a fair number of people who read through the questions before an exam to try to find issues like that, but there are always some that sneak through!

      The good news is that, once we pointed out that we were burning things in the lab, students pretty much got the question.

      • I got the connection to NPP, but IMO the question seems needlessly indirect. Its really a question about the relationship between NPP and plant biomass, right? I think your point about the complexities such as how different materials burn etc illustrates why there’s limited room for change at the intro level – there’s only so much knowledge you can impart in a quarter or semester.

      • Aha, I would have gotten it right, albeit for the wrong reason! I’ve had this same problem trying to write good questions: the extra context I’ve put in to force the thinking kind of takes over, and the question becomes more about that than the biology underneath. In the end I like your other questions much better; this one seems to work too hard to conceal what’s fundamentally just a memory question anyway!

  2. I think every instructor runs into this. It’s hard to keep students thinking along the right lines.

    I struggle with this in biostats when asking experimental design questions. I have to give many practice questions and talk about the answers in advance so that the students learn how to answer them. For instance, that they can assume that any aspect of the experiment that wasn’t mentioned in the question was done properly. And that they’re not supposed to draw on their background knowledge of biology or physics or whatever to try to answer the question.

    One trick I use sometimes in ecology courses is to ask questions about made up organisms. Jackalopes or whatever. That ensures that the students have no background knowledge except the information I give them in the question, so it’s harder for them to go off track. But of course, that somewhat decreases the realism of the question.

    • We do this, too, and sometimes make up locations to try to avoid a student having background knowledge that gives them an advantage (or extra baggage to think through!)

      As an undergrad, I was a TA in an autotutorial Intro Bio course at Cornell. My main responsibility was to give oral quizzes on each unit to the students. It was such a great way to explore what students knew, and to avoid a student overthinking something or getting sidetracked — or, sometimes the path they took was more interesting and we went there. I don’t know of any other large intro course that is run this way, but I think it was a great experience for the students. It was definitely a great experience for me as a TA.

      • I’ve never heard of autotutorials, sounds like an interesting approach. We have what we call tutorial sessions at Calgary, which often function as review sessions, but I don’t think they work this way. Any chance of a future post on autotutorials?

  3. Something I have been searching for, but haven’t found is a text reference on how to write exams in biology/ecology. Have I overlooked one? There has to be a education scholar out there who has enough knowledge to write a book to help us out!

  4. Pingback: Taking a writing-intensive evolution course was transformative for me | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s