We asked more of our Intro Bio students; they rose to the challenge

Over the past year+, my fellow Intro Bio instructors (Cindee Giffen, Jo Kurdziel, and Trisha Wittkopp) and I have invested a huge amount of effort in overhauling the course. The most substantive changes we made involved 1) moving some content outside of class, 2) frequent quizzing (on the readings they are supposed to do before class as well as harder questions about material covered in class), and 3) increasing the amount of active learning occurring in the classroom (through the use of clickers and in-class activities).

What was the effect of all this effort? We all had a sense that we were asking more of the students and like they were rising to the challenge. But a vague sense of that is not a particularly convincing argument to give colleagues, especially given the substantial stress that accompanied the first semester of teaching with this new model.

I realized that one way to compare things was to compare exams from Fall 2012 (traditional lecture format) and Fall 2014. (I was involved in writing exams both semesters, which is why I chose those semesters.) When I said above that we had a sense we were asking more of students, what I mean is that I felt like we were asking students to really process information and think deeply on exams, rather than to memorize a lot of facts and then regurgitate them.

Given that impression, we decided to apply Bloom’s taxonomy (another link) to exams for the two years. Bloom’s taxonomy allows you to assess where questions fall on a 1-6 scale from lower order thinking (simple recall of facts) to higher order thinking (analyzing and evaluating information that is presented). We decided that we had four of these levels on the exams:

1. remembering – that is, basic recall
2. understanding – for example, given a new example, being able to recognize a concept we’d covered in class
3. applying – for example, recognizing what equation needed to be used, figuring out what information from a question is relevant to the equation, and correctly solving
4. analyzing – giving students completely new data that they had to interpret in light of concepts covered in class, or having students make predictions (verbally or graphically) based on a novel scenario.

To give examples of a multiple-choice question in each category:

Level 1:

Level 2:

Level 3:

Level 4 (I wrote this question on an early morning run. Multitasking FTW!):

After going through each of the four exams from both years, I calculated the mean Bloom’s level for each exam and then used the four exams as replicates within a year.* Ideally we’d have more years to compare, but there were too many other factors that changed with other semesters.

What did we find? We found that the exams asked students to think at higher levels, but there was no difference in performance on the exams in the two years (if anything, they did a little better on the harder exams):

In other words, we asked more of our Intro Bio students and they rose to the challenge. To me, this is really exciting evidence that all the effort we put into overhauling the class was worthwhile. But you could argue we still have a way to go — is an average Bloom’s level of 2 high enough for Intro Bio? I truly don’t know, and this is something I should think about more.

It does raise an interesting question: if students are learning more, should grades go up? Does every student deserve an A? The answer to the second question is clearly “no”. (Edited to add: I say “no” because, while the overall performance went up, some students still clearly did not grasp the material, let alone master it. If everyone had mastered the material, I’d happily give everyone an A.) The answer to the first question is less clear. The new model we’re using for the class requires the average student to spend much more time on the class, which surely contributes to the increased performance. Should that be rewarded? I feel okay with the grade distribution being slightly higher (say, a tenth of a point) than it traditionally has been, because I think we can demonstrate that our students are learning more. But I don’t think there should be a substantial shift. There’s still a wide range in performances on the exams – with 650 students, we naturally get a normal distribution on exams. I think the grades should reflect that variation.

I also think it’s worth pointing out that the previous exams weren’t easy. They were difficult in their own way. I’m not sure how well I would have done on them as a student, to be honest, because they required memorizing lots of facts and I’m not particularly good at memorizing. I think they were hard because students had to jam a lot of information in their brains and then spit it back out. I suspect most of them forgot that information before long. My hope is that, by having repeated testing on the same topics (via the quizzes) and then by needing to work with the information at a higher level (including on the exams), the information and skills the students learned in the new format of the course will stick with them for much longer.

What I would love is to follow up with these students in a few years to see how they’re doing. Are they retaining more information from Intro Bio when they arrive in upper-level ecology, evolution, and genetics? Do they come in with better process of science skills (e.g., figure-reading abilities)? I would love to know! The problem is that we don’t have a good control group, since we’ve fully shifted to the new format. Given that, I’m not sure how to tackle these questions. But I hope we come up with a way, because I would love to know if this format has longer term benefits for our students, too.

*Data and code are available on GitHub.

Update 6/12/15: People have been asking for the answers. I should have thought to include them! Here’s a key:
Level 1 question: A
Level 2 question: B
Level 3 question: C
Level 4 question: C