We flipped our huge intro biostats course. Here’s how we did it. (UPDATEDx2)

In a previous post I talked about why we flipped the intro biostats course here at Calgary, even though the old course was working fine. Now I’ll talk about how we did it. If you haven’t read that previous post, please do so before reading this one. To decide if what we did might work for you as well, you definitely want to know why we did what we did, and about relevant context like the size and makeup of the class. And if you want to know how others teach intro biostats, see the comments here.

Overview: a team-based flipped classroom

I’ll first just summarize the course structure, then talk about why that structure works.

I should emphasize up front that this structure wasn’t my creation, though I’m liking it so far. My colleague Kyla Flanagan took the lead on developing the new structure, drawing in part on the experience of another Calgary instructor who’d used a similar structure in one of her classes.

The new course has a team-based structure. For the lecture sessions (50 minutes 3x/week), students are divided into permanent teams of 5-7 people (students work independently in the labs, which we didn’t revamp).

The course material is divided into 8 modules, each a few lectures in length. Each module covers a different topic or set of related topics. For instance, we have a module on Experimental Design, a module on ANOVA, a module on Regression and Correlation, etc. Each module follows the same sequence:

1. Pre-module: before the first lecture of each module, students are assigned background readings, usually 1-3 chapters from the textbook. We also provide a reading guide summarizing the key things we expect them to get out of the readings (key terms and concepts, things we expect them to be able to do, etc.).
2. First lecture: Readiness Assurance Tests (RATs). Yes, that’s an unfortunate acronym. The students take a quiz on the background reading, to assess their “readiness to learn”. It’s usually 10 multiple choice questions. The students first take the quiz individually; that’s the iRAT. Then they take the same quiz with their team, submitting one set of answers for the entire team; that’s the tRAT. Both the iRAT and tRAT are for marks, but they’re low stakes; they’re not worth a big fraction of the final grade either individually or collectively.
3. Next 1-2 lectures: Lectures focused on those aspects of the topic with which the students are struggling or are likely to struggle, based on the instructor’s experience and the students’ answers on the iRAT and tRAT. Of course, the instructor needn’t just talk the whole time—the lectures can and do incorporate clicker questions, pair-and-share, etc. How many lectures to include in the module varies a bit depending on how many you think you’ll need for that topic.
4. Final lecture: Team activity. Working in their teams, students complete a written assignment in which they apply what they’ve learned in the module. Most but not all of these team activities are for marks (low stakes, as with the RATs).

The remaining lectures are given over to one-offs like start-of-term organization, review sessions before the midterm and final exams, etc.

Besides the RATs, team activities, midterm and final exams (and the labs), students also are marked on their performance as teammates. Halfway through the term, and again at the end, students do peer evaluations of their teammates and themselves. These contribute to their final mark by scaling their scores on the tRATs and team activities. Doing the first peer evaluations halfway is useful because it gives teams time to settle in, but gives students a chance to change their behavior if they receive negative peer evaluations.

How’s it working?

Pretty well, I think. Last term, my colleague Kyla taught the new structure for the first time. The class average went up by 10% even though the exams were similar. That’s a big improvement given that the class was doing fine under the old course structure. And so far this term, with me teaching the new structure for the first time, students seem to be doing well. But I haven’t marked the midterm yet, so it remains to be seen if the new structure is as effective with me at the helm as with Kyla at the helm. Kyla’s a better teacher than me. It’s quite possible that some of the apparent benefits of flipped classrooms in the pedagogical literature depend on them being implemented by excellent teachers. And of course, there’s often a fair bit of variation in student performance from one class to the next for reasons having nothing to do with how the class is taught. So my judgment that it’s working is just that—a professional judgment call, not a rigorous statistical inference. And of course, if it turns out not to work, I’ll try to improve it.

(UPDATE #2: I wrote this post right before the midterm. On which the class as a whole did very badly. They did ok on the final exam, but no better than they did when I was just lecturing. So overall, it turns out that flipping the class didn’t work for me this term. I have some ideas about why, and I’m going to try to improve it next time I teach the class. More on that in a future post But long-term, the jury’s still out on whether flipping the classroom will work any better than lecturing for me.)

Why is it working?

Several reasons, some of which Meg’s also talked about:

• Pedagogical research says that students and instructors should get frequent feedback on what students do and don’t know. The biggest obstacle to learning is students mistakenly thinking that they know something, when in fact they don’t. And getting quiz questions wrong creates teachable moments; students pay more attention to lectures if they know the lecture will be focusing on stuff they’ve just gotten wrong on a quiz.
• One of the best ways to learn is to explain your thinking to others. Even the best students benefit from this. In this class, a team’s tRAT score usually exceeds the iRAT scores of every individual on the team. Most every team gets 90-100% on the tRAT, even though only a few students get 90-100% on the iRAT. Talking to your classmates also gets you comfortable with technical vocabulary.
• It makes sense to spend class time on activities that can only be done with the instructor and all students present. Rather than lecturing on stuff the students could have learned just as well on their own outside of class.
• It makes sense to spend class time on what the students don’t know, rather than wasting time lecturing on stuff they already know.
• Many students these days struggle to pay attention to lengthy lectures, and to take good notes.
• The more students are actually using and applying what they’ve learned, the better they’ll retain it. It’s like learning a language. (Yes, the labs are also good for this. But the labs are also for teaching the students R, which is hard for many of them.)
• This structure didn’t oblige us to sacrifice much breadth for depth. Brian’s noted that a lot of teaching comes down to trade-offs between breadth and depth of coverage. But we didn’t sacrifice much material to move to this new structure. We only dropped the equivalent of one module from the old version of the course (axioms of probability).
• Putting the students in teams reduces the number of RATs and team activities to be marked.
• It forces students to spend more time on the class than they otherwise would. If it weren’t for the RATs, most students would not do the assigned readings. And if it weren’t for the RATs and team activities, some students (about 20% at Calgary) would not attend lectures, at least not without some other incentive to do so. This isn’t just speculation on my part. When we ask them, students report spending more time on this class than they do on other classes, and more time than they would if it were a conventional lecture.
• It forces students to keep up with the class, rather than falling behind during the term and then trying to cram for exams.

I’m not sure about the relative importance of these bullets, but I think the first two and the last two are the most important. Advocates of flipped classrooms tend to focus on how flipped classrooms increase the effectiveness of each unit of effort students put in. But in my admittedly-limited experience, another big reason flipped classrooms work is by forcing students to put in more effort than they otherwise would. Meg and I have discussed this a bit in old comment threads.

I know that forcing students to read the textbook and attend class isn’t every prof’s cup of tea. Some profs take the view that students are adults who should be left totally free to allocate their time as they see fit, suffering the consequences if they turn out to have made poor choices. That’s not a silly view, and those who hold it are not lazy and are not just rationalizing their own desire to keep lecturing. But personally, I’m fine with giving students strong incentives to allocate their time in a way that many of them wouldn’t otherwise choose ex ante. One reason I’m fine with it is because I think the students will be fine with it too after they complete the course. Even responsible adults sometimes want to be “nudged” toward making choices that they themselves would regard as the right choices ex post but not ex ante. The other reason I’m fine with it is because it’s my job to teach the course in the most effective way. Even if that’s not what the students might say they want at the time. My job is to help them master the material, not make them happy.

In practice, pedagogical approaches that increase the amount of effort the students put in might be difficult to separate from approaches that increase the effectiveness of each unit of effort. I don’t know how else you’d force the students to do all the readings and attend all the classes besides having lots of low-stakes quizzes on the readings before the material is covered in lecture. Such quizzes also give you and the students frequent feedback on what they do and don’t know, and enable you to focus your class time on what students don’t know. Thereby increasing the effectiveness of each unit of class time.

Any other details I should know if I want to try this myself?

Lots! The course as a whole is like a house of cards: it’s mutually supporting, and if you removed any piece the whole thing might fall apart. I don’t say that as a criticism; it’s just the way it is. Some things to be aware of:

• You need to maximize within-team diversity in terms of academic ability, relevant background knowledge and skills, where students fall on a leader-follower spectrum, and probably gender. We use a free online tool called CATME to do this. Don’t just assign students to teams randomly, and definitely don’t let students just team up with their friends.
• Students take a while to get the hang of working in teams, and to buy into the way the course works. Even though we spend much of the first two lectures explaining how the class will work and why, including showing them results from pedagogical research (sample: “Be honest, folks: how many of you would actually read the textbook if you weren’t being quizzed on it?” I got a lot of rueful smiles and you’ve-got-a-point nods when I told them that.) Even though we do a mock iRAT and tRAT on the course syllabus before the first real RAT. Even though we ask them about what they like and don’t like about team work, and tell them how we’ve designed the class to prevent the things they don’t like (e.g., we prevent “free riders” by having them write and sign team “contracts” spelling out what’s expected of everyone on the team and the penalties for failure to comply.) And even though we talk to them about the characteristics of effective teams, and of individuals who work as part of effective teams (e.g., effective teams come to a consensus, rather than just taking decisions via majority vote). Until they see it work, some of the strong students are going to believe that they could do just as well or better on their own. Some of the weak students who just want to pass the class are going to resent being forced to put in more effort than they’d like. Many students will complain that the RATs should be after the lectures rather than before (which of course would defeat a key purpose of the RATs: to force students to do the readings in a timely fashion, rather than just relying on the lectures). They don’t understand that the marks they feel they’re “losing” by having to take the RATs before the lectures are an investment that will pay off in spades on the midterm and final exam. And they’ll often fail to recognize how much they’re learning, because many of them will experience subjective feelings of struggling with the material and just being uncomfortable with the newness of a flipped classroom. Never mind recognizing how much more they’re learning than they would be learning in a traditional lecture! Some of them won’t “buy in” until they take the midterm and are pleasantly surprised at how easy they find it and how well they do. A few may never buy in.
• Teaching this course involves a lot of behind the scenes logistics. I can talk more about that in the comments if anyone’s curious.
• Every class session is scheduled in advance, and you basically can’t deviate from the schedule. No letting your lectures run long!
• Kyla tells me that team sizes of 5-7 are optimal. I buy that. Some teams are down to 4 this term because some students dropped the class, and they seem to be doing fine. But my gut says that teams of 2-3 would not work as well. And more than 7 would definitely be too big, I think.
• There are some technologies that make bits of the course easier and more effective. For instance, students take the iRATs by filling in bubbles on forms that are marked by scanning them. And they take tRATs on scratch cards much like instant-win lottery cards, which lets them see instantly if they got the correct answers or not.
• It’s difficult to write good team activities. The ideal team activity is one that’s not open ended (at least, not entirely), but that the whole team needs to work together to solve. Hopefully Kyla will comment on this, I know she’s thought and read a lot about it.
• Yes, team work might be uncomfortable at first for introverts. But here’s the thing: any way you chose to teach would be disliked by someone, or even work less-than-optimally for someone. This isn’t a matter of people having different “learning styles” (which aren’t a thing). It’s a matter of teaching in whatever way works best for the class as a whole. Would you rather have a few introverts feeling uncomfortable initially because they’re in teams, or lots of students feeling uncomfortable for the entire term because they’re not mastering the material?
• Recently, attendance at the mid-module lectures has been dropping. I’m mulling over whether to do something about that in future years, for instance by making a small part of the course mark dependent on answering a sufficiently-high fraction of clicker questions asked during the mid-module lectures. I used to ask more clicker questions during conventional lectures, back before we flipped the class. But I’ve cut back on clicker questions because under the new structure I find myself wanting to move quicker in the mid-module lectures to cover everything I feel I need to cover. That may be a mistake on my part.
• This term I had an unusually high number of students drop the class in the first week. We went from 121 students on the first day to 104 a week later. I’m not sure if those students dropped because they didn’t want to be in a team-based course, though. We didn’t have many drops last term, the first time we taught the new structure. And anecdotally, other courses in our department also had more drops than usual at the start of this term. But in any case, even if the course works this term, it’s possible that it will have worked in part because people for whom the course wouldn’t have worked dropped it.
• Prepping this for the first time was a massive amount of work. I was spoiled because Kyla took the lead on the new prep. Had it been my job to prep the new structure myself, I honestly don’t know if I would’ve gone to the new structure. And I say that even though I’m tenured and so wouldn’t face any risk to my job if I spent a lot of time prepping a new course, even if it turned out badly. Meg’s talked about how much time she had to spend prepping a flipped course the first time.
• Some teams are going to work better than others no matter what you do. For instance, if there’s a serious personality conflict within a team. Or if there’s a team on which everyone is content for a couple of team members to do all the work while everyone else free rides.
• In contrast to Meg’s experience, flipping the class does not seem to encourage more students to come to my office hours. If anything, just the opposite, though it’s hard to tell because it’s not as if many students came to my office hours under the old course structure. Flipping the class also seems to have reduced the number of students who take advantage of review sessions run by “peer mentors” (students who’ve taken the class in the past). I suspect that there’s some maximum number of hours that you can force a student to allocate to any given class, and that we’re running up against it. Students are spending sufficient time on the class already that they won’t choose to give any more by attending review sessions. But all this could just be a one-time blip, too.
• Lots of other details I’m happy to talk about in comments! Kyla will be following the comments as well and hopefully will chime in. She knows way more about this stuff than me! And in case it needs saying, I’m expressing my own views in this post. I’m not speaking for Kyla, and any mistakes are my bad.

UPDATE: Anyone who wants to tweet questions directly to Kyla, go ahead! 🙂

Follow-up post on how we flipped our bio-stats class #teambasedlearning
https://t.co/K0gYDRgjZp via @DynamicEcology

— Kyla Flanagan, PhD (@kyla_m_flanagan) March 9, 2016