We flipped our huge intro biostats course. Here’s how we did it. (UPDATEDx2)

In a previous post I talked about why we flipped the intro biostats course here at Calgary, even though the old course was working fine. Now I’ll talk about how we did it. If you haven’t read that previous post, please do so before reading this one. To decide if what we did might work for you as well, you definitely want to know why we did what we did, and about relevant context like the size and makeup of the class. And if you want to know how others teach intro biostats, see the comments here.

Overview: a team-based flipped classroom

I’ll first just summarize the course structure, then talk about why that structure works.

I should emphasize up front that this structure wasn’t my creation, though I’m liking it so far. My colleague Kyla Flanagan took the lead on developing the new structure, drawing in part on the experience of another Calgary instructor who’d used a similar structure in one of her classes.

The new course has a team-based structure. For the lecture sessions (50 minutes 3x/week), students are divided into permanent teams of 5-7 people (students work independently in the labs, which we didn’t revamp).

The course material is divided into 8 modules, each a few lectures in length. Each module covers a different topic or set of related topics. For instance, we have a module on Experimental Design, a module on ANOVA, a module on Regression and Correlation, etc. Each module follows the same sequence:

  1. Pre-module: before the first lecture of each module, students are assigned background readings, usually 1-3 chapters from the textbook. We also provide a reading guide summarizing the key things we expect them to get out of the readings (key terms and concepts, things we expect them to be able to do, etc.).
  2. First lecture: Readiness Assurance Tests (RATs). Yes, that’s an unfortunate acronym. The students take a quiz on the background reading, to assess their “readiness to learn”. It’s usually 10 multiple choice questions. The students first take the quiz individually; that’s the iRAT. Then they take the same quiz with their team, submitting one set of answers for the entire team; that’s the tRAT. Both the iRAT and tRAT are for marks, but they’re low stakes; they’re not worth a big fraction of the final grade either individually or collectively.
  3. Next 1-2 lectures: Lectures focused on those aspects of the topic with which the students are struggling or are likely to struggle, based on the instructor’s experience and the students’ answers on the iRAT and tRAT. Of course, the instructor needn’t just talk the whole time—the lectures can and do incorporate clicker questions, pair-and-share, etc. How many lectures to include in the module varies a bit depending on how many you think you’ll need for that topic.
  4. Final lecture: Team activity. Working in their teams, students complete a written assignment in which they apply what they’ve learned in the module. Most but not all of these team activities are for marks (low stakes, as with the RATs).

The remaining lectures are given over to one-offs like start-of-term organization, review sessions before the midterm and final exams, etc.

Besides the RATs, team activities, midterm and final exams (and the labs), students also are marked on their performance as teammates. Halfway through the term, and again at the end, students do peer evaluations of their teammates and themselves. These contribute to their final mark by scaling their scores on the tRATs and team activities. Doing the first peer evaluations halfway is useful because it gives teams time to settle in, but gives students a chance to change their behavior if they receive negative peer evaluations.

How’s it working?

Pretty well, I think. Last term, my colleague Kyla taught the new structure for the first time. The class average went up by 10% even though the exams were similar. That’s a big improvement given that the class was doing fine under the old course structure. And so far this term, with me teaching the new structure for the first time, students seem to be doing well. But I haven’t marked the midterm yet, so it remains to be seen if the new structure is as effective with me at the helm as with Kyla at the helm. Kyla’s a better teacher than me. It’s quite possible that some of the apparent benefits of flipped classrooms in the pedagogical literature depend on them being implemented by excellent teachers. And of course, there’s often a fair bit of variation in student performance from one class to the next for reasons having nothing to do with how the class is taught. So my judgment that it’s working is just that—a professional judgment call, not a rigorous statistical inference. And of course, if it turns out not to work, I’ll try to improve it.

(UPDATE #2: I wrote this post right before the midterm. On which the class as a whole did very badly. They did ok on the final exam, but no better than they did when I was just lecturing. So overall, it turns out that flipping the class didn’t work for me this term. I have some ideas about why, and I’m going to try to improve it next time I teach the class. More on that in a future post But long-term, the jury’s still out on whether flipping the classroom will work any better than lecturing for me.)

Why is it working?

Several reasons, some of which Meg’s also talked about:

  • Pedagogical research says that students and instructors should get frequent feedback on what students do and don’t know. The biggest obstacle to learning is students mistakenly thinking that they know something, when in fact they don’t. And getting quiz questions wrong creates teachable moments; students pay more attention to lectures if they know the lecture will be focusing on stuff they’ve just gotten wrong on a quiz.
  • One of the best ways to learn is to explain your thinking to others. Even the best students benefit from this. In this class, a team’s tRAT score usually exceeds the iRAT scores of every individual on the team. Most every team gets 90-100% on the tRAT, even though only a few students get 90-100% on the iRAT. Talking to your classmates also gets you comfortable with technical vocabulary.
  • It makes sense to spend class time on activities that can only be done with the instructor and all students present. Rather than lecturing on stuff the students could have learned just as well on their own outside of class.
  • It makes sense to spend class time on what the students don’t know, rather than wasting time lecturing on stuff they already know.
  • Many students these days struggle to pay attention to lengthy lectures, and to take good notes.
  • The more students are actually using and applying what they’ve learned, the better they’ll retain it. It’s like learning a language. (Yes, the labs are also good for this. But the labs are also for teaching the students R, which is hard for many of them.)
  • This structure didn’t oblige us to sacrifice much breadth for depth. Brian’s noted that a lot of teaching comes down to trade-offs between breadth and depth of coverage. But we didn’t sacrifice much material to move to this new structure. We only dropped the equivalent of one module from the old version of the course (axioms of probability).
  • Putting the students in teams reduces the number of RATs and team activities to be marked.
  • It forces students to spend more time on the class than they otherwise would. If it weren’t for the RATs, most students would not do the assigned readings. And if it weren’t for the RATs and team activities, some students (about 20% at Calgary) would not attend lectures, at least not without some other incentive to do so. This isn’t just speculation on my part. When we ask them, students report spending more time on this class than they do on other classes, and more time than they would if it were a conventional lecture.
  • It forces students to keep up with the class, rather than falling behind during the term and then trying to cram for exams.

I’m not sure about the relative importance of these bullets, but I think the first two and the last two are the most important. Advocates of flipped classrooms tend to focus on how flipped classrooms increase the effectiveness of each unit of effort students put in. But in my admittedly-limited experience, another big reason flipped classrooms work is by forcing students to put in more effort than they otherwise would. Meg and I have discussed this a bit in old comment threads.

I know that forcing students to read the textbook and attend class isn’t every prof’s cup of tea. Some profs take the view that students are adults who should be left totally free to allocate their time as they see fit, suffering the consequences if they turn out to have made poor choices. That’s not a silly view, and those who hold it are not lazy and are not just rationalizing their own desire to keep lecturing. But personally, I’m fine with giving students strong incentives to allocate their time in a way that many of them wouldn’t otherwise choose ex ante. One reason I’m fine with it is because I think the students will be fine with it too after they complete the course. Even responsible adults sometimes want to be “nudged” toward making choices that they themselves would regard as the right choices ex post but not ex ante. The other reason I’m fine with it is because it’s my job to teach the course in the most effective way. Even if that’s not what the students might say they want at the time. My job is to help them master the material, not make them happy.

In practice, pedagogical approaches that increase the amount of effort the students put in might be difficult to separate from approaches that increase the effectiveness of each unit of effort. I don’t know how else you’d force the students to do all the readings and attend all the classes besides having lots of low-stakes quizzes on the readings before the material is covered in lecture. Such quizzes also give you and the students frequent feedback on what they do and don’t know, and enable you to focus your class time on what students don’t know. Thereby increasing the effectiveness of each unit of class time.

Any other details I should know if I want to try this myself?

Lots! The course as a whole is like a house of cards: it’s mutually supporting, and if you removed any piece the whole thing might fall apart. I don’t say that as a criticism; it’s just the way it is. Some things to be aware of:

  • You need to maximize within-team diversity in terms of academic ability, relevant background knowledge and skills, where students fall on a leader-follower spectrum, and probably gender. We use a free online tool called CATME to do this. Don’t just assign students to teams randomly, and definitely don’t let students just team up with their friends.
  • Students take a while to get the hang of working in teams, and to buy into the way the course works. Even though we spend much of the first two lectures explaining how the class will work and why, including showing them results from pedagogical research (sample: “Be honest, folks: how many of you would actually read the textbook if you weren’t being quizzed on it?” I got a lot of rueful smiles and you’ve-got-a-point nods when I told them that.) Even though we do a mock iRAT and tRAT on the course syllabus before the first real RAT. Even though we ask them about what they like and don’t like about team work, and tell them how we’ve designed the class to prevent the things they don’t like (e.g., we prevent “free riders” by having them write and sign team “contracts” spelling out what’s expected of everyone on the team and the penalties for failure to comply.) And even though we talk to them about the characteristics of effective teams, and of individuals who work as part of effective teams (e.g., effective teams come to a consensus, rather than just taking decisions via majority vote). Until they see it work, some of the strong students are going to believe that they could do just as well or better on their own. Some of the weak students who just want to pass the class are going to resent being forced to put in more effort than they’d like. Many students will complain that the RATs should be after the lectures rather than before (which of course would defeat a key purpose of the RATs: to force students to do the readings in a timely fashion, rather than just relying on the lectures). They don’t understand that the marks they feel they’re “losing” by having to take the RATs before the lectures are an investment that will pay off in spades on the midterm and final exam. And they’ll often fail to recognize how much they’re learning, because many of them will experience subjective feelings of struggling with the material and just being uncomfortable with the newness of a flipped classroom. Never mind recognizing how much more they’re learning than they would be learning in a traditional lecture! Some of them won’t “buy in” until they take the midterm and are pleasantly surprised at how easy they find it and how well they do. A few may never buy in.
  • Teaching this course involves a lot of behind the scenes logistics. I can talk more about that in the comments if anyone’s curious.
  • Every class session is scheduled in advance, and you basically can’t deviate from the schedule. No letting your lectures run long!
  • Kyla tells me that team sizes of 5-7 are optimal. I buy that. Some teams are down to 4 this term because some students dropped the class, and they seem to be doing fine. But my gut says that teams of 2-3 would not work as well. And more than 7 would definitely be too big, I think.
  • There are some technologies that make bits of the course easier and more effective. For instance, students take the iRATs by filling in bubbles on forms that are marked by scanning them. And they take tRATs on scratch cards much like instant-win lottery cards, which lets them see instantly if they got the correct answers or not.
  • It’s difficult to write good team activities. The ideal team activity is one that’s not open ended (at least, not entirely), but that the whole team needs to work together to solve. Hopefully Kyla will comment on this, I know she’s thought and read a lot about it.
  • Yes, team work might be uncomfortable at first for introverts. But here’s the thing: any way you chose to teach would be disliked by someone, or even work less-than-optimally for someone. This isn’t a matter of people having different “learning styles” (which aren’t a thing). It’s a matter of teaching in whatever way works best for the class as a whole. Would you rather have a few introverts feeling uncomfortable initially because they’re in teams, or lots of students feeling uncomfortable for the entire term because they’re not mastering the material?
  • Recently, attendance at the mid-module lectures has been dropping. I’m mulling over whether to do something about that in future years, for instance by making a small part of the course mark dependent on answering a sufficiently-high fraction of clicker questions asked during the mid-module lectures. I used to ask more clicker questions during conventional lectures, back before we flipped the class. But I’ve cut back on clicker questions because under the new structure I find myself wanting to move quicker in the mid-module lectures to cover everything I feel I need to cover. That may be a mistake on my part.
  • This term I had an unusually high number of students drop the class in the first week. We went from 121 students on the first day to 104 a week later. I’m not sure if those students dropped because they didn’t want to be in a team-based course, though. We didn’t have many drops last term, the first time we taught the new structure. And anecdotally, other courses in our department also had more drops than usual at the start of this term. But in any case, even if the course works this term, it’s possible that it will have worked in part because people for whom the course wouldn’t have worked dropped it.
  • Prepping this for the first time was a massive amount of work. I was spoiled because Kyla took the lead on the new prep. Had it been my job to prep the new structure myself, I honestly don’t know if I would’ve gone to the new structure. And I say that even though I’m tenured and so wouldn’t face any risk to my job if I spent a lot of time prepping a new course, even if it turned out badly. Meg’s talked about how much time she had to spend prepping a flipped course the first time.
  • Some teams are going to work better than others no matter what you do. For instance, if there’s a serious personality conflict within a team. Or if there’s a team on which everyone is content for a couple of team members to do all the work while everyone else free rides.
  • In contrast to Meg’s experience, flipping the class does not seem to encourage more students to come to my office hours. If anything, just the opposite, though it’s hard to tell because it’s not as if many students came to my office hours under the old course structure. Flipping the class also seems to have reduced the number of students who take advantage of review sessions run by “peer mentors” (students who’ve taken the class in the past). I suspect that there’s some maximum number of hours that you can force a student to allocate to any given class, and that we’re running up against it. Students are spending sufficient time on the class already that they won’t choose to give any more by attending review sessions. But all this could just be a one-time blip, too.
  • Lots of other details I’m happy to talk about in comments! Kyla will be following the comments as well and hopefully will chime in. She knows way more about this stuff than me! And in case it needs saying, I’m expressing my own views in this post. I’m not speaking for Kyla, and any mistakes are my bad.

UPDATE: Anyone who wants to tweet questions directly to Kyla, go ahead! 🙂

14 thoughts on “We flipped our huge intro biostats course. Here’s how we did it. (UPDATEDx2)

  1. So since nobody else is commenting, I’ll give myself a pity comment. 🙂

    I wish the course did a better job of teaching students how to come up with a question and choose a statistical analysis. Students who come out of the course can figure out what statistical analysis is appropriate if we describe a study and its scientific question and tell them what data the study collected. If we describe an experiment to them, they can identify flaws in the design. Etc. But down the road, students who’ve been through the course often still struggle to figure out what experiment one might do to answer a scientific question and what data one would collect from that expt.

    Any ideas on how to teach this? We do have one team activity that tries to teach this, but no single team activity is going to make much difference on its own. So maybe if we’re serious about teaching this we’d have to cut back significantly on other material to make room for many such activities?

  2. A couple of additions:
    – We used Instant Feedback Assessment Technique (IF-AT) cards for the team quizzes (http://www.epsteineducation.com/home/about/). These are the ‘scratch and win cards’ Jeremy mentioned students use for the team quizzes. Students loved these! It’s like winning the lottery every time you get a question correct. I’ve never heard cheering and clapping during a quiz before, but this is common using these. Students know if they got the correct answer on the first try and they can keep trying (for fewer marks) until they get to the correct answer (hence the “instant feedback”). So they are really LEARNING during the quiz. Also, I think these help build team morale as students really do get excited when they get the answer correct.
    – Much of the format we used was from the Team Based Learning Collaborative (http://www.teambasedlearning.org/). Just to give credit where credit is due….I certainly didn’t develop this method. 🙂
    – Having Peer Mentors (more information on our model: https://peermentoring.ucalgary.ca/) in class during the team activities and quizzes, is a HUGE help. It’s hard for me to get from team to team if there are a lot of questions, so extra help is a life-saver.

    Teaching in this way was the most rewarding educational experience I’ve had. I knew almost all the students by name, there was a great collaborative attitude in the class, and I think students learned more than I’ve seen in any other semester before.

    • Ooh, I’d love to hear more about the peer mentors! On my list of “blog posts to write” is one about how I was an undergrad TA for an autotutorial intro bio course, and my thoughts for how to use undergrad mentors/TAs more in my current intro bio course. The two main uses I see right now are: 1) to have them circulate around to help with active learning exercises in class (usually ~280 students attend my first lecture and ~180 attend the second, based on clicker responses), and 2) to do oral quizzing of students to help them learn material and learn what they need to study more. It sounds like the first option is similar to what you were doing, and I’d love to hear more about how it works. What training do you give to the peer mentors? One fear of mine is what the effect would be if one of the peer mentors was confused about a topic.

      • I’ll let Kyla say more, but briefly:

        -Our peer mentors are students who’ve taken the course before and are now taking a peer mentoring course offered by a different department. That peer mentoring course is where most of their training comes from.

        -The peer mentors are trained (and instructed by Kyla and I) *not* to teach content. They’re supposed to be facilitating learning, not teaching. In practice, that’s sometimes a fuzzy and difficult line for them to walk. There have been years where I’ve had to intervene to stamp out confusion among the students that’s been created by a peer mentor saying something incorrect. It helps if the peer mentors have taken the class recently (so it’s fresh in their minds), and if they did very well in the class.

      • Yes, we have a curricular peer mentoring program that is university wide. The student take a for-credit (and a grade!) course to learn about pedagogical theory and how to facilitate small groups. Then they have 3 hours of practicum a week, where they work in their host class helping students learn! I supervise their practicum aspect and therefore usually have the Peer Mentors meet with me weekly, to plan the team activities and RATs.Then they are available to help in class. I try to encourage the Peer Mentors to show students how to tackle problems, how to figure out where they are getting stuck, and how to get the team discussing a problem, rather than just telling students the correct answer. It is a learning curve for the mentors throughout the semester, but they have been so valuable to me. I also use them to get feedback about how they think the class is going. I run the assignment questions by them to see if they are clear, or if there are spots where students are going to get stuck. The Peer Mentors really help me develop/design the course and I really value their perspective as students that just completed my course.

  3. Thanks for sharing your experience! I’ve been pondering this for a while; been doing a hybrid lecture and team-based learning style for my Into to Environmental Sciences course. I’d be curious to hear more about how to write good team projects. Mine are different – they work on 3 projects throughout the semester and have 25 days to complete each one and get some time in class to do so – but I specifically make mine open-ended so they can choose what interests them the most. e.g. create a program on campus that makes it more sustainable, identify a top predator in peril and discuss a bunch of things related to that.

    • Open ended, extended team projects are quite different from the sort of team activities we’re doing, so I can’t offer much advice. I’ve only once taught a class in which students worked in pairs on an extended (6 week) project. And even then it wasn’t totally open ended. In my case, they had a choice of Canadian species at risk for which recovery plans exist, and they had to evaluate those recovery plans. We broke this down into a series of steps (e.g., the first week, they had to do a presentation summarizing the recovery plan for their chosen species.).

  4. So many thoughts on this post! I’m going to go with a list:
    1. I love the idea of having them do the quizzes on their own first and then as a group. It’s not surprising that they do better as a group, but it’s still great to hear that. Based on the writing (point 2 in your overview section), it sounds like that takes the whole class period. Is that right? Or do you then move on to do other material in that first lecture period?

    2. Struggling to take good notes: I know this isn’t a main point of your post, but it’s one I’ve been thinking about. I’ve been thinking about how, if I moved away from powerpoint/prepared slides to ones that I write out in class, would it really help? Yes, research shows that writing notes by hand is better than typing them or just receiving them, but that’s because writing by hand forces the student to think about the material and process it as they write it (since they can’t write quickly enough to just transcribe what the instructor said). But, if I write out material, they write out exactly what I wrote, in exactly the way I wrote it. If I say something important but don’t write it, most students don’t write it. So is me writing out notes that they then transcribe exactly really effective? I don’t really know a way around this problem.

    3. Only 20% of students wouldn’t attend lectures without the RATs? That seems like an impressively high attendance rate to me! I think we have about 20% who don’t show on any given day and that’s with giving them (optional) participation points for coming.

    4. I’d be interested in knowing more about how CATME makes groups (but haven’t gone to look it up myself yet). Sometimes you actually want to reduce diversity in a group – e.g., if there is a strong gender skew in a class, you don’t want a bunch of groups with just one woman; rather, you’d want some groups with all men and some groups with multiple women. And what demographic data do you have on your students when they enroll? Most of this is protected at Michigan, so I’m not sure how we would be able to create groups that were cognizant of this.

    5. I would love to hear more about the behind-the-scenes logistics, and on what you aim for in the activities!

    Okay, I think that’s enough for now. 🙂

    • Re: 1, yes, that takes the whole class. We give the students 10-12 minutes individually, and then the remaining time to retake the quiz with their team. Many (not all) teams finish early, and are free to leave if they do. Don’t have any great ideas as to what to do about that. If you just made the quizzes longer, some teams would be rushed.
      2. Kyla and I have different lecturing styles. She only uses bare-bones “skeleton” power point slides and writes on them in class with an electronic stylus. I tried that but it didn’t work for me, I found it awkward. Old dog, new tricks, apparently. 😦 So I just do straight power point in 315. In other classes I just use the chalkboard–keeps me from going to fast since I can’t write any faster than they can. But I can’t do that in 315, there are too many graphs and diagrams I need to show.
      3. Clearly Calgary students are more dedicated than the slackers you admit to Michigan. 🙂 Although actually, I think it was probably more like 25-30% who wouldn’t attend if there were no incentive to do so. Increasing to about 90% attendance when 5% of the course mark depended on participation points (as it did for a while under the old course structure). And close to 100% attendance in the new structure on the days when there’s a RAT or team activity.
      4. I don’t know exactly what CATME’s group assignment algorithm is. You can tweak it by telling it how much weight to put on different variables. But both Kyla and I usually end up taking the CATME-generated group assignments and then tweaking them by hand. Re: demographic data, I don’t recall that we use any. And in any case, you can customize (to some extent) what variables you want CATME to use.

    • Re: 5, a lot of it is photocopying and stuffing file folders. Each team gets a file folder, with their names and a team selfie on it (helps the instructor learn names, though I confess I’m still terrible at that). For every RAT, each folder has to be stuffed with photocopies of the quiz for every team member, scantron sheets for every team member, and a scratch card. For every team activity, each folder has to be stuffed with several photocopies of the activity (I usually do 1 copy/2 team members).

      I’m crap at this because I’m disorganized. I often forget to send the copy jobs to the campus printing centre at least 2 business days in advance. With the result that I’ve twice had to print them out at home the night before and burn through a bunch of my own printer cartridges. This of course says something about me, not about the course structure! 🙂

      There’s also a lot of one-off logistics at the beginning of the term. Getting all the students to promptly complete the CATME info so they can be assigned to teams. Drawing a map of the room with team seating assignments, that you can project so that all the teams know where to sit (it’s chaos if you let teams try to find their own seating areas). Setting up the teams in the course management software (a task that is extremely painful in our course management software, Desire2Learn).

      And of course, writing all those quizzes and team activities! (Thank you again for taking the lead on that, Kyla!)

  5. Pingback: Flipping our big intro biostats class didn’t work (for me) (yet) | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.