As I’ve written about before, we’ve moved Intro Bio towards a flipped model: students have to do pre-readings (and sometimes watch videos) prior to coming to class, they get quizzed before every class (on both the pre-readings and material from the earlier class), and we’ve incorporated much more active learning into the classroom (e.g., clickers, drawing and interpreting figures). I am 100% certain that my students learned more with this format. I haven’t yet done a formal analysis of the Bloom’s taxonomy levels of exam questions, but I know I was writing many more questions that required application, analysis, synthesis, and evaluation than I did in the past, and the mean exam grades stayed about the same. In other words, the exams were harder but the students did just as well.
So why the hesitation to recommend this approach? Part of it is the personal cost in terms of how much effort it took on my part. Last semester was pretty stressful for me. But part of it is also because I don’t think all that effort will be rewarded in terms of increased student evaluations. In my case, my teaching evaluations last semester were almost identical (within a few hundreths of a point on a 5 point scale) to what they were when I taught with a much more traditional format. Perhaps I should consider myself lucky that evaluations didn’t decrease, because evidence suggests that, all else being equal, more effective teaching strategies leave students frustrated.
I’ve been thinking about this as I read through Make It Stick, by Peter Brown, Henry Roediger III, and Mark McDaniel. I’m still reading it, but I’ve found it to be a really engaging read related to how to best promote learning. In what I’ve read so far, I’ve been struck by statements like “When learning is harder, it’s stronger and lasts longer” (page 9). I was especially struck by this passage where they compare how interleaved learning (where students move between tasks before fully mastering any one of them) versus massed practice (where students practice-practice-practice one skill, getting that down before moving on to learning the next skill):
The learning from interleaved practice feels slower than learning from massed practice. Teachers and students sense the difference. They can see that their grasp of each element is coming more slowly, and the compensating long-term advantage is not apparent to them. As a result, interleaving is unpopular and seldom used. Teachers dislike it because it feels sluggish. Students find it confusing: they’re just starting to get a handle on new material and don’t feel on top of it yet when they are forced to switch. But the research shows unequivocally that mastery and long-term retention are much better if you interleave practice than if you mass it.
– Make It Stick, page 50
For example, if you teach a taxonomy course, you might think it makes sense to drill students on one group, having them get that down before moving on to the next, but research (including a study on learning bird taxonomy) shows that interleaving the different groups is more effective since it allows students to figure out what traits are common between the groups and which allow you to distinguish them (see chapter 3 in Make It Stick for more discussion of this.) The problem is that:
Despite these results, the students who participated in these experiments persisted in preferring massed practice, convinced that it served them better. Even after they took the test and could have realized from their own performance that interleaving was the better strategy for learning, they clung to their belief that [massed practice] was better. The myths of massed practice are hard to exorcise, even when you’re experiencing the evidence yourself.
– Make It Stick, page 54
For me, I plan to increase the amount I use interleaved learning (and the other practices summarized in Make It Stick); thanks to having tenure, I don’t have to worry nearly as much about potential negative impacts on my teaching evaluations if students are frustrated and unable to accurately evaluate that they are learning more. (I would, of course, prefer that my students enjoy my class and like me as an instructor, but it is more important to me to be an effective teacher than a popular one.)
But I am now in the position of advising junior faculty. Should I recommend to them that they use they use this approach? Even aside from the personal costs mentioned in my earlier post, I’m not sure. It might be in their best interest for them to use a more-or-less traditional approach, and take baby steps towards active learning. At a research university, teaching evaluations are not the most important factor in a tenure dossier, but they are considered. Life is easier when teaching evaluations are good. In this case, I struggle with what to recommend because I have an obligation to look out for the best interests of my junior colleagues, but also for the students they teach, and what is best for them seems at odds. The calculus is probably even more complicated for faculty whose primary responsibility is teaching.
For those of you who are pretenure: how do you weigh the costs and benefits of different pedagogical techniques? Does fear of lower teaching evaluations keep you from using more active techniques, and especially something like interleaving? For those of you in the position of mentoring folks who are pre-tenure: what do you recommend? How do you resolve the conflict to your junior colleagues and the students at your university? And for those who have teaching-intensive, non-tenure-track positions: how do you balance the costs and benefits? How do you think they differ compared to those of folks on the tenure track?
Dynamic Ecology 
As junior faculty, we are all encouraged to do a flipped classroom, along with the senior folks. I think its a good idea for a few reasons. One is that I am writing my lectures from scratch so the incremental work of this approach is not so much higher. The other that if it’s better, as the evidence seems to suggest then why start off with a suboptimal approach.
There is some trepidation because my classes as a trainee were not flipped, but that’s the case for almost everyone. I think you are suggesting that this is a higher variance approach, which might be true, I’m just not sure that’s enough reason to be safe and (potentially) worse.
I think that students learn more, but that they might like it less. So, to the extent that student evaluations are considered for tenure decisions, that would be a concern.
I have found that communicating to your students the reasons behind the change in methodology goes a long way to getting them on board with innovation. You won’t convince everyone, but the majority of my students have been happy with a (more-or-less) flipped approach to Intro Bio, at least judging by my course evals. I also recommend polling them during the semester to get feedback about what is working and what isn’t. This not only gives you a chance to address issues during the course, it also tends to improve student satisfaction– and thus evals– before its too late.
Agree! Students are far more open to different teaching strategies when it’s explained both in the first (few) lectures and on the course outline. And, I suspect, such explanations can probably offset some of the (potentially) negative teaching evaluation scores.
Yes, a colleague at another university apparently spent a *lot* of time cheerleading the approach when he flipped his classroom. He said it took the whole semester to get them on board. We explained in the syllabus why we were doing things the way we were, and covered this a bit during the semester, but covering it more would probably be useful.
Good post: I like the honesty and candid approach. And if, I’m to be completely honest, I’m far more willing to take risks as a teacher, post-tenure. But, at the same time, trying different approaches to teaching is so rewarding for everyone (teacher, student), and although some don’t work all that well, many do; it’s how we grow as instructors. Perhaps I way to take a step in this direction, pre-tenure, is to consider some simple active learning strategies in the classroom: devote part of each lecture to carefully designed discussions, pair & share activities, etc. Many active learning strategies don’t take too much time or effort, and certainly are not as ‘risky’ as flipping the classroom. In other words, potentially a good choice for pre-tenure.
Yes, I completely agree with this strategy (at least for big lecture courses). This is pretty much the approach I used — adding in clickers, think-pair-share prior to tenure, flipping after tenure. That timing wasn’t entirely planned related to tenure, but I think it’s convenient that it worked out that way.
@davebridges, the mean evaluation score is lower (I don’t know about the variance). Part of the problem is that students are very poor judges of how successful their learning is, and simply telling them “This works better” doesn’t change their real feeling that frustration=failure=the instructor’s fault.
I’m curious about the analysis that suggests that interleaved learning has an advantage over massed learning. I have zero evidence in either direction to suggest that it’s wrong or right (and I don’t have the book, which is part of why I’m asking), but I’m curious whether the evidence really supports the conclusion.
At least on the surface, an alternative interpretation is that increased effort produces superior results, and that forcing students into an unfamiliar learning modality induces them to expend greater effort – i.e., that it’s not the specifics of interleaved learning that produce better outcomes, but rather simply that it’s different.
Again, I have no evidence for, or against any conjecture regarding this, I’m simply curious. “Different” frequently produces results that are “better” at the outset, and then those benefits disappear once it’s not different anymore.
(An alternative, alternative conjecture, for which I do have some interest in the validity, is that it’s not the interleaving that’s beneficial, it’s the pacing. In practically every _manual_ skill task that can be learned, it is definitely the case that over-practice is not beneficial, and that short, intense practice sessions with adequate time for the “body” to learn before going at it again, produces superior results. I don’t know the education/cognition literature, so I don’t know if the same holds true for more typical classroom education. If it does, it’s at least possible that the benefits being seen from interleaved learning are actually due to the change of pacing in the material, and that massed learning provided at the pace of a singular topic within interleaved learning, is superior to either, and/or that the specifics of what is interleaved with what, can produce a significant impact upon the overall benefits of interleaving.)
The studies they report on in Make It Stick appear to control for effort. For example, they report about one study where all baseball players at a university took the same amount of batting practice. One half were given the standard batting practice, which is a version of massed practice: they were given a bunch of fastballs in a row, then a bunch of curveballs, then a bunch of changeups. (I’m doing this from memory, so it’s entirely possible I have the order wrong!) The other group was thrown the same number of pitches, but the pitches were interchanged at random. The players who got the interleaved pitches were frustrated and didn’t feel like they were improving as much, but their actual performance in games improved much more than the massed practice group. (Jeremy, have I piqued your interest now with a baseball example? 😉 )
Some of the specific studies on interleaving that they cite are:
Click to access CepedaPashlerVulWixtedRohrer-PB-2006.pdf
Click to access Rohrer%26Taylor2007IS.pdf
http://pss.sagepub.com/content/19/6/585.abstract
Two others that I found but haven’t read yet and that aren’t covered in Make It Stick are:
Click to access Birnbaum_Kornell_EBjork_RBjork_inpress.pdf
Click to access ED536925.pdf
Thanks Meghan,
I’m quite far a field from you folks – a strange gemish of Biophysics and Visual Analytics here – but I’m constantly fascinated by the discussions over here on DynamicEcology.
You’ve given me some more background reading and thinking to do. The Visual Analytics side of my brain is always looking for, and frequently suspicious of, things that “get information into people’s heads” more effectively. There’s so much that looks like promising new approaches, that later turn out to only be transiently better, with initial promising results deriving from novelty rather than a fundamental benefit. When something that really is better comes along, it’s always a treat.
I would like to add two points. 1) Although flipping a class de novo is a tremendous amount of work, as you described, if the pioneers are willing to share their work, junior faculty won’t have to reinvent the wheel, and should be able to adopt the flipped techniques with much less work. 2) Evaluation of faculty teaching MUST go beyond student evaluations. As someone charged with coordinating teaching evaluations, I make sure to note and praise faculty who do put in effort and are willing to take risks.
+1 on point #2! The attitude to teaching evaluations has varied from campus to campus. But I’ve seen places where they’re taken way too far. At some liberal arts colleges people are basically told to teach to maximize their student evaluation scores to get tenure (resulting in very easy courses, spoon feeding, no innovation and I suspect an extreme lack of learning). And I still recall my teaching evaluations when I taught intro bio – the majority were pretty darn shallow (lots of comments about movies and my attire). Upper level evaluations are more valuable. But studies have shown the single best predictors of student evaluation scores are class size and whether the course is required or not. Controlled experiments with teacher switching have shown that teacher effects are below the other two factors. So student evaluation scores are analogous to impact factors – correlated with what you’re trying to measure but a very lazy way to measure. There is no substitute for peer teaching evaluations but they do take time.
At Calgary the collective agreement with the faculty union specifies that student teaching evaluations may not be the sole means by which teaching is evaluated.
@Jeremy the same at Maine – a combination of having a union and being a research university I think.
Its actually a subset of the 4 year colleges that claim to value teaching quality so highly that get the most carried away with student evaluations in my experience. Its really a thin veneer over give the students what they want so we can charge them a lot. And again I emphasize this is not universal but definitely exists at places that you would think would know better. I don’t know how general it is.
Brian – I’d love to hear more comments on your perception of student evaluations at liberal arts colleges. I think of these schools as the epitome of what teaching-scholarship should be. But maybe I live in a naive bubble – having neither attended nor taught at a liberal arts college.
@Jeff – all my information is anecdotal (I also neither attended nor taught at one).
And its clearly a mixed bag. It is evident for example on smallpond how teaching is valued at a serious level. And I know of a number of core contributors to the teaching sessions at ESA that come from liberal arts college. But I also know of more than one story at top 30 liberal arts institutions in the US where student evaluations run riot. Nobody flips their classes because it is risky and not always well received by students. People dumb down their classes to get better evaluations and are careful to “not expect too much”. etc. And when I hear these stories I think there is a certain freedom that comes from being at a research university where as long as you’re not bombing you can do anything you want. That freedom can and is being abused of course But I think it might have benefits to allow risk taking too.
The original innovator in this area was Mazur at Harvard (an ultimate research focused university) although if you trust Wikipedia’s history of flipping many of the ensuing champions were at 4 year schools (but not the really big name ones). Bottom line is I don’t know but its not a given to me that top 4 year colleges are the most focused on “true teaching quality” as opposed to student evaluations whatever those two terms mean*. I would be really curious to hear others experiences. A survey would be even more interesting but its kind of a hard thing to get an honest survey answer on.
*not to deny other obvious benefits of smaller four year schools like class sizes that are undeniable.
Yes, I agree with both points. Regarding 1: the people teaching Intro Bio here have all been sharing resources, and it should really help new people who rotate in. As far as I know, Genetics has been flipped, but not Ecology or Evolution. So, for someone coming in to teach Ecology, those resources wouldn’t exist here.
For the second: most definitely. I had a post on this planned, and hope to get it written this week. I’ve been fortunate that both at GT and UMich, there were means of evaluating teaching beyond student evaluations, and there was real support for innovative teaching. But, still, low teaching evaluations would have made me very nervous while pre-tenure.
For the first few years on the tenure-track teaching was mostly about surviving the experience. Now that I have several years of teaching under my belt I’m more comfortable integrating active learning into my class. I’ve toyed with the idea of doing a flipped classroom, but I’ll likely take the approach of flipping a single class slot or module rather than trying to do a whole course all at once. Several people in another department have flipped their classrooms, but in one case the person devoted their entire sabbatical to setting it up.
Wow! And here I thought I was focusing on teaching more during my semester of leave than I should have! You approach sounds really reasonable to me.
Pingback: Idiosyncracies of Academia – Part 1 in a Never-Ending Series.. |
Like the students, I’m somewhat surprised that ‘interleaved’ learning is *better* than ‘massed’ — simply because people learn more when they are happy learning as opposed to being frustrated learning. I think it matter whether your students are frustrated with the material, even if they’re learning a lot. If they’re frustrated, perhaps they’re less likely to pursue similar courses going forward. “I hate science. It’s so frustrating.” Test scores may not be the best way to measure whether one approach is better than another. Just being devil’s advocate here.
This is an excellent point, and one I hadn’t considered. I wonder if it’s been studied? I’m not sure. It relates to a conversation I had yesterday with a colleague who has also been reading Make It Stick. She said she got about 90% of the way through, then stopped because it was frustrating how it kept looping through different material, jumping around. I laughed because she was frustrated by the interleaving — just as the research suggests!
I thought I would add in my 2 cents here as a new faculty member (just started Fall 2014) at a teaching oriented institution. As a postdoc, I was fortunate enough to co-teach with a senior lecturer who was an early adopter of the flipped-classroom, active learning lecture style for Intro Bio at a large R1 (~1100 students). It was an incredibly informative experience, I borrowed heavily from him, not only content (slides, clicker questions, worksheets) but also in-class techniques (eg. how do you respond when you pose a question and the student gives a totally wrong, off the wall answer?). I think postdocs (who wish to continue in academia) should be the target audience for adoption of flipped classrooms because they are building their courses from scratch. If you are building a course from the ground up, it is just as much work to create all your lecture slides as it is to create active-learning style slides.
However, that being said, I think this is only feasible if there is sharing of materials (not reinventing the wheel) between people have use a flipped classroom approach with new adoptees. Having a good base of questions to draw from, vetted worksheets and exercises, etc. are all invaluable for young faculty trying to learn to teach and stay afloat with their demanding workload. Because I was fortunate enough to have access to this wealth of resources, I have been able to retool these techniques for new classes that I’m teaching without it being too onerous. It is a large initial investment, but the earlier you start, the sooner you can start reaping the rewards… even pre-tenure.
How to handle it when a student gives a totally off-the-wall answer is something I definitely struggle with!
I would love it if there was a good pool of questions to draw from, and considered what would be involved in creating and maintaining such a database. But an issue I ran into this year is that it is so hard to have generally useful questions. I had access to a lot of questions that someone else had written, but found I couldn’t use many of them because:
1. I was covering material in a different order, so many of the distractor answers made no sense to students,
2. I was covering different material, again causing problems with the distractor answers, and
3. I wanted the quiz question style to match my personal style, so that they’d be good practice for exams.
All of that meant that writing the questions was a lot more work than I anticipated. It will certainly be less work next time, but I am sure I will still do lots of tweaking.
So, I very much agree that sharing of resources makes it more feasible, but it didn’t help me as much as I thought it would before jumping in. Still, having easier access to vetted questions and exercises would be great, if nothing else for getting ideas that I could modify. So, perhaps I should go back to thinking about how such a database would work!
Yes, I agree, you bring up really good points. So far I’ve been sharing questions with people who teach the same course in a similar timeline (ie. generally the same order of topics). I think there will always be a need to adapt questions to your own liking… but for me (and I have found this to be particularly true with exam questions), it was just helpful to see how other people frame their questions. And also, to see how professors who have taught a lot, hone in on student’s misconceptions about commonly confusing topics in biology.
Yes, having information on what the common misconceptions are would be really useful, rather than needing to figure those out on one’s own. I feel like I know the major stumbling blocks for some areas (e.g., competition) much better than for others (e.g., the fossil record). Having more insight into what those are would help a lot!
Pingback: Take the active learning challenge | Arthropod Ecology
The problem lies in the expectation that learning is to be “fun.”
The courses in which I retained the most information or which provided a basis to apply knowledge from other disciplines were rigorous and un-fun. In most other parts of the world, students are handed a text book, attend lectures, and take a mid term and a final.
In real life, when one works a job, the process of acquiring skills involves a great deal of interleaving. This is across all fields.
But there’s research showing that when people are having fun learning, they learn more.
I’ll be your anti-anecdote: the courses in which I retained the most information where the ones where I was most engaged (i.e. having the most fun). I found that if it was hard to learn the material, then it was both not fun and I didn’t retain the info as much.
Math and other basic subjects are taught in un-fun ways without technology in the third world. I encounter many more foreign born people, regardless of occupation, who can perform mathematical operations mentally with an ease that their counterparts produced in “learning is fun” American environments sadly do not. Perhaps because without disciplining themselves to do well in school they would be relegated to even less fun activities, like working in the fields or in a factory for the rest of their lives. There is nothing wrong with having fun while learning. But to make learning fun as the main objective in education is very damaging and immature and promotes weak characters lacking resiliency. Working a job is not fun. Dealing with failures is not fun. Pounding the pavement with resumes is not fun. Successful people with real educations are who they are because of the un-fun self discipline they had to exert.
In summary, whining about learning not being fun enough is the ultimate first world problem.
Pingback: Evaluating Teaching | Dynamic Ecology
This is an interesting study that relates to this post: https://www.ncbi.nlm.nih.gov/pubmed/28145270
The abstract:
The flipped classroom was utilized in a two-semester, high-content science course that enrolled between 50 and 80 students at a small liberal arts college. With the flipped model, students watched ~20-min lectures 2 days/wk outside of class. These videos were recorded via screen capture and included a detailed note outline, PowerPoint slides, and review questions. The traditional format included the same materials, except that lectures were delivered in class each week and spanned the entire period. During the flipped course, the instructor reviewed common misconceptions and asked questions requiring higher-order thinking, and five graded case studies were performed each semester. To determine whether assessments included additional higher-order thinking skills in the flipped vs. traditional model, questions across course formats were compared via Blooms Taxonomy. Application-level questions that required prediction of an outcome in a new scenario comprised 38 ± 3 vs. 12 ± 1% of summative assessment questions (<0.01): flipped vs. traditional. Final letter grades in both formats of the course were compared with major GPA. Students in the flipped model performed better than their GPA predicted, as 85.5% earned a higher grade (vs. 42.2% in the traditional classroom) compared with their major GPA. These data demonstrate that assessments transitioned to more application-level compared with factual knowledge-based questions with this particular flipped model, and students performed better in their final letter grade compared with the traditional lecture format. Although the benefits to a flipped classroom are highlighted, student evaluations did suffer. More detailed studies comparing the traditional and flipped formats are warranted.