How to make your graduate student seminar series better training

Most graduate programs in ecology and evolution have a graduate student seminar series in which students in the program talk about their ongoing work. These seminars give students practice at public speaking, and give them feedback on their science and their presentation of it. Usually this feedback takes the form of post-seminar question-and-answer, just like after any other scientific talk. Sometimes audience members also fill out written comment forms. That’s all well and good. But here at Calgary, we have a much more structured and, I think, more effective system for giving feedback to student speakers. I’ve never heard of anyplace else that does anything like what we do, so I want to share our system in the hopes that you might find it of interest and perhaps even try it out at your own university.*

First, the student speaker is introduced by another student (chosen by the speaker). Then the student speaks for about 20 minutes.

So far, so conventional. But now it gets different. Rather than have question-and-answer after the talk, or fill out comment forms, here’s what we do. The graduate students in the audience break into two groups (they actually get up and arrange the chairs into circles for this purpose). One group is assigned to give feedback on presentation, the other group is assigned to give feedback on content. Each group discusses its assigned topic and agrees a list of comments, both positive and negative. Each group appoints a secretary (someone different every week) to write down the comments on an overhead, in order of importance. Further, the “content” group has to phrase their comments as questions to the speaker (the purpose of these last two features will become clear in a moment). Last year, we also tried giving each of the groups a “rubric”–basically, a list of content and presentation features that a good talk should have–in an effort to guide and improve their discussions. But I didn’t see that it made much difference, so I don’t know that we’ll bother with rubrics this year.

Meanwhile, the speaker goes to the back of the room where the faculty (and any postdocs in attendance) have gathered around a table. The faculty provide private feedback to the speaker on both content and presentation. This feedback is fairly unstructured, except that if there are many faculty in attendance, we’ll go around the table with each faculty member getting to provide one piece of feedback or ask one question. The student’s supervisor takes notes and doesn’t ordinarily speak, so that the student is free to interact with the faculty without having to take notes.

About 15 minutes after the end of the talk, we all get back together. The speaker goes and sits at the front of the room, and the secretary for the “presentation” group goes up to the overhead projector and presents to the speaker (and everyone else) the group’s presentation feedback. That only takes a few minutes. Then the “content” secretary goes to the overhead and poses to the speaker the most important content-related question, which the speaker addresses. They work their way through the other content-related questions in this fashion, and at the end the secretary gives any other (generally minor) content-related comments the group had that couldn’t easily be phrased as questions. At the end, the overheads (which are written in permanent marker) are given to the speaker, so the speaker doesn’t have to take notes.

Then, and only then, is the floor opened to any remaining questions from the audience, with student questions getting priority. Faculty aren’t allowed to raise their hands until the students are done asking questions. By this point, there are only a few minutes left for this. The whole thing takes 50 minutes.

I think this system is terrific. It has a long list of virtues:

  • It gives students feedback on presentation, which they wouldn’t get from conventional question-and-answer.
  • It teaches students how to listen to talks as well as how to give them. Listening well is active, not passive. Good listeners think about and evaluate what the speaker is saying as the speaker is saying it. They don’t just sit there with their ears open but their brains off, and then turn their brains back on at the end of the talk to figure out what they think about what they heard.
  • It leads to better questions. For instance, let’s say a student just misses something the speaker said. Instead of having to waste everyone’s time asking the speaker to repeat part of her talk, the student can just ask her group, who’ll say “Oh yeah, she went over that, you just missed it.” It also leads to better questions in more substantive ways. For instance, when group members disagree on some point, they can discuss the issue, and then turn that disagreement into a well-developed question for the speaker.
  • Obliging students to introduce each other, and present group feedback, gives the students more practice standing in front of an audience and talking.
  • It encourages students who are reluctant to raise their hands after a talk to contribute to giving feedback.
  • Giving faculty feedback to the student privately prevents the student from being embarrassed by very negative faculty feedback. (In my experience, student feedback is never extremely negative, which is a good thing in some ways but a bad thing in other ways. Students also tend to be better than faculty–well, better than me–at framing negative comments in a constructive way.)
  • It ensures that the speaker gets substantial feedback from a much broader group than just their supervisor, labmates, and committee. In particular, faculty not on the speaker’s committee often draw on their “outsider” perspective to raise really good points which the speaker’s supervisor, labmates, and committee never thought of.
  • Separating faculty and student feedback encourages speakers to compare and contrast them. In general, I find that faculty tend to hit on different points than the students, faculty tend to be more demanding (and thus more negative), and faculty often raise deeper and more challenging questions about the speaker’s whole approach than do the students.
  • Giving faculty feedback before the student feedback means that speakers usually get positive feedback last (since students tend to be more positive than faculty) and so hopefully leave the room feeling good about the experience.
  • The formality of the whole process encourages students and faculty to take it seriously, and encourages students to take it seriously as both speakers and audience members. Not that it’s not fun. For instance, there’s a tradition that the secretaries draw some sort of cartoon of the speaker’s study system on the bottom of the overhead, which is always a good laugh. But student seminar series aren’t very useful as practice if speakers give poorly-prepared, overly-casual talks, and if audience members don’t put much effort into thinking about what they’re seeing and hearing.

I’d welcome any comments you have on the way we do things here at Calgary, and any other suggestions you have for making graduate student seminar series more useful as training exercises.

*In passing, I’ll note that at Calgary, the graduate student seminar series is a pass-fail course that graduate students are required to take for two years, during which time they’re required to attend all the talks, participate in the feedback, and deliver two talks of their own. But nothing about how our feedback system is structured depends on the seminar series being a formal course.

17 thoughts on “How to make your graduate student seminar series better training

  1. I don’t think anyone will disagree with you that this is a fine system with many positive qualities! I think the big issue with a seminar series like this is time and participation.

    At my graduate institution in our volunteer seminar series we often would schedule 2 graduate student talks (30 min’s each including Q&A). Additionally we rarely had more than 1 or 2 faculty attending the graduate student seminars. A shame really, while graduate student feedback is better than nothing, it can sometimes be the blind leading the blind.

    I would have loved the supportive and structured seminar series you have going at Calgary!

  2. I really like this system! As a grad student at the Kellogg Biological Station (KBS), we were all required to give a talk every year about our research. It was really excellent preparation for later going on the job market. KBS is relatively small, but broad in terms of research areas, so it was great practice at giving a talk that a general ecology & evolution audience could follow. This is critical — in other places I’ve been, it seems like students often give defense seminars targeting their lab groups, and we’ve probably all been to job talks where the person could not explain his/her research to a general audience.

    Another key aspect of the KBS system was that there was a lot of critical feedback. A lot. It was definitely hard to take at first, but the feedback was all constructive, and definitely led to improved presentation skills, and to some new ways of thinking about data or analyses or whatever, and so better science. I also found it difficult to respond to questions at first — I used to get very defensive when asked a question about my research. I think having to give yearly seminars as a grad student went a long way towards making me much better able to handle questions.

    The key is for the student to receive critical, constructive feedback. Giving such feedback seems to be ingrained in the KBS culture, but less so other places I’ve been. It seems like, other places, people ask more softball questions. I think this is a disservice to the grad students — they are not going to get softball questions when they are on the job market!

    I will be interested in seeing how the student seminars are here at Michigan. But it sounds like it’s an important part of the culture, and I think that’s a very good sign.

    • I agree 1000% with the need for constructive but *critical* feedback. Nobody needs any practice fielding softball questions, and asking such questions of students is absolutely a disservice.

      Here’s the way I put it to our students: would you rather get asked tough questions by your friends and colleagues, or would you rather get those questions for the first time at your defense or at a conference or at your first job interview?

      As a grad student, it bugged me that our grad student seminar series had such a casual vibe. I *wanted* people to push me and nobody would! I once even had the organizer hand out comment forms to the audience–something we had once done but had stopped doing because the other students thought it made things too high-pressure.

      It’s no accident that many of the best people in ecology and evolution went through graduate programs where there’s an ethos of asking tough questions at graduate student seminars. Peter Morin had great stories about Duke in the ’70s and ’80s. A whole generation of the very best people in EEB came out of Duke in the ’70s and ’80s–Peter, Mathew Leibold, Tom Meagher, many others. One big reason why is that Duke were *tough*–as a speaker, you had to be *ready*. Indeed, it sounds like it probably crossed some lines sometimes; audience members would just interrupt speakers and question the point of their entire research program. And apparently, it was actually the *students* who were often toughest on each other!

      Note to any students reading this comment who find this vision of how graduate seminars ought to work really scary: you’ll get less scared with practice, and come to see it as a great experience, the way Meg did. But if nobody ever pushes you, you’ll *never* learn how to handle it, and your job prospects will suffer badly.

      • Well, I would count Mathew as a KBS alum. πŸ™‚ But, yes, I absolutely agree with your last point. When I gave my job talk at Georgia Tech, someone interrupted me about 5 minutes into the seminar with a tough question. I don’t remember the exact question, but the implication was clearly “Should I care about this?” I think I answered it reasonably well — and, just as importantly, didn’t get completely flustered by being interrupted at the beginning of my talk — and was able to move on with my seminar. I am certain that I would not have been able to do that if I hadn’t had so much practice as a grad student.

    • I agree that the KBS “brown bag seminar” works well, but I really like Jeremy’s suggestion to also provide feedback on the presentation itself. Something we should try out.

  3. Yes, if you’re going to operate the way Calgary does, you need the time not to double-book speakers, and you need faculty to attend. Encouraging faculty attendance is hard if it’s not already happening. If you’re not in the habit of ever attending graduate seminars, doing so feels like an interruption to your usual schedule. But if you’re in the habit of attending, then attending *is* your usual schedule, and so it doesn’t feel like it’s taking time away from anything else. (it’s the same with blogging, actually…) Plus, if nobody else attends, then attending is just seen as “not the sort of thing that faculty do”.

    FWIW, we had the same issue at Rutgers when I was a grad student, where the graduate seminar series was student-organized (albeit with only 1 talk/week). Faculty attendance was usually 2-3, and the only faculty member who attended every single week without fail was my supervisor, Peter Morin.

    So how do you push the system from the “bad equilibrium” of low faculty attendance to the “good equilibrium” of high attendance? Well, students as a group could try emailing the faculty and raising the issue, explaining that you really want them to attend and give you feedback, that you don’t want the seminars to be a “student thing” just because they’re student-organized (might help to tell them you’ll all be giving well-prepared talks that will be well worth their time to watch). This might help to the extent that faculty think (incorrectly) that they’re not actually wanted at graduate seminars, that the students want a low-pressure, faculty-free environment. You could also try making the seminars late in the afternoon and have beer available if you don’t do that already. Anyway, good luck with it!

  4. Another comment for students who find this vision of graduate student seminars really scary: there are few feelings in science better, and few things that make your supervisor prouder, than fielding a tough question and knocking it out of the park.

    One of the proudest moments of my career came when I was a postdoc, and my former supervisor Peter Morin and I were both attending a food web conference. I’d spoken earlier in the day, and gotten some good, tough questions from Andre de Roos, which I answered just fine. Mathew Leibold was there as well, as was his former student Amy Downing, who like me had also spoken earlier that day and had handled some tough questions. In the evening, I was having a beer with Peter and Mathew. Mathew said that when Amy got her first question, he had to hold himself back from jumping in and answering it for her. All faculty have that feeling–that’s your student up there, your professional “progeny”, the last thing you want is to see them embarrassed by a tough question. And Peter said, “Yeah, but Amy handled it great. Jeremy handled Andre’s questions great. That’s the best thing, when you can just sit in the audience and relax, because they can handle themselves.”

    When we faculty ask you tough questions, it’s not because we want to see you fail–it’s because we want to see you succeed!

    • I know I should leave this one alone, but I’m really not a big fan of the ‘hard knocks’ approach to graduate seminars. I think the difficulty is that the environment created to ‘learn how to field hard questions’ quickly can become ‘prove how smart you are by flustering/destroying the speaker’. I deeply dislike the latter and I have to admit I’ve seen more of the latter than the former. I have found it creates an atmosphere that focuses on finding ‘flaws’ as opposed to a more balanced scientific thought process and tends to stifle productive scientific discourse because conversations become about ‘winning’ as opposed to listening and thinking. It also selects for specific personality types – which IMHO is not necessarily to science’s benefit. Having said this, I really like the Calgary scenario you describe because it sounds like it creates a constructive atmosphere. We don’t have anything like that for our students here and I can see the strong training value in what you describe.

      • How dare you disagree with me! I’m going to destroy you! Justify your entire existence!

        Just kidding of course. We don’t necessarily disagree as much as you might think. I freely admit that there are better and worse ways to ask tough questions and give feedback.

        The folks who ran the CPB immediately before and during my time as a postdoc there provide a useful contrast here. John Lawton had a reputation as very positive. I’ve heard that students would come out of meetings with John flying high. Conversely, Charles Godfray had a reputation for being very tough and I think many students found him intimidating. But here’s the thing–it’s not that John was a soft touch. I’m sure if a student’s idea was rubbish, John was just as quick to tell them as would Charles. But somehow John was able to pitch the same feedback in such a way that students would feel good about it.

        I’ll admit to mixed feelings here. I don’t mind aggressive questions and my own natural tendency is to be aggressive, so there’s definitely a part of me that values those same traits in others and sees them as virtues. And I really do see some training value in being exposed to aggressive questioning early on, if only because you’re going to encounter it at some point and you need to learn to handle it. But there’s also a part of me that feels like, as long as you get your substantive point across, you ought to ask your question or make your point in the nicest way you can, and I admit I’m bad about doing that.

        Can a ‘hard knocks’ approach just deteriorate into people showing off by attacking the speaker, setting out to look for flaws, and trying to ‘win’ rather than trying to have a productive debate? Sure, though in my experience that’s rarer than it apparently is in your experience. But I don’t think it necessarily has to be that way. I think the ideal is something like a ‘tough but fair’ atmosphere, which is challenging to achieve. Good formal procedures help (and I think we have those at Calgary), but you also need people who have the right informal skills. People who can ask a tough question, then switch to listening carefully to and thinking about the answer. Who can phrase and deliver a tough question in a way that reduces its perceived scariness without actually softening it. Who know how to intermingle negative and positive feedback. Whose body language conveys that they’re not out to “get” anybody (even if they’re being quite critical). Etc. As I’ve admitted in other posts, I mostly lack those informal skills, though I think I’m slowly getting better.

        I will say that your comment does come as a bit of a surprise, coming from someone who is still legendary at a certain university for going at it hammer and tongs with the chair of the search committee while on a job interview visit. πŸ˜‰

      • Like many things in ecology, there’s some intermediate optimum. And it’s probably better to err on the restrained side.

        In my experience, almost all faculty recognize the inherent power differential and keep themselves in check when grad students give talks. That doesn’t apply to grad students, who more easily veer into overly aggressive questioning. I’ve heard stories about a tradition where grad students would essentially hijack a talk and see how long it would take for the speaker to “grab the chalk back”. I might jump up to draw a graph on the board to illustrate a question, but I’ll return the chalk!

        There also seem to be cultural differences between fields. Math talks seem to be often interrupted by the audience with statements like “you can’t do that” or “that’s not right”. But I don’t think its taken personally, just the audience trying to follow the argument.

      • Your grad students easily veer into over-aggressive questioning? Our students *never* do! Clearly there’s a lot of “cultural” variation from place to place.

        There certainly is a culture in math of interrupting the speaker to ask questions. In my limited experience, not really aggressive questions so much as clarification questions. Mathematicians, not surprisingly, want to know *exactly* what you’re assuming! When I’m giving talks to biomathematics groups, I always include way fewer slides than I would when preparing to talk to EEB types, and I prep less formally, knowing that I’m not going to have time to deliver a 50 minute talk.

      • On average I’d agree, but there’s much greater variability among grad students. It’s those outliers that I’m talking about.

      • I can neither confirm nor deny that I can confirm or deny that said story about one of my interviews may or may not be true. πŸ™‚

        I enjoyed your insights from CPB. My advisor was a lot like Lawton. Every meeting with Jim Brown was a motivational experience, but I also knew when a lot more work was needed to get where I needed/wanted to go! I think my sensitivity on this issue comes from the dynamic that lowendtheory discusses and what this can do to the graduate/postdoc culture at an institution. Not everyone is psychologically well-suited to either conduct or deal with aggressive questioning, but that’s not a trait I would want my students filtering themselves on! Finding flaws is relatively easy. Understanding what those flaws may or may not mean for the study is more nuanced. That’s what I like about the Calgary set up, it provides a forum to not only train students in how to handle questions but also to train students what kind of questions to ask and how to ask them. A similar set up for training students how to review manuscripts would also be a great training opportunity!

  5. We had no grad student seminar requirement, or series, at any time that I was in grad school. All seminars were always given by faculty or guests. Furthermore, there was no oral defense of the thesis either. In my entire grad school time I never gave, nor heard from any other student, a talk about my/their research. That didn’t happen until I went to the ESA for the first time. There was very little interest shown in the grad students at all frankly. It was as if we weren’t really there.

    So, pretty much anything’s better than that.

  6. It sounds like a great system. I like that the students have time to discuss and that the point is to give a balanced report on both the +ves and -ves. The traditional Q&A framework more heavily emphasizes the areas for improvement and may not give the student a good picture of the overall talk quality. I’m interest to know, though, if the presentation group is able to make substantative comments. From reading the post, I thought: ‘oh, content group = fun’… and then ‘presentation group… possibly fun, but… do I find myself wishing I was on the content group?’… or is that just me?

    Really enjoying the group blog, btw!

    • Hi Amy,

      yeah, typically the ‘presentation’ group will finish their discussion in less than the allotted time, and so will move on to talk about content a bit. And sometimes the content group will also throw in a bit of presentation feedback. And there are occasional joking debates about whether some aspect of the talk was “really” content or “really” presentation (e.g., if you omit some substantive piece of information the audience wanted to see, is that a “content” problem or a “presentation” problem?)

      Note that the membership of each group changes every week, so every student gets to participate a lot in both groups over the course of the year.

  7. Pingback: Friday links: profs don’t retire, republishing open-access articles, and more | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s