Why do we publicly criticize bad papers, but not bad talks?

Over at NeuroDojo, Zen Faulkes makes an interesting observation: in public, scientists seem to be harder on bad papers than bad talks. This seems right to me, and not just because papers, but not talks, get criticized in peer review. It’s not uncommon for papers to receive public post-publication criticism, whether in formal comments to journals or on blogs. But it seems like this never happens with talks. I’ve never seen a blog post criticizing a named speaker for giving a bad talk. And in my admittedly-limited experience, I’ve never seen anyone who was live-tweeting a talk do anything other than summarize or compliment the talk.

Indeed, in an old post on predictors of talk quality, I told commenters not to name anyone who gives a bad talk. Now that I think about it, I’m not sure why I did that, as I have no problem with naming the authors of papers I’m criticizing.

I emphasize that I’m talking about public criticism here. In private conversation, I don’t find people to be any more hesitant to criticize speakers and talks than they are to criticize papers and authors.

Maybe it’s that we care more about bad papers contaminating the peer reviewed literature, but just find it mildly annoying to have to sit through a bad talk? Or maybe criticism of a bad talk inevitably seems like personal criticism of the speaker, while criticizing a bad paper seems more like criticism of the science rather than the author? (You see Dr. Joe Poortalk standing in front of you, mumbling into the floor or showing you horribly-designed slides or whatever, whereas the author of the paper you’re criticizing may be someone you’ve never seen.) Or is it something else? I’m not sure.

What do you think? Are we really more publicly critical of published papers than we are of talks? If so, why do you think that is? And what, if anything, could or should be done about it?

21 thoughts on “Why do we publicly criticize bad papers, but not bad talks?

  1. Regarding live-tweeting of talks: I’ve done this a little bit, and found it gets really awkward if the talk is bad (either because I can’t follow or don’t believe the data). If I can’t follow, there isn’t much to do, other than stop tweeting. If it’s because I stop trusting the data, I have made a note along the lines of “Current fig seems to contradict previous one”, mainly to try to correct the record of earlier tweets.

    I do find the issues related to live-tweeting talks interesting, and perhaps worthy of further discussions. 🙂

    • Re: live tweeting, I have an old post in which I wrote some old codger-y, “hey, you kids, get off my lawn and tweet someplace else!” things. The commenters smacked me down, quite rightly.

      I agree there is more that could be said about live-tweeting, though. Another post on the topic would be good, but the problem is I’m not really the best person to write it. 😉

  2. Part of the issue is audience I would imagine. Published papers have a much broader audience than talks do, and are therefore more likely to reach someone who would disagree or cirticize the science of the paper. Since they are more public, papers should have more public criticism as well.

    Also, unlike talks I think it is rare to comment on the style of a paper. Most criticisms are aimed at the science contained within them (although that may not be the case for poorly used figures). A poorly given talk does deserve feedback, but when it is mostly about style or the public speaking aspect of the talk I think it is probably better to give that feedback in a more private setting. Yet, if there is something to be said about the science given int he talk, you can voice your concerns with a well worded question afterwords. At least, that is my opinion.

  3. Presumably we’re talking about bad science here, since bad presentation, whether in a paper or in a talk as sort of secondary. I’d be far more motivated to write a response about bad science than about bad graphs (although I’ve done both: http://downwithtime.wordpress.com/2012/08/03/our-pnas-younger-dryas-impact-letter-is-out/)

    I wonder how much of this is the perception that public talks often deal with material that is in preparation, whereas papers are finished products. If that’s the case then I think you’re right about the difference between mild annoyance at being bored and the frustration about papers making it into the literature.

    I’ve definitely seen people take apart talks during the question period that follows, but afterwards, what’s the point in continuing to be upset? That talk won’t live on in any form other than a mention on a CV, and perhaps in a complete re-tooling of the project once the speaker gets home.

    I suspect the other problem is that when speakers do not have English as a first language it can be difficult to parse what is bad presentation skill and what is bad science. Did the listener miss a critical point because the speaker couldn’t communicate it properly (because they’re forced to present in English) or because the speaker didn’t consider it? This, more broadly, is why we’re more forgiving of talks: papers as a whole are far more formalized so there is less leeway for presentation to obscure the underlying science. That must be some sort of McLuhan thing. . .

  4. Seems like a good thing. Bad talks are often bad for reasons that have little to do science, and it’s easy (for most people) to sympathize with the struggling speakers. And it *is* much more personal. You can do great science and be a lousy speaker for tons of reasons: anxiety, no/bad sense of humour, poor English, etc. As we try to achieve a meritocratic system, it seems like a very good idea to harshly criticize citable contents and be more tolerant (i.e.: human) for talks. Not, of course, that scientific writing skills are equal or that talks have no influence, but the scientific paper is the more impersonal and polished manifestation of science, it’s our job to criticize it.

    • Yes, as you and several other commenters have noted, I think what it comes down to is that we’re prepared to hold every scientist to a high standard on content. That’s what you get a PhD for, the demonstrated ability to do science at a certain level. But while we certainly train our students in public speaking, good public speaking isn’t seen as part of the “core” of what a scientist is, or should be, trained to do. So if you publicly trash someone’s talk, it seems like criticism of them personally, rather than them as a scientist.

      Whether this is the way things ideally ought to be, I’m not sure. I’m unsure not because, or not just because, there can be practical problems in distinguishing “content” and “presentational” issues in a talk (e.g., if the speaker’s first language isn’t English, it’s hard to tell poor science from poor presentation). I’m also unsure because I’m not sure that there really is a completely clear, sharp dividing line between “content” and “presentation”. If, for instance, you begin your talk by providing an unconvincing motivation for some aspect of your research (say, choice of question, or choice of study system), is that a content issue or a presentational issue? Or both? Other examples could be given. Another commenter tossed out Marshall McLuhan of “the medium is the message” fame; I guess what I wonder a little is the extent to which “the presentation is the content”. In many ways, they’re distinct–but I do think there are ways in which they shade into each other.

  5. I think it has to do with the degree of physical disconnect. In a bad talk we were just sitting in the same room with the person. In such situations, it is awkward to be critical and we’re apt to have a higher threshold – e.g. ask what do I really have to gain from being critical – is it worth annoying this person. With a paper, its just, well, a piece of paper, with no personal interaction. Criticizing a paper by some random author I’ve never seen feels no different than say criticizing my favorite sports team – I’ll never interact with them.

    A testable hypothesis deriving from this is that we are less likely to publically criticize papers by authors we know we have/will interact with personally. I know that is true for me. And its probably one reason graduate students are much more prone to criticize papers – they just don’t know the authors yet, but for more senior faculty, it feels a rather small world.

    • Hmm–I think I’d be at least as likely to sharply criticize someone I know well, because I know we’re friends and we’ll still be friends even if we have a knock-down, drag-out argument. But it’s a difficult hypothesis to test, because my friends are my friends in part because they mostly agree with me. I can’t actually recall ever wanting to totally rip the science one of my friends did. I suppose the closest I’ve come is in comments on this blog, where I have more than once had pretty sharply-worded and serious disagreements with people I know, or know of (or who’ve commented so often that I feel like I know them).

      Which I suppose might just be evidence for your point about physical disconnect, since people are famously ruder to one another on the internet than they would be face to face. 😉

  6. I think there’s something more personal about critiquing a talk. A paper is pretty faceless, despite seeing the authors’ names. But a talk – you’re critiquing a specific person’s delivery of information. Everyone who was in the room saw that person get up and give it, and now you’re firing back – albeit not in the moment. So, there’s something more agressive about it – more personal, and less seemingly objective, even if your comments are designed to be objective.

    Can we get around this? I think the answer comes back to something that many scientists are not good at – constructive criticism. That is, criticism that clearly acknowledges what the author was trying to do, why it didn’t work, and what can be done about it. One can think offhand of a number of examples of constructive v. destructive criticism, but, it can even extend to more innocuous comments, such as, “I just didn’t understand what she was saying there.” – phrasing it rather as, “When the speaker expanded on her conclusions to show X should then lead to why, I found wasn’t able to follow her chain of logic because of the gap at Y.” It’s something that I have been shocked to see many excellent scientists fairly bad at being able to do – perhaps because that style of criticism is often taught in English and Writing classes. It’s a huge help for critiques of papers, but to declaw the personal attack nature of critiquing a bad talk, it’s triply necessary, as it were, I think.

    All of this reminds me I have some questions of yours about my own recent talk to answer. HA!

    • Hi Jarrett,

      I think you’re absolutely right that constructive criticism is a skill that many of us are bad at. ESPECIALLY me! And it’s much broader than just making sure your criticism–of a talk, a paper, whatever–isn’t seen as a personal attack. It’s how to get the person your criticizing to engage with the criticism and take it seriously. Which of course they won’t do if they see it as a personal attack, but which they might well not do even if they don’t take the criticism personally. Outside of the context of peer review (where the editor acts as an arbiter and authors *have* to take criticism seriously on pain of having their paper rejected), how do you make productive debates (not just discussions, *debates*) happen in science. I think it’s hard. In some ways, this blog is kind of my personal ongoing experiment in learning how to start and pursue productive debates.

    • And yes, I believe you do have some questions of mine to answer about your own recent talk. Which was good, by the way, except that you tried to cram too much in. 😉 Occupational hazard of being enthusiastic about the subject of your talk. 😉

    • Not exactly your point, but on a related point. My PhD adviser (Mike Rosenzweig) always used to say we’re really good at training students to rip apart a paper, but terrible at training people to find the good in a paper. He used to start every reading seminar class by making each student say one thing they liked about the paper, a practice I’ve continued. I think this is a really profound insight. True judgement and engagement in a paper is being able to list the weaknesses AND strengths and weight them. This is not how the discourse usually works. The “I’ve found one flaw in the methods and so I’m done” approach is much more common.

      • Speaking as a former handling editor, it’s *really* annoying for the reviewer to stop the review upon reaching what the reviewer considers to be the first serious mistake. Because what if the editor doesn’t consider the mistake that serious, or even a mistake at all? Fortunately, such instances were rare.

        Back on topic, Mike was teaching you something like what I’ve heard called “sandwich technique”. If you’re going to criticize someone or something, first find something positive to say, then say something negative, then conclude with something positive. I practice this technique (or at least the “say something positive first” part) in reviews. And I try to remember to do it in conversation, but I often don’t. And I think I’m pretty bad about it on the blog.

  7. I think the broader question is “why are people so prone to be critical in the first place?” I mean, were we born critical or something? If somebody’s gotten something wrong, just help to identify and correct it, depending on what’s needed. This stuff is mostly caused by the academic environment as infuenced by our societal ways.

    • “why are people so prone to be critical in the first place?”

      Because their blogs would be boring if they weren’t. Well, this one would be. 😉

      I’m kidding of course, but only sort of. As I’ve noted in another post, the blog may well give the mistaken impression that I think everyone is wrong about everything. I don’t. It’s just that the subjects on which I think I have something interesting or important or useful to say are often those few subjects on which I disagree with many other people. It’s boring to write posts agreeing with the conventional wisdom, so I don’t, even though I do often agree with the conventional wisdom. Similarly, I don’t write many posts on good new papers that have just come out, even though good new papers come out all the time, because I often don’t have much to say besides “this is a good new paper”.

      Joking aside, it’s a fair point (and I know you were making a much broader point than just about the negativity of much of my blog content). As other commenters on this thread have noted, there are ways and ways of being critical, some effective, some not, and some with effectiveness that depends on the situation. Whether science as a whole is too critical, not critical enough, both, or neither, I’m not sure. I’d probably lean towards “not critical enough”, but I’m a critical person, so I would say that. It’s something Dustin Marshall and I debated back when I was blogging for Oikos.

      • I should try to explain myself further Jeremy.

        I wasn’t really targeting my comment towards you at all, directly or indirectly. I actually think you do criticism in a very good way–which is to say that you stay **very focused on the subject matter at hand** and you hit the critical points of why something is wrong, which is exactly as it should be. And you try to keep it lighthearted as well with the whole “zombie” theme and other things, so that it’s not taken personally. Not only is this good, it’s necessary, and I actually really agree with your point that science is not critical enough in some (many) respects. Indeed, I have a long story I could launch into on that topic, but I won’t, for lack of time.

        I think what I was trying to say was that we should strive to be critical of ideas, but not critical of people who espouse those ideas. And everyone needs to understand that criticism of the ideas we espouse is fair game (don’t have a thin skin or take things personally!), because the whole point of science is to try to get a correct view of things. At the same time, most papers also have something positive about them, and/or something to build upon, and we should strive to temper our tendency towards being mostly critical, by also discussing what is positive. This helps everyone, and it is especially important for young scientists, because yes, egos can be bruised, and they can be bruised irreparably sometimes.

      • Very well put Jim. I agree 100% with everything you said.

        Don’t worry, I knew you weren’t really directing your previous comment at my specifically (as I know you would’ve said so had that been your intent). I just used your previous comment as an excuse to repeat some things I’ve said before but that I think bear repeating.

        I’m flattered that you think highly of how I criticize ideas, and I’m glad you find all the zombie stuff lighthearted. I do wonder if, say, Michael Huston or Phil Grime would see it the same way (I don’t know them at all so I have no idea…)

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s