So, how do you decide what material to include in the main text of your papers, vs. in appendices?
Some things are easy. Raw data, code, and lengthy derivations belong in appendices.* Alternative ways of running the analysis that lead to the same conclusions belong in an appendix. And these days some journals increasingly only want a summary of your methods in the main text, with the details relegated to appendices.
But what about tougher calls? For instance, you might say that the main text is where you tell the “main” story of the work and present “key” results and analyses, with mere “supporting details” relegated to appendices. I’d say that myself. But if you take that approach to an extreme, it basically says that your main text is your abstract (plus maybe a couple of figures), and everything else is mere supporting detail! I have the impression that Nature and Science papers have kind of gone down this road. I’m old enough to remember a time when Nature and Science papers really were quite different beasts, in both form and content. A good Nature or Science paper told a clear, deep story in a very compact, incisive way. Nowadays, it seems like the main text of many Science and Nature papers is like an extended abstract of an ordinary paper, with the rest of the paper buried in lengthy appendices.
This isn’t just an issue in where to draw the line between main text and appendices. It’s also about which material goes on which side of the line. A result that one person thinks is a mere supporting detail may be viewed by someone else as the most important and interesting result in the paper. Now, you could argue that’s no big deal. After all, the main text and appendices are all there, so anyone who’s more interested in Appendix 27 than in the main text can just read accordingly. But I’m not so sure I buy that. We present our work in narratively-structured chunks–called “papers”–for a reason. As readers, we want and need authors to tell us what to focus on, not to just give us an unstuctured stream of material and say “figure it out for yourselves”. As a reader, I’m trusting you as an author to make a good professional judgement about what the story is. I might not necessarily agree with your judgement, of course, though ordinarily I probably will. I need to trust your judgement because frankly, while in principle I could read all your appendices, in practice hardly anyone (even specialists on the topic of your paper) is ever going to do so.
So, how do you decide what goes in the main text vs. appendices? Have you ever had a decision you struggled with? And does anyone share my sense that the advent of online appendices isn’t just allowing people to report lots of supporting detail that otherwise would’ve gone unreported, but is actually changing how we write our papers in more subtle ways?
p.s. A recent post of Brian’s discussed a case in which he found an appendix of a paper very interesting and important and so wondered why it wasn’t discussed at greater length than the main text. My post was actually in the works before that, so the timing is fortuitous. And that particular case was already discussed in the comment thread on Brian’s post and I don’t want to reopen that discussion. My interest here is in the general issue I’ve raised, not in debating specific examples.
*In ecology, of course–if it’s a math paper the derivation is the paper.
Dynamic Ecology 
I think a point that you haven’t mentioned is that Supplementary material is not peer reviewed to the same extent. I’m sure some reviewers do sometimes check it. But journals don’t vouch for it or put effort into copy editing it.
Therefore the tendency to put everything in supplementary material is dangerous.
With online journals I think word counts should be more flexible to account for some of this. While derivations can be long and boring, they may still be vital to the paper and should be reviewed to the same extent. I’ve had this problem with a recent paper. The methods required deriving ~20 different models. It is completely boring, but is the main result of the paper. I had to put it in supporting information for the sake of the word count.
I’m not sure about journals not “vouching” for supplements. If it’s material they’re hosting, and that was made available to the referees, then they’re vouching for it just as they are the main text. At least in principle. In practice, I suspect you’re right–the supplements aren’t always checked.
In the few journals I’ve published in recently, there are explicit statements that the supplementary material will not be peer-reviewed. In practice, reviewers do seem to look at them, but the publisher is making it clear that it is the authors’ responsibility to vouch for and edit the supplements.
I decide what goes to the main text or the appendices based on the strategy of writing backwards (proposed by Bill Magnusson). The first thing I write in a paper is the main conclusions and then the discussion as a whole. Based on that, I check which results and figures are extremely necessary to support those main conclusions. Those pillars go surely to the main text, and everything else goes to the appendices (with one exception or two).
Worth noting that the Journal of Neuroscience stopped accepting online supplements a few years ago. See here for some discussion: http://johnhawks.net/weblog/topics/metascience/supplementary-info-neuroscience-2010.html
Isn’t the notion of a “supplement” an artifact of paper publishing, in the awkward transition-phase when journals were both bound in libraries and online?
Good question. I’d say yes and no. That’s the historical origin of a supplement, but I think the distinction between main text and supplements retains value in the fully-online era.
Your question hints at some big issues. For instance, at what point does the whole notion of a scientific paper as the “unit” of scientific communication become obsolete (if it ever does)? I have no idea.
Yes, I’d say my writing has changed based on the ease of putting things in online supplements. If it’s something that is only likely to be of interest to a small (say, <5%) fraction of the readers, I tend towards putting it in the online supplement, in favor of making the rest of the paper more readable to the other 95% of the readers. If online supplements weren't possible, I would put that in the main text.
Other times, I add something to the appendix that I would have just left out if not for the possibility of an online appendix (e.g., some data that's related but not enough to stand on its own and that I'm not planning on really following up on to turn into its own full study).
And, in still other cases, things that are in the appendix are there because a reviewer wanted them moved to the appendix.
I don’t thik raw data belong in the supplementary. They belong in searchable repositories like datadryad or figshare.
Same as Terry above, I think that the notion of a separated appendix is making less sense with electronic journals. F1000Research does this smart thing where they append the supplementary to the main text of the paper. You still have a relatively short and concise main text, but within the same document, you can see the whole information.
Other than that, yeah, appendix are just here to make a referee happy.