If you email it, they will comment

July 23, 2012

Today I made an appearance on everyONE, the PLoS ONE blog. I relayed my experience using email to solicit comments on my lab’s recent PLoS ONE paper about the antidepressant Zoloft’s accumulation in yeast cell membranes. Here I’ll present supplemental materials that include the minutiae (and extended discussion) that didn’t make it into the guest post due to space limits.

 

To recap, here are the take-home messages of, and response rate data for,  my sociological experiment:

  1. 8 out of 166 (~5%) university professors agreed to move post publication review from the email world to my article’s comment thread
  2. Professors with whom I had prior contact replied at nearly thrice the rate of strangers
  3. The 8 solicited professor comments placed my article in the top 0.1% of PLoS ONE articles
  4. Commenting is not a favorable process, to borrow a phrase from thermodynamics; one must supply energy to make the reaction go forward

Now let’s dig into the numbers. There were 122 non-responders. 12% (15/122) of “no replies” were actually autoreplies, or my email bounced outright because the recipient’s email address is defunct. In the food processing industry, they call unavoidable losses “shrink.”

 

What about the 14 professors who engaged in multiple rounds of email colloquy but never commented? Why didn’t these professors surmount the seemingly tiny hurdle separating emailing from commenting? It wasn’t for lack of interest, because two of the 14 non-commenters volunteered to chat over the phone, which is certainly worth more than a comment. Some professors may prefer private over public correspondence, period.

 

In addition to the eight successful cases of email-to-comment conversion, there are six other comments on my PLoS ONE paper for a total of 14 comments. Two of those comments don’t really count: one is a press coverage digest generated by PLoS ONE, and the other one is a digest of relevant tweets about the paper that I posted. So that leaves 12 bona fide comments.

 

Only one of the 12 was unsoliticited and anonymous, and, not surprisingly, a bit odd: “I followed this discussion from In the Pipeline. I’m not sure this is related, as the information seems self-reported, and therefore anecdotal. The sample size is quite small, as well. The reference is from a study published in the Canadian Journal of Psychology about the rare side effect of yawning as a cause of orgasm in female study patients taking clomipramine, which can be found at the link below.”

 

In the Pipeline” is a blog maintained by noted science blogger Derek Lowe, which brings me to the most interesting comment of the 12. Lowe reviewed my paper, and his post garnered a whopping 25 comments! And it isn’t just quantity. The “In the Pipeline” comment thread is more complex than my paper’s comment thread, with later comments responding to or amplifying earlier comments, and multiple posts by the same commenter, etc. Interestingly, most of the commenters posted pseudonymously, a custom in many online comment forums, and yet the content didn’t suffer one iota.

 

However, the “In the Pipeline” comments are invisible in the official PLoS count, and, regrettably, isolated from the discussion already in progress on the PLoS site, even though after the fact Lowe was kind enough to copy and paste them into a comment for me. Alas, I don’t have a good answer to comment-thread balkanization. I wish it were as easy as highlighting text in an email, and then clicking a “send to comment thread” button that would instantly cross-publish at the PLoS ONE comment thread, or any science comment thread of one’s choosing.

 

With more online comment forums adopting a common infrastructure, e.g., DISQUS, the fragmentation should lessen over time (I hope). Yet even with a stable commenting identity that is portable across a wide swath of blogs, there’s no way to recreate overnight the informed, loyal readership of “In the Pipeline,” which took years to cultivate I’m sure. But I’m not asking for miracles, just more efficient coordination, and where that fails more technology that facilitates cross-blog pollination.

 

So how many comments are enough? 25? 100? 200? Put another way, what is the natural comment ceiling for a given research article? Obviously the answer is complex, and depends on the popularity of the research area, the reputation of the authors, the reputation of the journal, among others. However, as I argued in my everyONE post, the 90-9-1 engagement rule is a good rule of thumb. 90-9-1 also jibes with my paper’s article level metrics.

 

Let’s assume that the 440 PDFs downloads of my paper are a proxy for university- and industry-based readers, as I argued in a previous post, and as others have recently argued. For that many PDF downloads, I would expect several dozen comments. If you add the 12 bona fide comments residing at PLoS ONE to the 25 comments residing at “In the Pipeline,” that comes out to 8.4% (37/440), which is almost exactly equal to the aforementioned email-to-comment conversion rate.

Related Posts

  • http://blogs.plos.org/mfenner Martin Fenner

    Ethan, I like the term “comment-thread balka­niza­tion”, which really is a big problem. Some people have spent years working on this problem (talk to Euan Adie if you want a historical perspective), and article-level metrics is one of the newer approaches.

    The most important strategy to increase commenting is to build a community, and good bloggers like Derek Lowe on average are much better doing this than authors and journal publishers.

    • http://twitter.com/eperlste Ethan Perlstein

      Thanks, Martin! I owe a debt of gratitude Euan Adie for enabling my site to use the Altmetric API.

      I’m going to experiment with integrating G+ hangouts on my site as an alternative to posting article comments. In today’s distraction-filled world, the theoretical advantage of asynchronous commenting is a liability in practice, because there’s a critical “freshness” window in which coincident conversation must occur before the article fades from hive mind consciousness.

      Scheduling an online, locally hosted journal club might be a mechanism to capture post publication review, and to me more compelling and interactive than posting and running..

      • http://abdallahalhakim.tumblr.com/ Abdallah Al-Hakim

        The Google hangout is a good idea but in my opinion will not be a replacement to comments. I don’t completely agree with the the ‘freshness’ window argument. I have commented and gotten involved in conversations on several blog that were in some cases a few months old. They key part of this is to be on a network where your friends’ social media activity especially commenting is visible to you. This is where Engagio has been a terrific product to rehash old but still very relevant conversations. This is critical because even the most socially connected people do miss on some interesting conversations but will get engaged if they become aware of it.

    • http://engag.io/ William Mougayar

      Indeed, if the starting point is a comment, the end-point is the community. In-between are the discussions and level of engagement. We have done lots of thinking and research into this topic (see my thoughts on the blog http://blog.engag.io and the Engagio product behind it). Another important factor is what the blog owner/author does in terms of participation.

      Engagio unifies users conversations across the social web, hence helping to lessen the impact of this balkanization. We call it fragmentation of the social web.

      • http://twitter.com/eperlste Ethan Perlstein

        Thanks for the commenting here! I look forward to engaging with Engagio!

        • http://engag.io/ William Mougayar

          :)

      • http://www.ivpcapital.com/blog Michael Elling

        William, I’m really focused on lower layer projects (access, bandwidth, etc…) right now so I don’t have the time to focus on this, please let me know what you think about my suggestion elsewhere on this thread about not just unifying user conversations, but making the whole mass more intelligible to everyone from THEIR perspective; not the bloggers perspective per se.

  • Barry Bunin

    I appreciated your blog: http://blogs.plos.org/everyone/2012/07/23/no-comment/ and figured I would at least leave a comment here

  • http://www.ivpcapital.com/blog Michael Elling

    Why does Disqus force bloggers to their platform? Why don’t they just provide a template that records where your post or comments are cited and let you include them? Something like an automatic or semi-automatic comment bookmarking tool so it cuts across all silos. Today, I keep a record of where I leave comments out of the Disqus silo in a word doc and its a real hassle.

    The template/tool should also provide for categorizing, indexing, sorting, filtering and searching through a blog, regardless of which platform based on the “commentator’s perspective”. This would make the whole process more valuable and the “long-tail” of comments (and hence original posts) would become godzilla-like in size.

    Has anyone proposed this or is working on this?

    • http://twitter.com/eperlste Ethan Perlstein

      Hmm, so you haven’t drunk the Disqus Kool-Aid, I see! ;)

      Seriously, I think you raise good points. I’m agnostic as to who provides the technology, so long as we begin to chip away at comment-thread fragmentation.

      • http://www.ivpcapital.com/blog Michael Elling

        I like Disqus. Want them to do more. Too many companies get the network effect virus and become immune to its positive qualities. A network with a million users might benefit even more from a network with only 10,000 users, particularly if the latter has a high quality user base or one that utilizes features better. There is definitely value in joining the two networks. Likewise, Disqus should realize that there is broader value in going after the whole commenting universe and monetizing it all via direct and indirect means; not just forcing a small portion of the universe into its domain. I wonder what the technological issues are for them to cross blog/comment silos.

        • http://twitter.com/eperlste Ethan Perlstein

          I stumbled across Disqus because I follow Fred Wilson (AVC) on Twitter and he tweeted about it.

          Are there Disqus clones? Is it known what % of websites with commenting use Disqus? They’re nowhere close to fixation, right?