It seems like there is no institution that is more criticized in science than that of the peer-review system — an no one that is less mutable. While published paper evaluation metrics are being  revised (such as the recently introduced PLoS article level metrics, or the Australian National Health and Medical Research Council abandonment of the Thomson Reuters impact factor system), the peer review system seems like it is here to stay. When asked, most scientists would probably paraphrase Churchill: “peer review is the worst system for judging science, except all others that have been tried from time to time”. (However, Churchill did have other working state models to compare with Democracy, whereas peer-review seems to have no, um, peers.) The latest diagnostic comes from Errol Friedberg (no relation to me), editor in chief of DNA Repair.

“—if peer review is so central to the process by which scientific knowledge becomes canonized, it is ironic that science has little to say about whether it works.” — J.P. Kassirer and E.W. Campion, Peer review: crude and understudied, but indispensable, JAMA 272 (1994), pp. 96–9

The conclusion that was reached after a few annual scientific conferences published in the Journal of the American Medical Association as to the merit of peer-review were: “(i) blinding reviewers to authors’ identity does not usefully improve the quality of reviews, (ii) there is no association between reviewers signing their reviews and the quality of the review, (iii) passing reviewers’ comments to co-reviewers has no obvious effect on the quality of review, (iv) reviewers aged under 40—-write reviews of slightly better quality, (v) appreciable bias and parochialism exists in the review system. Finally, and perhaps most significantly, developing a useful instrument(s) to measure manuscript quality remains a huge challenge“.  [and in the final analysis peer review]   “can screen out [studies] that are poorly conceived, poorly designed, poorly executed, trivial, marginal, or uninterruptable.” No mean feat, really. But many scientists maintain that peer -review is a screen for quality and impact, not just for screening out bad science for funding agencies and for journals.

Neither Errol Friedberg, nor the authors of the congress proceedings seem to suggest alternatives. Rather, they present examinations of the process and its effect upon the final outcome.  Friedberg also suggests that one constraint, that of page numbers in a journal, has been essentially removed with the advent of electronic publication, and thus more meritorious articles can now be published. Interestingly enough, many scientists — and journals — seem to value publication quotas, as those add prestige to those papers that do get accepted.

However, there are two things of which I’m certain: change, if any, will not come soon, but also we have not heard the last critique of the peer-review system.

Friedberg, E. (2010). Peer review of scientific papers—A never-ending conumdrum DNA Repair DOI: 10.1016/j.dnarep.2010.03.003

JSUR Call for Participation

Submit your short (2-4page) and full length manuscripts to the Journal
of Serendipitous and Unexpected Results.

Over the past month we’ve received a great amount of press and
publicity for the Journal of Serendipitous and Unexpected Results
(JSUR). Thanks to everyone who helped spread the word, please keep it
up!

In Richard Feynman’s 1966 Nobel Lecture, he said, “We have a habit in
writing articles published in scientific journals to make the work as
finished as possible, to cover up all the tracks, to not worry about
the blind alleys or describe how you had the wrong idea first, and so
on. So there isn’t any place to publish, in a dignified manner, what
you actually did in order to do the work.”

We’re writing to invite you to solicit short (2-4page) and full length
submissions to JSUR.  Why not prepare a 2-4 page writeup discussing
side-investigations, alleyways, or false-starts in your latest
published or unpublished research? Papers of this length place a
minimal burden on the authors, while providing extremely valuable
research insights to a broad audience.

Journal website: http://www.jsur.org

Sincerely,
The JSUR Editorial Board

Overall, we are starting to see a divergence in sequencing technologies, as each tech concentrates on having clearly defined advantages and potential applications that differ from all others. This means that the scientists themselves can more closely tailor their choice of tech to fit their situation. Are you a small lab that needs 10 high-quality genomes on a budget? Go to Complete. Want a cheap, fast machine for library validation? Use Ion Torrent. Setting up a pipeline for sequencing thousands of genomes? Go Illumina.

The review article on metagenomics I recently published in PLoS Computational Biology (yeah, yeah, shameless plug) already starts to feels somewhat outdated on the sequencing technology front.

Carnival of Evolution #21 the superstar edition is up: check it out. It’s a nice and detailed one,. Some posts I liked included talking about how human fingers evolved, and why it is important to consult evolutionary biologists when making decision about conservation.

An interesting email I got yesterday: PubGet, a search engine for PDFs of scientific articles, is no linked to PLoS. PubGet is a very useful service that gets  you the article PDF immediately, without going through he usual clickeroo via Google,  pubmed, publisher’s gateway, journal gateway and then squinting along the sidebar to find the PDF link. Nice to see that these two are teaming up.

Finally, two reminders. First, Ada Lovelace day, a blogging day celebrating the achievements of women in science and technology is coming up, March 24. Go ahead, pledge and blog! Second, the Byte Size Biology will be hosting a Carnival of Bioinformatics. Quite a few posts have been submitted already, please submit yours, deadline: March 9.

The most exciting phrase to hear in science, the one that heralds new
discoveries, is not ‘Eureka!’, but ‘That’s funny…’ -Isaac Asimov

Thanks to Ruchira Datta for pointing out this one.

Science is many things to many people, but any lab-rat will tell you that research is mainly long stretches of frustration, interspersed with flashes of satisfying success. The best laid schemes of mice and men gang aft agley. A scientist’s path contains leads to blind alleys more than anything else, and meticulous experimental preparation only serves to somehow mitigate the problem, if you’re lucky. This doesn’t work, that doesn’t work either and this technique worked perfectly in Dr. X’s lab, why can’t I get this to work for me?  My experiment was invalidated by my controls; my controls didn’t work the way the controls were supposed to work in the first place. I keep getting weird results from this assay. I can’t explain my latest results in any coherent way… these statements are typical of daily life in the lab.

This stumped and stymied day-to-day life is not the impression of science we get from reading a research paper, when listening to a lecture, or when watching a science documentary show. When science is actually presented, it seems that the path to discovery was carefully laid out, planned and  flawlessly executed, a far cry from the frustrating, bumbling mess that really led to the discovery. There are three chief reasons for the disparity between how research is presented, as opposed to what really goes on. First, no one wants to look like an idiot, least of all scientists whose part of their professional trappings is strutting their smarts. Second, there are only so many pages to write a paper, one hour to present a seminar or one hour for a documentary: there is no time to present all the stuff that did not work. Third, who cares about what didn‘t work? Science is linked to progress, not to regress. OK, you had a hard time finding this out, we sympathize and thank you for blazing the trail for the rest of us. Make a note for yourself not to go into those blind alleys that held you back for years and move on. We’re not interested in your tales of woe.

Only maybe these tales of woe should be interesting to other people. If you make your negative results public, that could help others avoid the same pitfalls you had. If you share the limits of a technique, a protocol or software then someone can avoid using it in a way that does not work. A lab’s publications are actually the tip of the sum total of its accumulated knowledge.Every lab has its own oral tradition of accumulated do’s and dont’s. Not oral in the literal sense: they may even be written down for internal use, but never published. UPDATE (2-FEB-2010): most peer-reviewed journals don’t like stuff that does not work. Thanks to Mickey Kosloff for pointing out the Journal of Negative Results in Biomedicine and The Journal of Negative Results – Ecology and Evolutionary Biology.

Until now.

The Journal of Serendipitous and Unexpected Results aims to help us examine the sunken eight-ninths of the scientific knowledge iceberg, in life science and in computer science. (So an additional field over JNRB and JNREEB). From JSUR’s homepage:

Help disseminate untapped knowledge in the Computational or Life Sciences

Can you demonstrate that:

* Technique X fails on problem Y.
* Hypothesis X can’t be proven using method Y.
* Protocol X performs poorly for task Y.
* Method X has unexpected fundamental limitations.
* While investigating X, you discovered Y.
* Model X can’t capture the behavior of phenomenon Y.
* Failure X is explained by Y.
* Assumption X doesn’t hold in domain Y.
* Event X shouldn’t happen, but it does.

The problem with the JSUR model, and the nature of discovery

I expect JSUR will be a great way to comment on  methods and techniques. Indeed it will codify a trend that has been going on for some time: public protocol knowledge sharing. Many sites like openwetware,  seqanswers or the UC Davis bioinformatics wiki have been doing this for a while. Not to mention a plethora of blogs. Scientists are willing to share their experience with working protocols and procedures, and if this sharing of knowledge can be now monetized to that all-important coin of academia, the  peer-reviewed publication, all the better.

So where is the problem? The problem lies with discovery, and credit given towards it. It would be very hard to get anyone to share awkward, unexpected or yet-uninterpreted results. First, as I said, no one wants to look like an idiot. Second, unexpected or yet uninterpreted results are often viewed as a precursor to yet another avenue of exploration. A scientist would rather pursue that avenue, with the hope of  the actual meaningful discovery occurring in the lab. At most, there will be a consultation with a handful of trusted colleagues in a closed forum. If the results are made public, someone else might take the published unexpected and uninterpreted results, interpret them using complementary knowledge gained in their lab, and publish them as a bona-fide research paper. The scientist who catalyzed the research paper with his JSUR publication receives, at best, secondary credit. The story of Rosalind Franklin’s under-appreciated contribution to the discovery of the structure of DNA comes to mind. Watson and Crick used the X-ray diffraction patterns generated by Franklin to solve the three dimensional structure of the DNA molecule. Yet she was not given a co-authorship on the paper. (And she did not even make the results public, they were shared without her knowledge.) Unexpected results are viewed either as an opportunity or an embarrassment, and given the competitive nature of science, no on wants to advertise either: the first due to the fear of getting scooped, the second for fear of soiling a reputation. I expect JSUR would have a harder time filling in the odd-results niche, but I hope I am wrong.

But if you have protocols you are willing to share…what are you waiting for? Get those old lab notebooks, 00README files, forum posts  and start editing them to a paper. You are sitting on a goldmine of publishable data and you did not even realize it.

Finally, here are two scientists who never declined sharing their unexpected results.

This post has been slashdotted. Exercise extreme caution.

Dear Editor,

I would like to thank the editorial board and the referees for their comments and contributions to our manuscript. We have carefully considered the comments when rewriting the manuscript, and believe it to be much improved now…

…Oh, screw this. Let’s cut the bull. Mmkay?

Referee #1 did not even bother to read the paper. He basically glanced at the references, realized he was not cited enough to his taste, got pissed off, and attached a Pubmed dump of his papers in the last 10 years. All three of them. There is a reason none of these papers went beyond a single digit number of citations: they suck! Also, I fail to see how a paper discussing semantic distances as applied to an “endoplasmatic reticulum membrane elasticity ontology” has anything to do with my paper. Or with anything of interest, for that matter.

Referee #2 requested reanalysis of our data, using Boyle-Scott statistics. Applying Boyle-Scott statistics to our work would be like draping a hornet’s nest with clingwrap while wearing a bathing suit: a long and painful process which is utterly pointless. B-S statistics are exactly what they are, and if you think I will be bothered to do that, with my grad student finally graduating and taking off, you’re as delusional as Dr. Boyle was when he was researching REM sleep in cannabis-treated amphibians just before he went completely schizo and had to be locked up.

Referee #3 Actually read the manuscript carefully. Which is both commendable and rare. Unfortunately, judging by the comments presented, it was not my manuscript.

Finally, I would request that you as an editor grow a brain. Did you even read their comments before passing them on to me? Shipping out papers to referees, then getting them back, pasting them together and slapping on some boilerplate text from your journal’s editor’s site is not editorial work. In fact, a middle school student that volunteers in my lab wrote up a script yesterday that does just that. We are thinking of installing it in your esteemed journal’s author’s website and waiting to see if this editorial version of the Turing test would pass. We are very optimistic about the results, and we plan to write a paper about them.

Sincerely,

Prof. I. M. Irritated

Today CBSG continues with its pointers for budding scientists with the second part on serving as a peer reviewer for papers and grants.

Okay, you’ve decided that you are going to reject a manuscript. The naive reviewer might think that it is enough to simply state the reasons for the rejection as clearly and succinctly as possible. But this overlooks a major issue: ensuring that the authors do not know that it is you who rejected the manuscript.

Because the peer review process is anonymous, this may seem like no concern, as long as you extirpate all references to your own work to keep your identity secret. Wrong! You have to keep in mind that no matter how crappy the paper is, the authors are going to be pissed that it is rejected, and they are going to immediately begin wracking their brains to identify referees who might have done the dirty on them. Most will form a list of at least 5 or 6 people that they think are likely to have screwed them. Since most papers are reviewed by no more than 2-3 reviewers, this means you have a good chance of being on the list even if you were NOT the reviewer. Thus, particular pains must be taken to direct the authors ire elsewhere. Several different means to accomplish this are described below:

1. Pretend that you are British. (Note — this does not work well if you actually are British).

Just a few decades ago, it was enough to include a liberal sprinkling of “rathers” and “doubtlesses” throughout the review, and convert all colors to colours, analyze to analyse, polymerize to polymerise, etc. However, the increasing intellectual and cultural cross-pollination brought by the internet has rendered such limited measures ineffective. Thus, you need to be au courant with all the most specific idioms available to the average Brit.

For example, you might want to refer to a poorly run gel as being “dodgy”, “gammy” or “a bit pear-shaped”. Especially effective are slang terms derived from cricket. This is because no self-respecting American knows anything about this sport (indeed, outside the British Commonwealth, cricket is universally reviled as the one sport even more boring than baseball). Here are some cricket-based phrases worked into sentences that you might include in a review. Instead of writing “Some of the data presented by the authors are mutually contradictory” write “The authors seem to have gotten themselves into a bit of a sticky wicket”.

Instead of writing “The documentation of morpholino efficacy by monitoring expression of exogenously provided target rather than the endogenous target is not quite fair” write “Using GFP-ponticulin as a read out for the morpholino effects is not quite cricket”. And, instead of writing “I was chagrined to see that the authors ignored the previous studies by the Jones lab”, write “the failure of the authors to cite the seminal studies of Jones and colleagues hit me for six”.

1B. Pretend that you are an American pretending to be British (Note: this does work if you are British, but does not work if you are American.) The strategy here is similar to #1 above, but instead of being a little bit subtle, you go straight over the top. Thus, instead of writing “I seriously doubt that anyone will believe …”: “Blimey! Blokes would have to be right daft if they were to believe …”

2. Pretend that you are Canadian. This is harder because the only major language difference between Americans and Canadians is that the latter tend to mispronounce words with the short O sound such that they rhyme with newt. Needless to say, this sort of thing is not manifest in written reviews.

However, the canny reviewer can draw on the one or two features of Canadian culture that are unique. Interestingly (in light of the cricket discussion above) most of these revolve around Canadian football. For example, you might allude to a paper not being ready for the Grey Cup yet (a reference to the Canadian equivalent of the Super Bowl), describe an experimental situation as being “3rd and long” (an allusion to the fact that there are only three downs in Canadian football) or argue that the authors need to “bring in a couple more coaches” (referring to the fact that Canadian football teams have 4 head coaches). Cite obscure Canadian journals: “J Can. Med. Assoc.” or “Can. J. Cardio.” No one outside of Canada reads these journals.

3. Pretend that you are German. This is even harder, because even if you know some German, you have to write your review in English for most journals. Be extremely precise and technical. You could also try simply putting the verb at the end of your sentences (as in “The experiments in figures 5 and 6 should repeated be”), however this runs the risk of having yourself labeled not as a German, but as an imbecile or an incarnation of Yoda. Alternatively cite organic chemistry articles from the late 19th and early 20th century that have never been translated into English. Cite German aricles during the 30s and 40s when the rest of Academia was trying its best to ignore German science.

3B. Pretend that you are an American pretending to be German; sprinkle the text with flavorful comments such as “Ach mein lieber!” or “Du spinnst!” Or, if a line of reasoning is particularly awful, “Ist gibt ein Blutbat en der Hoelle!” Stick umlauts on random words, and make liberal use of the eszett. Downside: the editor will conclude you have flipped.

4. Pick one of the people from you own list of 5-6 enemies and pretend to be that person. Heavily cite their work. Reference their obscure conference presentations. Arrogantly suggest that person’s methods in favor of the methods used in the paper, especially where they are clearly inapplicable

One problem that I am facing is convincing colleagues of the utility of an Open Access publication. The usual arguments: more visibility, retention of the right to re-use material, the Greater Good, taxpayer access to taxpayer-funded research and so on don’t stick very well when faced with a $1500-$2500 or higher publication fee. These can be very big expenses if one is working on medium to small size grants, and where publication fees are sought, in part, from the College. Note: in many case the OA fees are not unaffordable; one would not request, in good faith, that the fees be waived or discounted by the publisher. But if one can use this money to pay the summer salary of a couple of more students, go to a conference, or upgrade / repair equipment, then the utility of shelling out this money for a publication seems marginal and pying this money for publication fees seems almost frivolous. In the US, funding agencies require, at most, that publications resulting from their funding would, be available on Pubmed Central within a certain time period and many non-OA publications comply, or they would lose the ability to publish a large chunk of NIH/NSF funded research projects. But doing so is not really timely OA. The bottom line is, if the grant is smaller than R01 size, many applicants would rather budget the expected $8000 of OA fees for the 3-4 year grant period for other line items that have a more palpable payoff, so to speak.

I don’t really have a point to this post, other than raising a problem that seems to be ignored, or marginalized, by many OA advocates. Not everyone operates on large grants. Many lab budgets leave very little room to buy a new laptop, let alone pay for an OA publication (typically the price of two of said laptops).

I am Distinguished Professor First Class Nebulous Nimbus, Department of Organismal Motility of the University Technicality of Upper Freedonia. I have many articles accepted and pending in PLoS Biology, PNAS, and BMC. Unfortunately I cannot pay the Open Access publication costs as my University has suffered abysmally from ill-advised investments in derivatives both partial and directional applied by the Math & Freakonomics department. A plaque on both their houses.

Sir, your reputation as a reverent and eminent scientist proceeds you. I have carefully sifted you for to assist Freedonian science from bottomless finance pit. I would be graciously to add  you  as honorific author in good position and standing to my articles, if you would be so kind as to send me Western Union the publication money needed by these journals in most urgent immediacy.

Please contact me in highest importunate on this matter: nebnim@ufd.ac.fd

Sincerely,

Docent Professor Doktor Nebulous Nimbus

(Celebrate #oaw09 Open Access Week)

But we do not only look for first impressions. We look for long-lasting impressions, we want to see the future. Our future of course, but also the future of our kith and kin. After all, our kin carry some of the genes we are imbued to disperse: we would like to take care of that. But also our kith, our extended tribe members, current, future and pending: if we take this wolf to the pack will it be able to hunt as well as the rest of us? Will it slow down the pack during migrations?  Will it dominate the herd in a year? Will it steal all our females and eat all of our cubs? Will it not pull its weight during hunting expeditions?

Credit: WickedSunshine.com

Welcome to the loopy and lupine world of metrics.

The wolfpacks of academia (read: departments) have a whole culture of ranking and assessments. Before the tenure-track wolf is accepted, a long list of future metrics are being brought out: in which packs did he PhD and postdoc? What do the pack leaders say about him? (reference letters) How good are his hunting skills (papers, conferences, invited talks) How good are his social skills? (Interview, more reference letters, phone calls).

After Dr. Wolf is finally accepted in the pack (from about 150 howling to get in), the hunting and fighting skills are put to careful periodic testing: how many grants? How much money? From which agencies? How many conference talks? How many invited talks? How are the teaching evaluations? And of course: how is the research?  How many papers? Where? What is the impact factor of the journals in which Dr. Wolf publishes? In some (I would like to think more enlightened) packs, other article-level metrics are being used. At the same time there are, of course,  the personal metrics:  What is Wolf’s h-index? g-index? h-b index?

Dear wolves, cubs and assorted members of Kingdom Animalia: what is your favorite Canis lupus related metric if at all? Poll on the right, you know the drill.

• Send me sequence data in a MS-Word,  PDF or pretty much anything else that is not a text file. No, PowerPoint is not an acceptable file format either.
• Send me sequence data not in FASTA format. Unless there is a compelling reason, FASTA only please.
• Please compress big files before you email me. Or let me know in advance that they are big, we’ll get them across by FTP or somesuch.
• Send me image files of protein structure prediction from some online server with the tag “what do you think”? How should I know what to think?About what?  Nice colors man, try using green for your beta strands the next time, brings out your eyes. Also, if you want to perform structure prediction, approach it just like any other experiment. Take time to think what you are doing. Or come to me if you are not sure before you do a 3 day run.
• Say “78% homology”.. OK, but I wrote about that before. More than once.
• “Can you please BLAST this sequence for me and tell me what you think”? Huh? What is this? Why this particular sequence? How did you come by it? Why do you want to BLAST it? What is your scientific question?
• Actually, the above is probably the most common problem. No question on hand.  Usually, when I manage to pry the question out of you, we find out that BLAST against the nr database with default values might not be exactly what the doctor ordered. (At least not Dr. Friedberg).
• “I really need to get some nice blast/tree/multiple sequence alignments for this grant application I am writing”. Always happy to help, but not 48hrs before the submission deadline. I have my own research and a life too, such as it is.
• No follow-up: OK, my lab did some work for you, anything between a couple of days and a couple of  months.  Now what? Can you give a sign of life letting me know if anything came out of it? Most hypotheses go down the drain, sure. Or sometimes funding runs out, things get prioritized differently, a postdoc leaves… but let me know! I worked quite a bit on this problem, I think that I deserve to know what happened with my work.  Have some common courtesy.