Hats off to Jonathan Eisen for hosting this activity on his blog. (I’ll keep mine on, thank you. It’s raining cats and dogs here right now).

A couple of weeks ago I posted a discussion about two papers that challenged the ortholog conjecture. Briefly, both papers stated that orthologs may not be such great predictors for molecular function. One study from  Indiana University by  has shown that paralogs may be better predictors than orthologs for molecular function. Or, at the very least, paralogs should not be excluded as predictors. This paper has generated quite a bit of interest and controversy. Consequently, Eisen has invited Matthew Hahn, the lead author to write about “the story behind the story” in Eisen’s well-read blog. The post is a great read, and has generated an animated discussion in the comments area. You do need to clear quite a bit of time to go through both Hahn’s guest post and the comment thread: the topic is a rather complex one, and as explained in the comments thread, one problem is that the ‘ortholog conjecture’ itself seems to be not well-defined.

I kept checking in to Eisen’s blog to read the elongating comment thread. It seems that now a special session on the topic may be in the works for the 2012 annual meeting of the Society for Molecular Biology and Evolution following this discussion. So great to see such an involved community getting together.

The trouble with genomic sequencing, is that it is too cheap. Anyone that has a bit of extra cash laying around, you can scrape the bugs off your windshield, sequence them, and write a paper. Seriously?

Yes, seriously now: as we sequence more and more genomes, our annotation tools cannot keep up with them. It’s like unearthing thousands of books at some vast archaeological dig of an ancient library, but being able to read only a few pages here and there. Simply put: what do all these genes do? The gap between what we do know and what we do not know is constantly growing. We are unearthing more and more books (genomes) at an ever-increasing pace, but we cannot keep up with the influx of new and strange words (genes) of this ancient language. Many genes are being tested for their function experimentally in laboratories. But the number of genes whose function we are determining using experiments is but a drop in the ocean compared to the number of genes we have sequenced and whose whose function is not known We may be sitting on the next drug target for cancer or Alzheimer’s disease, but those proteins are labeled as “unknown function” in the databases.

The red line is the growth of protein sequences deposited in TrEMBL, a comprehensive protein sequence database. The blue line illustrates the growth proteins in TrEMBL whose function is know, or at least can be predicted with some reasonable accuracy. The green line is the growth in the proteins whose 3D structure has been solved. Note the logarithmically increasing gap between what we know (blue) and what we do not know (red). Image courtesy of Predrag Radivojac.

Enter bioinformatics. CPU hours are cheaper than high throughput screening assays. And if the algorithms are good, software can do the work of determining function much cheaper than experiments. But therein lies the rub: how do we know how well function prediction algorithms perform? How do we compare their accuracy? Which method performs best, and are different methods better for different types of function predictions? This is important because most of the functional annotations in the databases come from bioinformatic prediction tools, not from experimental evidence. We need to know how accurate these tools are. Think about it this way: even an increase of 1% in accuracy  would means that hundreds of thousands of sequence database entries are better annotated, which in turn means a lot less time in the lab or in high throughput screening labs going after false drug leads.

So a few of us got together and decided to run an experiment to compare the performance of different function prediction software tools.  We call our initiative the CAFA challenge: Critical Assessment of Function Annotation. There are many research groups that are developing algorithms for gene and protein function prediction, but those have not been compared on a large scale, yet. OK then: let’s have some fun. We, the CAFA challenge organizers, will release the sequences of some 50,000 proteins whose functions are unknown. The various research groups will predict their functions using their own software. By January 2011 all the predictions should be submitted to the CAFA experiment website. Over the net few months, some of these proteins will get annotated experimentally. Not many, probably no more than a few hundred judging by the slow growth of the experimental annotations in the databases. But we don’t need that many to score the predictions. A few dozen will do.

On July 15, 2011 we will all meet in Vienna, and hold the first-ever CAFA meeting as a satellite meeting of ISMB 2011. This will be the fifth Automated Function Prediction meeting we have been holding since 2005. Only this time, there won’t just be the usual talks and posters, there will be the results of a very interesting experiment. The International Society for Computational Biology is generously hosting our meeting, and judging by the response we are getting so far, we will need one of the larger halls.

Learn more at http://biofunctionprediction.org If computational protein function prediction is your thing, join the CAFA challenge. If you are just an interested observer, keep an eye on the site. In any case, please spread the word.  Finally, if your company wants some publicity, get in touch! We could use the sponsorship ^_^

Acknowledgements: I would like to thank the CAFA co-organizers, Michal Linial and Predrag Radivojac. The CAFA steeering committee: Burkhard Rost, Steven Brenner, Patsy Babbitt and Christine Orengo for supporting us, keeping us on the straight and narrow and for incredibly useful and insightful suggestions.  Sean Mooney and Amos Bairoch for hashing out the assessment.  Tal Ronnen-Oron and the rest of Sean Mooney’s group for setting up the CAFA website. The International Society for Computational Biology for sponsoring us. The community of computational function predictors that have participated in and supported past meetings on computational function prediction, the research groups that have registered to CAFA so far, and those that will register soon   Finally, Inbal Halperin-Landsberg for coining the name CAFA. I apologize in advance if I left someone out.

GO CAFA!

Fast forward to July 1, 2011: the meeting is so on. We have 38 teams participating in the challenge, with over 50 algorithms. We have about 600 proteins whose functions have been predicted. We have a great line up of speakers including Janet Thornton the Director of the European Bioinformatics Institute and Amos Bairoch, the founder of SwissProt. The full program is available. The Radivojac, Mooney and Friedberg labs have been working very hard, and we do have some really great results to report, and some surprising activities. We are also grateful to the NIH and the US Department of Energy for their support.

So if you are planning on being at ISMB this year, drop by  the AFP/CAFA SIG. We are messing with something brand-new, which is what science is all about.

The National Academies Press are offering all their books in PDF format for free. The announcement yesterday created a serious traffic surge on their site. But the books are still there, and are still free. Got to buy that new 5Tb external disk now….

A woman in Chesterfield, Ohio robbed a convenience store using her MRSA-infected arm as a weapon.  Warning: graphic picture of MRSA infected arm. Or a zombie limb. You can never know in northwestern Ohio.

Scanning Electron Microscope image of Methicillin-Resistant Staphylococcus aureus or MRSA. Credit: Janice Carr. Source: wikipedia

A US vector biologist got infected with Zika virus which is  a mosquito-borne pathogen causing joint pain and fatigue. He then passed in on to his wife during sexual intercourse. This would be the first documented case of sexual transmission of an insect-borne pathogen. Also, “Honey, Iswear, I got it from a mosquito bite” is the new excuse for two-timing spouses everywhere.

April’s fool came and went on the intertubes.  My favorite was the arsenic based sea-monkeys.

Great news for geeky kids of the 1980′s: the Commodore 64 is back!  Same nice bulky keyboard-in computer, with just a little bit more than 64K of RAM underneath:

“The upcoming Commodore 64 looks exactly like the ancient model which reshaped the home computer sector decades ago, though underneath the bonnet things have been brought well into the 21st century.

“Even though the computer refrains from relinquishing its old charm, it will come packed with modern computer hardware. The new Commodore 64 model will contain a mini-ITC motherboard, a dual core Intel 525 Atom processor, DDR3 RAM between 2GB and 4GB and Nvidia Ion2 graphics unit.

“Commodore says that the device will run on the Ubuntu 10.04 LTS OS and will come with a DVD drive, with support for Blu-Ray, a multi-format memory card reader and five USB slots. The new Commodore 64 will also come with Wi-Fi support and will come pre-loaded with 3D games and other applications including a Microsoft Office compatible productivity suite.
Read more: http://www.itproportal.com/2011/04/06/iconic-commodore-64-all-set-comeback/#ixzz1IqxQ1LXX

 

Finally, remember the name: Zhuchengtyrannus magnus (zoo-CHENG TEER-ah-nus MAG-nus) or Z. magnus. A new dinosaur discovered in Zucheng, China. Related to Tyrannosaurus rex it is 4m tall, 11m long, 6 tons of carnivorous mass. The original paper.

Credit: xkcd.com

But now the science in the article itself is coming under fire. Several blog posts by notable microbiologists and biochemists  have questioned the claims made in the paper. To sum those up: yes, the microbes contain arsenate, the can grow on arsenic-rich media but there is no convincing evidence that arsenic gets incorporated into DNA, much less other molecules that use phosphate. Because this research is so much in the spotlight, the comments on it are in the spotlight too. I believe we will see some very interesting correspondence on the website and in the upcoming issues of Science.

Which brings me to the point of this post: is the peer-review publication culture undergoing a reform?  The arsenate bacteria article itself went through the peer-review mill, which means that at least three scientists which are credited as experts in the field have looked at it and given it a clean bill of health. But once it got published, hundreds of microbiologists and biochemists had a look, and many were less than convinced of some of its claims.  So which is better for the process of peer-review: three anonymous referees before publication, or 100 after? Or maybe we should use both?

A personal example: I recently  published a paper  in PLoS Computational Biology, which went through two pre-publication review cycles making it much better. However, even after those revisions an error (minor, fortunately) slipped through. A reader emailed me about it, and I immediately went to PLoS-CB‘s site and addressed that error as an inline comment in the paper. This mechanism provided by PLoS is laudable: I wish it were used more, and that other journals could provide it.

So, post-publication peer-review seems to be a good thing: it quickly identifies issues with the science, and helps to fix them.  So why is it not done more? Well, for one, there is the lack of anonymity. Post-publication commentators do not have the luxury of the official peer-reviewers of hiding their identity. Another is lack of credit: while some credit is given for pre-publication review, which is recognized as service rendered to the community, none is given yet for post-publication review. But why not? It is scientists like Rosie Redfield, Larry Moran, Jim Hu and others who did a great public service by taking the time to carefully read and then publicly critique the paper.  And in case there are still doubters of the value of science blogging, please read this piece by Larry Moran and for blogging as a career enhancer in science, “10 benefits for my career of blogging/ tweeting etc.) #fb” by Jonathan Eisen.

Where am I going with this? I’m not sure. But it seems like the fallout from the arsenate bacteria paper brings to light a new kind of science culture, in which post-publication critiques in expert science blogs are given. Perhaps all this energy could be harnessed to provide a better publication environment for research papers.  This has been going on for some time, as many science bloggers emphasize paper critique. But high profile incidents like the arsenate bacteria bring the value of post-publication review to light. To paraphrase a quote by Carl Sagan which was mentioned at the press conference held when the paper was published: “extraordinary claims attract extraordinary blogging”.

Wolfe-Simon F, Blum JS, Kulp TR, Gordon GW, Hoeft SE, Pett-Ridge J, Stolz JF, Webb SM, Weber PK, Davies PC, Anbar AD, & Oremland RS (2010). A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus. Science (New York, N.Y.) PMID: 21127214

I would have put Botany in Fangorn (because of the Ents), Microbiology in the Sea of Rhûn for beyond it are “wide uncharted lands, nameless plains, and forests unexplored” and machine learning in the Misty Mountains (close enough to Bioinformatics, statistics and Computer Science).

Also, if the social sciences are in Mordor, what does that say about Sauron?

JSUR Call for Participation

Submit your short (2-4page) and full length manuscripts to the Journal
of Serendipitous and Unexpected Results.

Over the past month we’ve received a great amount of press and
publicity for the Journal of Serendipitous and Unexpected Results
(JSUR). Thanks to everyone who helped spread the word, please keep it
up!

In Richard Feynman’s 1966 Nobel Lecture, he said, “We have a habit in
writing articles published in scientific journals to make the work as
finished as possible, to cover up all the tracks, to not worry about
the blind alleys or describe how you had the wrong idea first, and so
on. So there isn’t any place to publish, in a dignified manner, what
you actually did in order to do the work.”

We’re writing to invite you to solicit short (2-4page) and full length
submissions to JSUR.  Why not prepare a 2-4 page writeup discussing
side-investigations, alleyways, or false-starts in your latest
published or unpublished research? Papers of this length place a
minimal burden on the authors, while providing extremely valuable
research insights to a broad audience.

Journal website: http://www.jsur.org

Sincerely,
The JSUR Editorial Board

The most exciting phrase to hear in science, the one that heralds new
discoveries, is not ‘Eureka!’, but ‘That’s funny…’ -Isaac Asimov

Thanks to Ruchira Datta for pointing out this one.

Science is many things to many people, but any lab-rat will tell you that research is mainly long stretches of frustration, interspersed with flashes of satisfying success. The best laid schemes of mice and men gang aft agley. A scientist’s path contains leads to blind alleys more than anything else, and meticulous experimental preparation only serves to somehow mitigate the problem, if you’re lucky. This doesn’t work, that doesn’t work either and this technique worked perfectly in Dr. X’s lab, why can’t I get this to work for me?  My experiment was invalidated by my controls; my controls didn’t work the way the controls were supposed to work in the first place. I keep getting weird results from this assay. I can’t explain my latest results in any coherent way… these statements are typical of daily life in the lab.

This stumped and stymied day-to-day life is not the impression of science we get from reading a research paper, when listening to a lecture, or when watching a science documentary show. When science is actually presented, it seems that the path to discovery was carefully laid out, planned and  flawlessly executed, a far cry from the frustrating, bumbling mess that really led to the discovery. There are three chief reasons for the disparity between how research is presented, as opposed to what really goes on. First, no one wants to look like an idiot, least of all scientists whose part of their professional trappings is strutting their smarts. Second, there are only so many pages to write a paper, one hour to present a seminar or one hour for a documentary: there is no time to present all the stuff that did not work. Third, who cares about what didn‘t work? Science is linked to progress, not to regress. OK, you had a hard time finding this out, we sympathize and thank you for blazing the trail for the rest of us. Make a note for yourself not to go into those blind alleys that held you back for years and move on. We’re not interested in your tales of woe.

Only maybe these tales of woe should be interesting to other people. If you make your negative results public, that could help others avoid the same pitfalls you had. If you share the limits of a technique, a protocol or software then someone can avoid using it in a way that does not work. A lab’s publications are actually the tip of the sum total of its accumulated knowledge.Every lab has its own oral tradition of accumulated do’s and dont’s. Not oral in the literal sense: they may even be written down for internal use, but never published. UPDATE (2-FEB-2010): most peer-reviewed journals don’t like stuff that does not work. Thanks to Mickey Kosloff for pointing out the Journal of Negative Results in Biomedicine and The Journal of Negative Results – Ecology and Evolutionary Biology.

Until now.

The Journal of Serendipitous and Unexpected Results aims to help us examine the sunken eight-ninths of the scientific knowledge iceberg, in life science and in computer science. (So an additional field over JNRB and JNREEB). From JSUR’s homepage:

Help disseminate untapped knowledge in the Computational or Life Sciences

Can you demonstrate that:

* Technique X fails on problem Y.
* Hypothesis X can’t be proven using method Y.
* Protocol X performs poorly for task Y.
* Method X has unexpected fundamental limitations.
* While investigating X, you discovered Y.
* Model X can’t capture the behavior of phenomenon Y.
* Failure X is explained by Y.
* Assumption X doesn’t hold in domain Y.
* Event X shouldn’t happen, but it does.

The problem with the JSUR model, and the nature of discovery

I expect JSUR will be a great way to comment on  methods and techniques. Indeed it will codify a trend that has been going on for some time: public protocol knowledge sharing. Many sites like openwetware,  seqanswers or the UC Davis bioinformatics wiki have been doing this for a while. Not to mention a plethora of blogs. Scientists are willing to share their experience with working protocols and procedures, and if this sharing of knowledge can be now monetized to that all-important coin of academia, the  peer-reviewed publication, all the better.

So where is the problem? The problem lies with discovery, and credit given towards it. It would be very hard to get anyone to share awkward, unexpected or yet-uninterpreted results. First, as I said, no one wants to look like an idiot. Second, unexpected or yet uninterpreted results are often viewed as a precursor to yet another avenue of exploration. A scientist would rather pursue that avenue, with the hope of  the actual meaningful discovery occurring in the lab. At most, there will be a consultation with a handful of trusted colleagues in a closed forum. If the results are made public, someone else might take the published unexpected and uninterpreted results, interpret them using complementary knowledge gained in their lab, and publish them as a bona-fide research paper. The scientist who catalyzed the research paper with his JSUR publication receives, at best, secondary credit. The story of Rosalind Franklin’s under-appreciated contribution to the discovery of the structure of DNA comes to mind. Watson and Crick used the X-ray diffraction patterns generated by Franklin to solve the three dimensional structure of the DNA molecule. Yet she was not given a co-authorship on the paper. (And she did not even make the results public, they were shared without her knowledge.) Unexpected results are viewed either as an opportunity or an embarrassment, and given the competitive nature of science, no on wants to advertise either: the first due to the fear of getting scooped, the second for fear of soiling a reputation. I expect JSUR would have a harder time filling in the odd-results niche, but I hope I am wrong.

But if you have protocols you are willing to share…what are you waiting for? Get those old lab notebooks, 00README files, forum posts  and start editing them to a paper. You are sitting on a goldmine of publishable data and you did not even realize it.

Finally, here are two scientists who never declined sharing their unexpected results.

This post has been slashdotted. Exercise extreme caution.