It seems like there is no institution that is more criticized in science than that of the peer-review system — an no one that is less mutable. While published paper evaluation metrics are being revised (such as the recently introduced PLoS article level metrics, or the Australian National Health and Medical Research Council abandonment of the Thomson Reuters impact factor system), the peer review system seems like it is here to stay. When asked, most scientists would probably paraphrase Churchill: “peer review is the worst system for judging science, except all others that have been tried from time to time”. (However, Churchill did have other working state models to compare with Democracy, whereas peer-review seems to have no, um, peers.) The latest diagnostic comes from Errol Friedberg (no relation to me), editor in chief of DNA Repair.
“—if peer review is so central to the process by which scientific knowledge becomes canonized, it is ironic that science has little to say about whether it works.” — J.P. Kassirer and E.W. Campion, Peer review: crude and understudied, but indispensable, JAMA272 (1994), pp. 96–9
The conclusion that was reached after a few annual scientific conferences published in the Journal of the American Medical Association as to the merit of peer-review were: “(i) blinding reviewers to authors’ identity does not usefully improve the quality of reviews, (ii) there is no association between reviewers signing their reviews and the quality of the review, (iii) passing reviewers’ comments to co-reviewers has no obvious effect on the quality of review, (iv) reviewers aged under 40—-write reviews of slightly better quality, (v) appreciable bias and parochialism exists in the review system. Finally, and perhaps most significantly, developing a useful instrument(s) to measure manuscript quality remains a huge challenge“. [and in the final analysis peer review] “can screen out [studies] that are poorly conceived, poorly designed, poorly executed, trivial, marginal, or uninterruptable.” No mean feat, really. But many scientists maintain that peer -review is a screen for quality and impact, not just for screening out bad science for funding agencies and for journals.
Neither Errol Friedberg, nor the authors of the congress proceedings seem to suggest alternatives. Rather, they present examinations of the process and its effect upon the final outcome. Friedberg also suggests that one constraint, that of page numbers in a journal, has been essentially removed with the advent of electronic publication, and thus more meritorious articles can now be published. Interestingly enough, many scientists — and journals — seem to value publication quotas, as those add prestige to those papers that do get accepted.
However, there are two things of which I’m certain: change, if any, will not come soon, but also we have not heard the last critique of the peer-review system.
Friedberg, E. (2010). Peer review of scientific papers—A never-ending conumdrum DNA Repair DOI: 10.1016/j.dnarep.2010.03.003
Submit your short (2-4page) and full length manuscripts to the Journal
of Serendipitous and Unexpected Results.
Over the past month we’ve received a great amount of press and
publicity for the Journal of Serendipitous and Unexpected Results
(JSUR). Thanks to everyone who helped spread the word, please keep it
up!
In Richard Feynman’s 1966 Nobel Lecture, he said, “We have a habit in
writing articles published in scientific journals to make the work as
finished as possible, to cover up all the tracks, to not worry about
the blind alleys or describe how you had the wrong idea first, and so
on. So there isn’t any place to publish, in a dignified manner, what
you actually did in order to do the work.”
We’re writing to invite you to solicit short (2-4page) and full length
submissions to JSUR. Why not prepare a 2-4 page writeup discussing
side-investigations, alleyways, or false-starts in your latest
published or unpublished research? Papers of this length place a
minimal burden on the authors, while providing extremely valuable
research insights to a broad audience.
Some interesting events cropped up recently. The Marco Island Advances in Genome Biology and Technology meeting was heavily tweeted and blogged about. Pacific Biosciences unveiled their third generation sequencer. Ostensibly, it can sequence reads of 20,000 length, but the fraction of actual long reads in a run, and their quality is still a bit hazy. The most interested to me is the Ion Torrent. Being rather low on budget, this seems like the family budget car of high throughput sequencing: cheap, reliable, and does not offer more than I really need. $50,000 for a sequencer with $500 runs with 160MB/hr? Nice. Genetic Inference has a great summary of the various technologies presented.
Overall, we are starting to see a divergence in sequencing technologies, as each tech concentrates on having clearly defined advantages and potential applications that differ from all others. This means that the scientists themselves can more closely tailor their choice of tech to fit their situation. Are you a small lab that needs 10 high-quality genomes on a budget? Go to Complete. Want a cheap, fast machine for library validation? Use Ion Torrent. Setting up a pipeline for sequencing thousands of genomes? Go Illumina.
The review article on metagenomics I recently published in PLoS Computational Biology (yeah, yeah, shameless plug) already starts to feels somewhat outdated on the sequencing technology front.
An interesting email I got yesterday: PubGet, a search engine for PDFs of scientific articles, is no linked to PLoS. PubGet is a very useful service that gets you the article PDF immediately, without going through he usual clickeroo via Google, pubmed, publisher’s gateway, journal gateway and then squinting along the sidebar to find the PDF link. Nice to see that these two are teaming up.
Finally, two reminders. First, Ada Lovelace day, a blogging day celebrating the achievements of women in science and technology is coming up, March 24. Go ahead, pledge and blog! Second, the Byte Size Biology will be hosting a Carnival of Bioinformatics. Quite a few posts have been submitted already, please submit yours, deadline: March 9.
Science is many things to many people, but any lab-rat will tell you that research is mainly long stretches of frustration, interspersed with flashes of satisfying success. The best laid schemes of mice and men gang aft agley. A scientist’s path contains leads to blind alleys more than anything else, and meticulous experimental preparation only serves to somehow mitigate the problem, if you’re lucky. This doesn’t work, that doesn’t work either and this technique worked perfectly in Dr. X’s lab, why can’t I get this to work for me? My experiment was invalidated by my controls; my controls didn’t work the way the controls were supposed to work in the first place. I keep getting weird results from this assay. I can’t explain my latest results in any coherent way… these statements are typical of daily life in the lab.
This stumped and stymied day-to-day life is not the impression of science we get from reading a research paper, when listening to a lecture, or when watching a science documentary show. When science is actually presented, it seems that the path to discovery was carefully laid out, planned and flawlessly executed, a far cry from the frustrating, bumbling mess that really led to the discovery. There are three chief reasons for the disparity between how research is presented, as opposed to what really goes on. First, no one wants to look like an idiot, least of all scientists whose part of their professional trappings is strutting their smarts. Second, there are only so many pages to write a paper, one hour to present a seminar or one hour for a documentary: there is no time to present all the stuff that did not work. Third, who cares about what didn‘t work? Science is linked to progress, not to regress. OK, you had a hard time finding this out, we sympathize and thank you for blazing the trail for the rest of us. Make a note for yourself not to go into those blind alleys that held you back for years and move on. We’re not interested in your tales of woe.
Only maybe these tales of woe should be interesting to other people. If you make your negative results public, that could help others avoid the same pitfalls you had. If you share the limits of a technique, a protocol or software then someone can avoid using it in a way that does not work. A lab’s publications are actually the tip of the sum total of its accumulated knowledge.Every lab has its own oral tradition of accumulated do’s and dont’s. Not oral in the literal sense: they may even be written down for internal use, but never published. UPDATE (2-FEB-2010):most peer-reviewed journals don’t like stuff that does not work. Thanks to Mickey Kosloff for pointing out the Journal of Negative Results in Biomedicine and The Journal of Negative Results – Ecology and Evolutionary Biology.
Until now.
The Journal of Serendipitous and Unexpected Results aims to help us examine the sunken eight-ninths of the scientific knowledge iceberg, in life science and in computer science. (So an additional field over JNRB and JNREEB). From JSUR’s homepage:
Help disseminate untapped knowledge in the Computational or Life Sciences
Can you demonstrate that:
* Technique X fails on problem Y.
* Hypothesis X can’t be proven using method Y.
* Protocol X performs poorly for task Y.
* Method X has unexpected fundamental limitations.
* While investigating X, you discovered Y.
* Model X can’t capture the behavior of phenomenon Y.
* Failure X is explained by Y.
* Assumption X doesn’t hold in domain Y.
* Event X shouldn’t happen, but it does.
The problem with the JSUR model, and the nature of discovery
I expect JSUR will be a great way to comment on methods and techniques. Indeed it will codify a trend that has been going on for some time: public protocol knowledge sharing. Many sites like openwetware, seqanswers or the UC Davis bioinformatics wiki have been doing this for a while. Not to mention a plethora of blogs. Scientists are willing to share their experience with working protocols and procedures, and if this sharing of knowledge can be now monetized to that all-important coin of academia, the peer-reviewed publication, all the better.
So where is the problem? The problem lies with discovery, and credit given towards it. It would be very hard to get anyone to share awkward, unexpected or yet-uninterpreted results. First, as I said, no one wants to look like an idiot. Second, unexpected or yet uninterpreted results are often viewed as a precursor to yet another avenue of exploration. A scientist would rather pursue that avenue, with the hope of the actual meaningful discovery occurring in the lab. At most, there will be a consultation with a handful of trusted colleagues in a closed forum. If the results are made public, someone else might take the published unexpected and uninterpreted results, interpret them using complementary knowledge gained in their lab, and publish them as a bona-fide research paper. The scientist who catalyzed the research paper with his JSUR publication receives, at best, secondary credit. The story of Rosalind Franklin’s under-appreciated contribution to the discovery of the structure of DNA comes to mind. Watson and Crick used the X-ray diffraction patterns generated by Franklin to solve the three dimensional structure of the DNA molecule. Yet she was not given a co-authorship on the paper. (And she did not even make the results public, they were shared without her knowledge.) Unexpected results are viewed either as an opportunity or an embarrassment, and given the competitive nature of science, no on wants to advertise either: the first due to the fear of getting scooped, the second for fear of soiling a reputation. I expect JSUR would have a harder time filling in the odd-results niche, but I hope I am wrong.
But if you have protocols you are willing to share…what are you waiting for? Get those old lab notebooks, 00README files, forum posts and start editing them to a paper. You are sitting on a goldmine of publishable data and you did not even realize it.
Finally, here are two scientists who never declined sharing their unexpected results.
This post has been slashdotted. Exercise extreme caution.
Floated in my email inbox recently. Bears blogging.
Dear Editor,
I would like to thank the editorial board and the referees for their comments and contributions to our manuscript. We have carefully considered the comments when rewriting the manuscript, and believe it to be much improved now…
…Oh, screw this. Let’s cut the bull. Mmkay?
Referee #1 did not even bother to read the paper. He basically glanced at the references, realized he was not cited enough to his taste, got pissed off, and attached a Pubmed dump of his papers in the last 10 years. All three of them. There is a reason none of these papers went beyond a single digit number of citations: they suck! Also, I fail to see how a paper discussing semantic distances as applied to an “endoplasmatic reticulum membrane elasticity ontology” has anything to do with my paper. Or with anything of interest, for that matter.
Referee #2 requested reanalysis of our data, using Boyle-Scott statistics. Applying Boyle-Scott statistics to our work would be like draping a hornet’s nest with clingwrap while wearing a bathing suit: a long and painful process which is utterly pointless. B-S statistics are exactly what they are, and if you think I will be bothered to do that, with my grad student finally graduating and taking off, you’re as delusional as Dr. Boyle was when he was researching REM sleep in cannabis-treated amphibians just before he went completely schizo and had to be locked up.
Referee #3 Actually read the manuscript carefully. Which is both commendable and rare. Unfortunately, judging by the comments presented, it was not my manuscript.
Finally, I would request that you as an editor grow a brain. Did you even read their comments before passing them on to me? Shipping out papers to referees, then getting them back, pasting them together and slapping on some boilerplate text from your journal’s editor’s site is not editorial work. In fact, a middle school student that volunteers in my lab wrote up a script yesterday that does just that. We are thinking of installing it in your esteemed journal’s author’s website and waiting to see if this editorial version of the Turing test would pass. We are very optimistic about the results, and we plan to write a paper about them.
I didn’t write this one, but I wish I did. I found it on Science after Sunclipse. I guess that a CC license can be safely applied to anonymous chain letters.
Today CBSG continues with its pointers for budding scientists with the second part on serving as a peer reviewer for papers and grants.
Okay, you’ve decided that you are going to reject a manuscript. The naive reviewer might think that it is enough to simply state the reasons for the rejection as clearly and succinctly as possible. But this overlooks a major issue: ensuring that the authors do not know that it is you who rejected the manuscript.
Because the peer review process is anonymous, this may seem like no concern, as long as you extirpate all references to your own work to keep your identity secret. Wrong! You have to keep in mind that no matter how crappy the paper is, the authors are going to be pissed that it is rejected, and they are going to immediately begin wracking their brains to identify referees who might have done the dirty on them. Most will form a list of at least 5 or 6 people that they think are likely to have screwed them. Since most papers are reviewed by no more than 2-3 reviewers, this means you have a good chance of being on the list even if you were NOT the reviewer. Thus, particular pains must be taken to direct the authors ire elsewhere. Several different means to accomplish this are described below:
1. Pretend that you are British. (Note — this does not work well if you actually are British).
Just a few decades ago, it was enough to include a liberal sprinkling of “rathers” and “doubtlesses” throughout the review, and convert all colors to colours, analyze to analyse, polymerize to polymerise, etc. However, the increasing intellectual and cultural cross-pollination brought by the internet has rendered such limited measures ineffective. Thus, you need to be au courant with all the most specific idioms available to the average Brit.
For example, you might want to refer to a poorly run gel as being “dodgy”, “gammy” or “a bit pear-shaped”. Especially effective are slang terms derived from cricket. This is because no self-respecting American knows anything about this sport (indeed, outside the British Commonwealth, cricket is universally reviled as the one sport even more boring than baseball). Here are some cricket-based phrases worked into sentences that you might include in a review. Instead of writing “Some of the data presented by the authors are mutually contradictory” write “The authors seem to have gotten themselves into a bit of a sticky wicket”.
Instead of writing “The documentation of morpholino efficacy by monitoring expression of exogenously provided target rather than the endogenous target is not quite fair” write “Using GFP-ponticulin as a read out for the morpholino effects is not quite cricket”. And, instead of writing “I was chagrined to see that the authors ignored the previous studies by the Jones lab”, write “the failure of the authors to cite the seminal studies of Jones and colleagues hit me for six”.
1B. Pretend that you are an American pretending to be British (Note: this does work if you are British, but does not work if you are American.) The strategy here is similar to #1 above, but instead of being a little bit subtle, you go straight over the top. Thus, instead of writing “I seriously doubt that anyone will believe …”: “Blimey! Blokes would have to be right daft if they were to believe …”
2. Pretend that you are Canadian. This is harder because the only major language difference between Americans and Canadians is that the latter tend to mispronounce words with the short O sound such that they rhyme with newt. Needless to say, this sort of thing is not manifest in written reviews.
However, the canny reviewer can draw on the one or two features of Canadian culture that are unique. Interestingly (in light of the cricket discussion above) most of these revolve around Canadian football. For example, you might allude to a paper not being ready for the Grey Cup yet (a reference to the Canadian equivalent of the Super Bowl), describe an experimental situation as being “3rd and long” (an allusion to the fact that there are only three downs in Canadian football) or argue that the authors need to “bring in a couple more coaches” (referring to the fact that Canadian football teams have 4 head coaches). Cite obscure Canadian journals: “J Can. Med. Assoc.” or “Can. J. Cardio.” No one outside of Canada reads these journals.
3. Pretend that you are German. This is even harder, because even if you know some German, you have to write your review in English for most journals. Be extremely precise and technical. You could also try simply putting the verb at the end of your sentences (as in “The experiments in figures 5 and 6 should repeated be”), however this runs the risk of having yourself labeled not as a German, but as an imbecile or an incarnation of Yoda. Alternatively cite organic chemistry articles from the late 19th and early 20th century that have never been translated into English. Cite German aricles during the 30s and 40s when the rest of Academia was trying its best to ignore German science.
3B. Pretend that you are an American pretending to be German; sprinkle the text with flavorful comments such as “Ach mein lieber!” or “Du spinnst!” Or, if a line of reasoning is particularly awful, “Ist gibt ein Blutbat en der Hoelle!” Stick umlauts on random words, and make liberal use of the eszett. Downside: the editor will conclude you have flipped.
4. Pick one of the people from you own list of 5-6 enemies and pretend to be that person. Heavily cite their work. Reference their obscure conference presentations. Arrogantly suggest that person’s methods in favor of the methods used in the paper, especially where they are clearly inapplicable
One problem that I am facing is convincing colleagues of the utility of an Open Access publication. The usual arguments: more visibility, retention of the right to re-use material, the Greater Good, taxpayer access to taxpayer-funded research and so on don’t stick very well when faced with a $1500-$2500 or higher publication fee. These can be very big expenses if one is working on medium to small size grants, and where publication fees are sought, in part, from the College. Note: in many case the OA fees are not unaffordable; one would not request, in good faith, that the fees be waived or discounted by the publisher. But if one can use this money to pay the summer salary of a couple of more students, go to a conference, or upgrade / repair equipment, then the utility of shelling out this money for a publication seems marginal and pying this money for publication fees seems almost frivolous. In the US, funding agencies require, at most, that publications resulting from their funding would, be available on Pubmed Central within a certain time period and many non-OA publications comply, or they would lose the ability to publish a large chunk of NIH/NSF funded research projects. But doing so is not really timely OA. The bottom line is, if the grant is smaller than R01 size, many applicants would rather budget the expected $8000 of OA fees for the 3-4 year grant period for other line items that have a more palpable payoff, so to speak.
I don’t really have a point to this post, other than raising a problem that seems to be ignored, or marginalized, by many OA advocates. Not everyone operates on large grants. Many lab budgets leave very little room to buy a new laptop, let alone pay for an OA publication (typically the price of two of said laptops).
I am Distinguished Professor First Class Nebulous Nimbus, Department of Organismal Motility of the University Technicality of Upper Freedonia. I have many articles accepted and pending in PLoS Biology, PNAS, and BMC. Unfortunately I cannot pay the Open Access publication costs as my University has suffered abysmally from ill-advised investments in derivatives both partial and directional applied by the Math & Freakonomics department. A plaque on both their houses.
Sir, your reputation as a reverent and eminent scientist proceeds you. I have carefully sifted you for to assist Freedonian science from bottomless finance pit. I would be graciously to add you as honorific author in good position and standing to my articles, if you would be so kind as to send me Western Union the publication money needed by these journals in most urgent immediacy.
Please contact me in highest importunate on this matter: nebnim@ufd.ac.fd
One aspect of living in any kind of social setting is being assessed, rated and tested by one’s peers. Constantly. We are social creatures: we need to know who we are up against in any given setting. It is, after all, a matter of life and death, or at the very of gene dispersal. We have replaced butt-sniffing, teeth baring and chest drumming with “..the firm handshake / A certain look in the eye, and an easy smile” for first impressions. (Although I would personally take butt-sniffing over certain club ties most days.)
But we do not only look for first impressions. We look for long-lasting impressions, we want to see the future. Our future of course, but also the future of our kith and kin. After all, our kin carry some of the genes we are imbued to disperse: we would like to take care of that. But also our kith, our extended tribe members, current, future and pending: if we take this wolf to the pack will it be able to hunt as well as the rest of us? Will it slow down the pack during migrations? Will it dominate the herd in a year? Will it steal all our females and eat all of our cubs? Will it not pull its weight during hunting expeditions?
Credit: WickedSunshine.com
Welcome to the loopy and lupine world of metrics.
The wolfpacks of academia (read: departments) have a whole culture of ranking and assessments. Before the tenure-track wolf is accepted, a long list of future metrics are being brought out: in which packs did he PhD and postdoc? What do the pack leaders say about him? (reference letters) How good are his hunting skills (papers, conferences, invited talks) How good are his social skills? (Interview, more reference letters, phone calls).
After Dr. Wolf is finally accepted in the pack (from about 150 howling to get in), the hunting and fighting skills are put to careful periodic testing: how many grants? How much money? From which agencies? How many conference talks? How many invited talks? How are the teaching evaluations? And of course: how is the research? How many papers? Where? What is the impact factor of the journals in which Dr. Wolf publishes? In some (I would like to think more enlightened) packs, other article-level metrics are being used. At the same time there are, of course, the personal metrics: What is Wolf’s h-index? g-index? h-b index?
Dear wolves, cubs and assorted members of Kingdom Animalia: what is your favorite Canis lupus related metric if at all? Poll on the right, you know the drill.
As resident bioinformatician in many places over the years, I got many of requests to help. Anything from a short blast run to a full-fledged collaboration. I love that. I always like learning about new problems, and those requests may blossom into full research collaborations. So yes, drop me an email or step into my office any old time. But here are some sure-fire ways to tick me off:
Send me sequence data in a MS-Word, PDF or pretty much anything else that is not a text file. No, PowerPoint is not an acceptable file format either.
Send me sequence data not in FASTA format. Unless there is a compelling reason, FASTA only please.
Please compress big files before you email me. Or let me know in advance that they are big, we’ll get them across by FTP or somesuch.
Send me image files of protein structure prediction from some online server with the tag “what do you think”? How should I know what to think?About what? Nice colors man, try using green for your beta strands the next time, brings out your eyes. Also, if you want to perform structure prediction, approach it just like any other experiment. Take time to think what you are doing. Or come to me if you are not sure before you do a 3 day run.
Say “78% homology”.. OK, but I wrote about that before. More than once.
“Can you please BLAST this sequence for me and tell me what you think”? Huh? What is this? Why this particular sequence? How did you come by it? Why do you want to BLAST it? What is your scientific question?
Actually, the above is probably the most common problem. No question on hand. Usually, when I manage to pry the question out of you, we find out that BLAST against the nr database with default values might not be exactly what the doctor ordered. (At least not Dr. Friedberg).
“I really need to get some nice blast/tree/multiple sequence alignments for this grant application I am writing”. Always happy to help, but not 48hrs before the submission deadline. I have my own research and a life too, such as it is.
No follow-up: OK, my lab did some work for you, anything between a couple of days and a couple of months. Now what? Can you give a sign of life letting me know if anything came out of it? Most hypotheses go down the drain, sure. Or sometimes funding runs out, things get prioritized differently, a postdoc leaves… but let me know! I worked quite a bit on this problem, I think that I deserve to know what happened with my work. Have some common courtesy.
What is it?Open Notebook means “no insider information” You lab notebook is on a wiki, out there for everyone to see. Negative results & all. You share your research process with the world as you go along. There are many shades to this process: you may share some of your data, edit it, sanitize it… but he general idea holds, that you share a major part of your data, methods and thoughts prior to the official publication.
Why doesn’t it work? Social and cultural reasons. A basic tenet of science culture is that competition breeds quality and innovation. Researchers need to pass a series of competitive thresholds to be able to continue and expand their research: secure a position to be able to start your independent research, compete for a grant to fund it (at a 10-15% funding rate in the US for biomedical research), compete for more grants so one can fund an expanding vision of one’s research, pass a threshold to receive tenure (or rather, not get fired after 6 years). In places with no tenure, pass periodic reviews. Search committees, grant review panels and tenure / periodic review committees judge a scientist by the number of publications, their innovation, how attributable they are to his group as opposed to the collaborating groups and how much impact they carry in the field. Of course the $$$ brought in by grant overheads. To reach a truly innovative leap in research, there is a period when you have to play your cards close to the chest, sharing your findings only with your lab, your collaborators and trusted colleagues. Revealing findings too early will get you scooped by a better equipped lab, or at best dilute the innovative impact: your open lab notebook wiki can and will be construed as a prior publication.
Taking openness and collaboration to the extreme, if you put your notebook on a wiki, and your field is “hot” enough, you can be sure someone will use those ideas to their own benefit, very likely at your expense. It need not be malign: they could make an intuitive leap of reasoning reading your notebook before you can. Even if they are honest and generous enough to credit you by co-authorship, how much of the innovation would be attributed to you? And if you receive less credit for research innovation than you could, that would lower your evaluation score at whatever career stage you are in. By and large, this culture does not appear to be changing. The need to be identified with a certain type of research you can call “your own” and the need to innovate trump those collaborations that, in the eyes of your peers and evaluators, only serve to dilute your achievements.
Therefore, in the foreseeable future, I believe that the Open Science vision will be limited to non-competitive endeavors that don’t have potential for high-impact research papers down the line. Those usually have more to do with tool and technology development rather than innovative research. That is actually a great thing: at least open-notebook science enables protocol, tool and software development more quickly. But anyone who has been involved with Free and Open Source Software has known that for three decades or more.
Different disciplines in science have different cultures. The biomedical field is known to be especially competitive. Also, the field is going through very fast changes. I am referring to this field. I realize that things are different in physics, for example, where pre-publication of results is encouraged and credited. All the more proof that openness, or lack of it, is a cultural issue, rather than inherent in academic research.
What does work? Collaborative technologies: wikis, blogs, discussion forums are great for publicizing oneself (HEY!), asking general questions about one’s methodologies, protocols, howtos, software or equipment. OpenWetWare is an example of such a success story for the experimental biology community, being a central repository for protocols and general lab how-tos. But the lab notebooks section only contains a handful of notebooks, most of them out of date. Social bookmarking like Delicious or specialized social bookmarking like citeulike are catching on, maybe a bit slower than expected. Wikis (not open ones) are great for internal lab management as well, as more labs are discovering.
The free and open source software culture, where one is free to modify and distribute software so licensed, has enabled new feats in scientific computation infrastructure by leveling the playing field so that anyone can use, modify and re-distribute software. In a similar vein, grid technologies are leveling the field of computational power and hardware. Publications likePLoS-ONE, which accept research based on scientific rigor rather than innovation leaps and “exceptional interest” have filled the gap necessary to communicate research that is of interest, yet will not be accepted to journals demanding an innovative edge. Freely available data, post-publication, makes it easier to validate research by third parties, and build upon it. And of course, Open Access which makes publications available to all: not only to read, but to further publicize.
For another view that advocates a change in scientific culture that will make Open Science part of the academic incentive structure, just as publications are today, read here.
Community annotation
Credit: victoriapeckham Flickr
What is it? Genomics has become a data rich science. The deluge of genomes and metagenomes are to be too much to handle for a group of curators. The idea some genomic database maintainers have come up with is borrowed from the success of Wikipedia. If enough users would come in to annotate their favorite genes, we will eventually end up with a comprehensive collection of annotations for most if not all genes in a sequenced genome. If ths system is good for Wikipedia entries, why not for genes?
Why doesn’t it work?
Why would anyone expect—or even worse, depend on—a community annotation effort? Imagine investing millions of dollars into state-of-the-art sequencing facilities, and then expecting volunteers from the community to stop by and run the sequencing machines. One might argue that this analogy is not valid because running a sequencing facility requires well-trained personnel, standardized protocols, clear procedures, quality controls and, most of all, tight coordination. Yet, the same professional standards are required for data curation, and it is precisely these aspects that are rarely achieved through a community contribution approach. Community annotation should be encouraged and facilitated, but the curation of biological data cannot depend solely on volunteer work. High standards and quality implies professionalism, and this, in turn, requires investing in dedicated professionals. Until this is done, data curation—and consequently the whole field of microbial genomics—will not move beyond the amateur stage.
What does work? The failure of community based annotations has brought the often overlooked but crucial activity of biocurators into the limelight. Recently, the International Society for Biocuration was formed. From the mission statement:
Strong support from the research community, the journal publishers, and the funding agencies is indispensable for databases to continue to provide the valuable tools on which a large fraction of research vitally depends. Structured ways for biocurators and associated developers to increase the sharing of tools and ideas through conferences and high quality peer-reviewed publications need to be developed. This will improve data capture, representation, and analysis. Secondly, biocurators, researchers and publishers need to collaborate to facilitate data integration into public resources. Researchers should be encouraged to directly participate in annotation. This will lead to improved productivity and better quality of published papers as well as stronger integrity of the data represented in databases. Thirdly, funding agencies need to recognize the importance of database for basic research by providing increased and stable funding. Finally, the recognition of biocuration as a professional career path will ensure the continued recruitment of highly qualified scientists to this field, which benefits the wider world of biomedical sciences.
So it’s back to expert handling of data, perhaps with some community assistance. This goes back to the attribution problem discussed above: in the current culture, there is hardly any career-building attribution to community annotations. For true community involvement, this would need to change. At the same time, biocuration needs to be recognized as a valid and important career path.
Virtual Conferences
Credit: NASA
What is it? Why pay over $2000 for an international conference, suffer through delayed flights, lost baggage, forgotten poster tubes, jet lag, overpriced meals and hotels (“conference discount” my a$$), sweaty poster sessions and tight-fisted finance admins when you finally get home and try to get reimbursed (phew!) — when you can attend a conference using webcasting in the comfort of your home for a fraction of the price if not for free?
Why doesn’t it work? First: virtual conferencing technology sucks. It doesn’t matter if you use a free Skype on a $150 netbook, or a state-of-the art teleconferencing equipment with a 52″ screen and Dolby Surround, piped through at hundreds of Gigabits per second. You will get interruptions, cuts, lags, annoyances and embarrassing moments. Second: social reasons. The important parts of a conference take place in the hallways, poster sessions, meals, banquets and, of course, the pub across the street. Incipient collaborations, exchange of ideas, brainstorming: all those take place around the dinner table and in the halls. With food, coffee and alcohol providing the social lubrication, and the talks and posters the intellectual one. A conference is much more than a series of talks.
To summarize: until we reach a level of virtuality akin to that of the Star-Trek holodeck, or at least something that manages to sync picture & sound without one or the other dropping every 3 minutes, we have no choice but to continue taking off our shoes and belts in front of uniformed strangers.
What does work? live and archived webcasts can be an acceptable substitute to the lecture part if you could not make it to the meatspace meeting. Although you probably will not spend the time at home watching all the webcasts of all the keynote speakers you would have gone to in the conference. Microblogging is emerging as a time-saving device for those who were not there: you don’t need to devote 45 minutes to read a microblog from that talk you really wanted to attend. Done properly, perhaps with the speaker’s slides shared somewhere, it is less time consuming than watching a day’s worth of webcasts. And you can filter your interests using the microblogging notes taken by your colleagues, posted on friendfeed or such. No substitution for the real deal, which is shmoozing in the hallways. But at least you’ll get an idea about the latest & greatest in research in your field.
This is not to say that the Internet obviates socializing and work collaborations, quite the opposite of course. Most of my collaborators are time zones away from me, and I use email, chat, wikis, Googledocs, and even (shudder) Skype conference calls for working with them. But the experience of a critical mass of people meeting for real and getting things done in a very short space of time has yet to be duplicated by technological means.
The “End of Theory” science
What is it? I am referring to the Wired article penned by Wired‘s editor-in-chief, Chris Anderson last year. It generated a large response, and a resounding echo of “me too” and “he’s so right” articles and blog posts. The message of this article was that with such a deluge of data in the natural scientists, scientists can stop going through the “hypothesize, model, test” cycle. Rather, they can simply look for statistical correlation and draw conclusions from them.
Why doesn’t it work? Because it was wrong from the get-go. I don’t think any serious scientist ever went through the cycle Anderson superficially outlined. He neglected to prefix the “observe” phase to “hypothesize, model, test”. Observation – a.k.a. data collection is the foundation to whatever comes after. Scientists first observe, then if enough observations are made that seem to fit a certain trend, they formulate one or more hypotheses. Those are tested, and the hypotheses refined or discarded based on test results. Finally, some model may or may not emerge. In any case, the empirical process of research is more of an “(1)observe, (2)hypothesize, (3)test, (4)observe again, (5)retest, (5)correct hypothesis,(6) bumble through previous 5 stages for quite a while, if you’re lucky you may have a (6)model”. This is the way science is done regardless of whether you have 20 data points or 20 trillion. There are, of course, qualitative differences to large quantities of data: methods of observation and sifting through data become rather different, technology starts playing a major role: you really need that computer cluster power (see also above, on community annotation). It does not preclude the need to go through the previous stages, even more carefully than you have done with 20 data points. In the end, science is about providing explanations for observed phenomena, and that is what a model is: an explanation, the best we can come up with at this time. If you don’t have hypotheses, models and theories you don’t have science.
What does work?
M. Mitchell Waldrop (2008). Science 2.0 — Is Open Access Science the Future? Scientific American, 298 (5), 68-73 DOI: 18444327
Hoffmann, R. (2008). A wiki for the life sciences where authorship matters Nature Genetics, 40 (9), 1047-1051 DOI: 10.1038/ng.f.217
Sagotsky, J., Zhang, L., Wang, Z., Martin, S., & Deisboeck, T. (2008). Life Sciences and the web: a new era for collaboration Molecular Systems Biology, 4 DOI: 10.1038/msb.2008.39
