First, I agree with Kyrpides – community annotation doesn’t work (very well), and I don’t think it ever will, so long as the goal is to generate a canonical annotation for all organisms. Realistically, we don’t have time or understanding enough (yet) in the community to do that job. I’ve been involved in a number of sequencing and annotation projects – mostly bacterial, but some eukaryotes. These projects have sometimes involved intensive manual annotation (the early ones), and later ones have been almost entirely community-based. Manual annotation is, everyone seems to agree, labour-intensive. For one bacterial genome, we had four or five annotators at a time, working for four months to get about 4500 genes annotated (with the usual proportion of hypothetical calls) – that’s approximately 18 person-months per genome, being optimistic. This is a long time, given the rate at which sequencing is progressing. Those projects that used community annotation have fallen between two extreme forms: (i) the active involvement of the organism’s research community, who are also invested in the sequencing project; (ii) passive involvement of anyone interested in an organism or gene family who might happen along to the community annotation database and make a contribution. The projects that fell closer to the first kind worked better than those that fell towards the second kind. Sadly, the second kind is nearer to the Wikipedia model that is often held up to ‘work’. In my opinion, the reasons that the Wikipedia model works for Wikipedia and not (so far) for community annotation are complex, but touch on: visibility of the annotation resource; time available to potential annotators; motivation/reward for annotators; perceptions of what an annotation *is*; and the absolute amount of expertise available (there are not, for example, as many people interested in polyketide synthases as there are pedantic Rush fans). Community annotation doesn’t, also, solve the problem of the ‘static’ GenBank annotations, and the model by which changes can only be made by submitters. That ever reliable repository remains relatively immutable and the community impact on annotation is minor.

Secondly, automated annotation remains a problem. The problem is not just one of faulty gene calls, incorrect annotation transfer, inconsistency and propagation of errors, but of capacity and server load. Public annotation servers like BaSys and RAST rapidly become swamped and less responsive, regardless of their quality or mutual inconsistency. Just as there is no ‘glory’ to be had in community annotation, there doesn’t seem to be enough ‘glory’ in funding a service that can accept the total increasing global demand for annotation capability. I suspect that the answer lies in a distributed service – everyone with their own annotation server, or a grid solution (maybe with a central clearing house), for example – rather than the current bottlenecks.

On the whole I’m currently of the opinion that we have to treat annotations as rolling hypotheses, and we should be ready to correct whatever is ‘canon’ as evidence comes in. I think that *this* is the role of community annotation, and that the central repositories should recognise and adapt to this.

C/N/S can give you serious grief over posters or public conference publications sometimes. Quite a few journals do. Open access journals are more liberal with this kind of information.

Also, the problem is not stealing, although cases of outright stealing of ideas and getting away with it do happen. I was referring to the less malign grey zone of taking someone’s quarter-baked idea as expressed in their lab notebook, and running with it quicker than they can: formulating it into a proper hypothesis, testing & publishing. Or recognizing a seeming aberration in their data for what it is before they do. Things like that.

@Ian:OK, I guess we agree that annotation can be a community + experts efforts.

However, I still maintain that the WP model of community annotation (or even monitored community annotation) is problematic when the community is small-ish, as is the case for many biomolecules, cell networks that need to be annotated.

Wikipedia is, as its name suggests, encyclopaedic. Yes, of course I use Wikipedia for initial reference to most things. What’s this “Britannica”? But I generally find that the more specific the subject, the less well-written is the WP entry, if it is at all. (Yes, I do fix ‘em when I can).

It’s probably high time someone did a survey of community annotation sites to see what works and what doesn’t. I know of a couple of white elephants that are out there, but that does not mean that they are empty because community annotation is bad. It might be due to mismanagement or lack of community interest and awareness. I think that those two factors are the main hindrances

Finally, I think that incentives, and cultural change are the key. Unfortunately, the CV line “annotated 5 paths in cellnetworkpedia” is considered a null line. Worse, it might be interpreted as if you are wasting your time instead of doing “real” science. You might recognize and commend your grad students for this kind of work, as will I, but we are a minority.

A reputation economy is still problematic, at least in Life Sciences. I don’t think I am being cynical about it, and I do hope that the culture changes somehow. CS departments will recognize contributions to Open Source projects as a valid part of one’s CV (depending on context), but it seems like LS departments are lagging behind in recognizing the analogous community effort of group annotations.

If this comment is a bit messy, it’s because I didn’t sleep much last night….

Regarding the community annotation debate, I still think it is a straw man, because I don’t really think anyone is realistically talking about depending “solely” on community annotation. I may be wrong, but that’s never how I’ve understood it. The Nupedia->Wikipedia example was meant to illustrate that even Wikipedia does not depend “solely” on community annotation, unless you define “community” to include the carefully-vetted community of experts that was already engaged in writing Nupedia.

This is how I understand “community annotation” — as an effort that involves curators at every stage (as seeders, reviewers, expert contributors, etc.) — not just a random untargeted exercise in unsupervised crowdsourcing (which clearly is risky unless your desired content is very well-defined & easily-understood).

At some points you seem dangerously close to an all-out assault on Wikipedia, which frankly, I think would be a doomed rearguard action. You referred to some well-known biases in Wikipedia that essentially reflect the composition of its community: computer-related content and geek-beloved fiction are both overrepresented. Yes, this is true, and there are several other valid criticisms of Wikipedia kicking around out there, but these criticisms are hugely outweighed by its advantages.

Let me ask, do you use Wikipedia or do you use Encyclopaedia Britannica?? hmmm????

Enjoy your road trip, I’ll be following on Twitter

So I am up to my eyeballs in boxes & bubblewrap. The movers are showing up in 10 hrs. I got to get some sleep, because the first leg of the journey is Vegas: 7 hr drive from San Diego.

But I will try, since you all tool the trouble to post comments. The main sticking point seems to be community annotation. The main argument seems to be: “Wikipedia works, so community annotation can too”.

Well, Wikipedia is not monolithic. There are many partial, bad & wrong entries. Also, certain entries in WP appeal to a larger audience, hence a larger pool of potential entry writers, which enhances quality. On the other hand, certain entries are so obscure or specialized, they never get written, or get written up badly by one person, and never corrected, because there is little or no interest. There was a well known criticism of Wikipedia that devoted a mass of words to a fictional war (Ent wars in LOTR), and very little to the largest conflict in the 20th century after WWII (Congo). The same rules on interest and lack of such apply in academia: for every Hammerhead and p53, there are ORFans with certain motifs and proposed activities that should be annotated somehow, but lack any community interest (or even knowledge). Curators are for that, as well as updating teh p53s and Hammerheads of this world.

I don’t think Nikos was advocating to doing away with community annotation. He does say not to depend “solely” on community annotation. No strawman argument here, IMHO.

Again, this is a rather superficial answer, which is the best I can do at this moment, and for a while now. Sorry. Thanks you all for taking the time & trouble to comment.

Regarding your incentive Ian: a crate of J&B is on its way…

OK, back to the boxes. I’ll be Tweeting on http://twitter.com/iddux on the Family Friedberg’s adventures over the next two weeks or so.

Peace.

Regarding community annotation, I agree with what Mitch said (perhaps unsurprisingly). The criticism of “community annotation” in your post actually seems like a bit of a straw man argument, apparently railing against the idea that community annotation aspires to do away with curators. This would indeed be a fairly bizarre idea.

Rather, when I think of “community annotation”, I think of the community as *including* those curators, as well as others who wouldn’t have been able to contribute under a more bottlenecked model.

You probably know that Nupedia, a precursor to Wikipedia, solicited input solely from experts; and that Wikipedia grew out of this, explicitly rejecting that model in favor of a mixture of expert and amateur input. It’s important to recognize that this was never meant to be EXCLUSIVELY amateur input: the experts who had contributed to Nupedia were encouraged to contribute to Wikipedia too. (Some of them walked off in a huff, but that’s life.)

In biology, WikiProject RNA is one of the best examples of how this can work. After RFAM decided to migrate their annotations onto Wikipedia, many of the entries grew in size far beyond what their small team of curators could have achieved. Go and look at the Wikipedia entry for the hammerhead ribozyme and tell me again that community annotation does not work.

Certainly, many questions remain about how best to organize the process of reviewing and approving biological annotations (especially tricky details like gene structures), and you are right that community annotation is never a panacea. Wikipedia had the advantage of a straightforward and easily-understandable model that everyone had already seen (encyclopaedias). Even with that advantage, Wikipedia’s evolved rules and procedures for reviewing entries have become quite complicated. But these are things that can presumably be worked out for biology too.

I also think you are unnecessarily cynical at times about the possibility of scientific progress in the absence of officially-blessed career incentives. People understand that their own research can benefit from good publicly-available annotations, and this can itself be sufficient incentive to contribute. Wikipedia demonstrates this, and also illustrates another (related) phenomenon: once a community effort gets big enough, it can develop its own internal reputation economy, independent of external incentives.

But in a web world, you can make things public first, and then fix and improve them as you go along. That public access allows you to draw on a much, much larger community of people than just your little group of in-house annotators. That’s the lesson of wikipedia: it’s very difficult for your group to beat the entire rest of the world, which turns out to be full of smart, interested, and capable people.

Some people worry that someone will waste work based on early poor-quality data. And there are several responses to that: one, that even professionally vetted data contains errors, and there needs to be a feedback process for fixing those in any case, and enabling the person who found the error to fix it themselves turns out to be a decent process; two, getting rough data earlier is sometimes better than cleaner data later; three, there can and should be quality information associated with the data that’s richer than the current vetted/not-vetted distinction, and four, what if you could get an email whenever the data you’re working from is updated? Then you know about mistakes as soon as they’re found.

The real barriers to community annotation are not the ones Kyrpides talks about; they’re the cultural barriers that Iddo talks about in this post. And there are also tool improvements that can help by making individual contributions to community databases more visible and easier to evaluate. Then it’s up to the tenure review boards and granting agencies to make use of that information.

It’ll happen, but it’s going to be a generational shift. There is an opportunity now for smart institutions to broaden their evaluations beyond journal articles, though.

There’s also an opportunity to build tools to facilitate community-based processes. If anyone is interested in collaborating on those, please get in touch.