World Association of Young Scientists

Evaluating the quality of scientific information, or its relevance for a specific purpose, is hardly a simple endeavour. With the transition from traditional paper-based means of communication to a wide variety of communication channels, particularly online, this problem is even growing in complexity.

Fortunately, scientific information is not the only type of information on the internet, and quality assessment — to various degrees and by diverse populations — is common in many branches of society. So it may be worthwhile to consider whether some of the systems in use for other kinds of online content may be transferable to adaptable for scientific purposes.

Consider the example of movies: If you're a scientist, it may have been a while since you last went to the movies, but your memory might still hold some references to this. Anyway, a rating scheme for movies is available via OpenCritics, a non-profit startup dedicated to facilitating and openly archiving online ratings (using a Creative Commons license, as given below), and let's see how well this goes with the Science Commons video (by Jesse Dylan)


which you can rate (or comment upon; give it a try!) using the following widget (ignore that it currently says "movie"):

And while we are at it, what about other kinds of content? Well, you might give it a try with this presentation by Björn Brembs which addresses important shortcomings of the current scientific publication system:

The rating procedure is the same, though some of the options will probably have to be finetuned, and one could envision several such widgets being associated with one item but targeted at different evaluation criteria:

For feedback on the usefulness of such an approach for scientific purposes (also at article level, as currently being implemented at PLoS ONE, Mendeley and discussed also elsewhere), please leave a comment here or directly at OpenCritics. Thank you!


Licensing information: Creative Commons video: CC-BY-NC-SA, Ratings: CC-BY-ND, Text & Slideshow: CC-BY.

Here is some brainstorming on criteria suitable for the evaluation of
scientific contributions (initially perhaps just journal articles, but
in the long run also blog posts, wiki edits, project proposals,
database entries and basically anything related to science that can be
described by a Uniform Resource Identifier).

For a start, I have listed below the criteria currently in use at PLoS ONE for pre-publication assessment of research manuscripts:

1. Results reported have not been published elsewhere.
2. Experiments, statistics, and other analyses are performed to a high technical standard and are described in sufficient detail.
3. Conclusions are presented in an appropriate fashion and are supported by the data.
4. The article is presented in an intelligible fashion and is written in standard English.
5. The research meets all applicable standards for the ethics of experimentation and research integrity.
6. The article adheres to appropriate reporting guidelines (e.g. CONSORT, MIAME, STROBE, EQUATOR) and community standards for data availability.
7. The study presents the results of primary scientific research.

These points raise a number of issues with respect to adaptations to online environments:

1. "Published" in this sense refers to "making the information publicly available after peer review", while the term has other connotations in the context of blogs or wikis where it simply means "making the information publicly available", and review by experts and non-experts can then take place afterwards.
2. Sufficient detail can be defined differently in paper-based and web-based (and particularly hyperlinked) contexts.
   
3. Support by the data is clearly crucial, and while traditional manuscripts have usually published only part of the data, online environments allow, at least in principle, direct links to or even embedding from the sites wherethe data are hosted.
   
4. Intelligibility is traditionally defined with respect to the scope of a journal, but with blogs, it is the content of the individual posts that define the thus evolving scope, and intelligibility is likely to vary heavily from reader to reader and from blog post to blog post. Also, online environments tend to be less formal than traditional journals when it comes to the use of English.
5. These standards and the concepts behind them do evolve.
   
6. So do these guidelines and standards.
7. Instead of describing only the results of primary research, online environments allow to describe the whole process that leads there.

Last but not least, it is crucial for an assessment system that the original contribution as well as its timestamp, contributor and evaluator can be uniquely identified, and that evaluations aggregated (e.g. via a PageRank system) across contributions, contributors and evaluators, as discussed here, here and here (evaluation on this page is unidimensional but you may use it anyway).