Criteria suitable for impact metrics - moving beyond papers
Here is some brainstorming on criteria suitable for the evaluation of
scientific contributions (initially perhaps just journal articles, but
in the long run also blog posts, wiki edits, project proposals,
database entries and basically anything related to science that can be
described by a Uniform Resource Identifier).
For a start, I have listed below the criteria currently in use at PLoS ONE for pre-publication assessment of research manuscripts:
- Results reported have not been published elsewhere.
- Experiments, statistics, and other analyses are performed to a high technical standard and are described in sufficient detail.
- Conclusions are presented in an appropriate fashion and are supported by the data.
- The article is presented in an intelligible fashion and is written in standard English.
- The research meets all applicable standards for the ethics of experimentation and research integrity.
- The article adheres to appropriate reporting guidelines (e.g. CONSORT, MIAME, STROBE, EQUATOR) and community standards for data availability.
- The study presents the results of primary scientific research.
These points raise a number of issues with respect to adaptations to online environments:
- "Published" in this sense refers to "making the information publicly available after peer review", while the term has other connotations in the context of blogs or wikis where it simply means "making the information publicly available", and review by experts and non-experts can then take place afterwards.
- Sufficient detail can be defined differently in paper-based and web-based (and particularly hyperlinked) contexts.
- Support by the data is clearly crucial, and while traditional manuscripts have usually published only part of the data, online environments allow, at least in principle, direct links to or even embedding from the sites wherethe data are hosted.
- Intelligibility is traditionally defined with respect to the scope of a journal, but with blogs, it is the content of the individual posts that define the thus evolving scope, and intelligibility is likely to vary heavily from reader to reader and from blog post to blog post. Also, online environments tend to be less formal than traditional journals when it comes to the use of English.
- These standards and the concepts behind them do evolve.
- So do these guidelines and standards.
- Instead of describing only the results of primary research, online environments allow to describe the whole process that leads there.
Last but not least, it is crucial for an assessment system that the original contribution as well as its timestamp, contributor and evaluator can be uniquely identified, and that evaluations aggregated (e.g. via a PageRank system) across contributions, contributors and evaluators, as discussed here, here and here (evaluation on this page is unidimensional but you may use it anyway).