World Association of Young Scientists

This is a response to Richard Grant's On article-level metrics and other animals where the html did not come through.

Yes, Richard, such an experiment is definitely overdue! I have actually sketched out a grant proposal along these lines, for a two-step funding scheme on Europe and Global Challenges (Eur 50,000 in the first phase, 1,000,000 in the second, for which independent submissions are permitted), to be submitted in collaboration with Richard Gordon and Bryan Poulin (the authors of the above-mentioned study) as well as Brian Czech. Since useful stats can only be gathered if there is a high number of projects, it was foreseen to have projects of Eur 1,000 each - a scale inappropriate for multiple research projects but probably suitable for multiple small projects that allow graduate students some a little bit of independent research, e.g. on a sideline of their PhD thesis. Due to the low overall budget, basically no money would be available for project administration, so the allocation procedure would have to be simple, and we chose a competition, which would probably fall somewhere next to C in your scheme - going for random allocation did not stand any chance to pass the peer review process for the call. Given that the funding agencies already have the data on group A (any other Eur 50,000 they have spent previously, perhaps even on peer review, as detailed in the paper the above-mentioned above), the aim of the project would be to test whether these multiple small-scale projects would result in higher impact per budget than the control group. The project should be conducted as an open science project, such that everybody can observe the results as they come in and see for themselves whether this model works at this scale. Knowing that all this is unusual, we contacted the funding agencies that had issued the call, and invited them to give some initial feedback. This turned out to be very discouraging (for three main reasons: i - the focus of the call is on global governance, for which scientists like Gordon and me and economists like Poulin and Czech were considered a non-ideal match, ii - the allocation of research funds was not seen as a global challenge, iii - some of their internal policies not mentioned in the call), and so we finally decided not to submit at this first stage. However, reason iii would not apply for the second stage, and the larger budget then (plus the additional time) would allow to build a team that fits better to their grid, and to make a stronger case for the proper allocation of research funds as a current challenge at both the European and the global level. So we definitely plan to submit a proposal for the second stage and possibly elsewhere as opportunities arise, and we would welcome anyone interested in these matters to work out such proposals together, preferably in the open, like in our wiki linked from above.

"Fantasy Science Funding" is an online game played by people with concrete ideas about science funding who are not currently in a position to put these ideas into practice. There are five rules to the game: 1 - choose a funding body whose funds you are managing in your fantasy, 2 - imagine how their funds could be distributed to the benefit of science, 3 - choose areas of science to be "fired" (i.e. whose funding should be decreased with respect to present state), 4 - choose areas of science to be "hired" (where funding should be increased with respect to now), 5 - blog about it.

Previous shindings that I am aware of were hosted by Duncan Hull, Björn Brembs and Cameron Neylon.

A recent paper entitled "Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant" suggests that there are opportunities to improve on currently prevailing research funding systems and gives concrete examples.

Abstract:

Just imagine if all authors currently writing up manuscripts about a subject were instead to coordinate their efforts by collaborating on a single but detailed and balanced citable reference in which the topic would be described in and linked to all relevant contexts, updated as new research results pass peer review.

Since the advent of printed scholarly periodicals in the late Middle Ages, context in scientific communication has mainly been established by providing each of these publication venues (now collectively referred to as journals) with a scope, typically in terms of topics or methods covered, or with respect to a perceived threshold in newsworthiness.

Besides establishing context, the scope also defined the audience -- and thus indirectly the number of printed copies, their pricing and their distribution amongst individuals and institutions -- as well as criteria to be met by manuscripts in order to be considered for publication. Given the scope of a particular journal, consequently, knowledge about specialist terms (which may describe completely non-congruent concepts in different fields), methodologies, notations, mainstream opinions, trends or major controversies could reasonably be expected to be widespread amongst the audience, which reduced the need of redundantly repeating the same things all over and again. Interestingly, redundancy is still quite visible nonentheless, especially in the introductory, methods and discussion sections and the abstracts, often in a way characteristic of the authors (such that services like eTBLAST and JANE can make qualified guesses on authors of a particular piece of text, with good results if some of the authors have a lot of papers in the respective database, mainly PubMed, and if they have not changed their individual research scope too often in between).

Of course, there would be side effects: A manuscript well-adapted to the scope of one particular journal is often not very intelligible to someone outside its intended audience, which hampers cross-fertilization with other research fields (we will get back to this below). When using paper as the sole medium of communication, there is not much to be done about this limitation, and we got so used to it that few indeed would perceive it as a limitation at all. However, the times when paper alone reigned over scholarly communication have certainly passed.

So, in principle, the online version of a manuscript could link directly to any appropriate source of information (even blogs, for that matter, if no better source is available or accessible; see here for an example) but in current practice, linking is usually achieved paper-style, i.e. indirectly, via a list of references which itself is often not linked to online versions (let alone openly accessible ones) of the references in question, even though Uniform Resource Identifiers like DOI and SRef have been around for about a decade now, and International Standard Book Numbers longer still.

The above-mentioned hampered cross-field fertilization is crucial with respect to interdisciplinary research projects, digital libraries and multi-journal (or indeed cross-disciplinary) bibliographic search engines (e.g. Google Scholar), since these dramatically increased the likelihood of, say, a biologist stumbling upon a not primarily biological source relevant to her research (think shape quantification or growth curves, for instance). What options do we have to integrate these cross-disciplinary hidden treasures with the traditional intra-disciplinary background knowledge?

Interestingly, lack of context is also a consistent feature of most "Facebooks for scientists" (including ways.org which hosts this blog) - in fact, the whole set of scholarly pages on the www is the appropriate network for researchers but so far it is not optimally connected, particularly because formal scholarly communication has not yet fully hatched from the structures it had during the paper-based era (see also this nice overview of the current situation). Just imagine if all authors currently writing up manuscripts about a subject were instead to coordinate their efforts by collaborating on a single but detailed and balanced citable reference in which the topic would be described in and linked to all relevant contexts, updated as new research results pass peer review. Of course, this would shift the focus away from periodicals (and, in passing, render things like a journal's scope and Impact Factor superfluous), which is likely to meet resistance from the publishing establishment.

Groupware comes to mind in this regard, and wikis in particular: They allow to aggregate and inter-link diverse sets of knowledge in an online-accessible manner, basically for free. The by now classical example are the Wikipedias, and one scientific journal - RNA biology - has already announced that it requires an introductory Wikipedia article for papers it is to publish on RNA families, an idea that recently spurred an ongoing debate on the merits of such an initiative and of doing it with Wikipedia.

An investigation (video lecture by Bill Wedemeyer here, my brief annotation here) of the quality of a set of science articles in the English Wikipedia is currently being written up for classical paper-style publication but the preliminary results indicate that "[t]here is a subset of reliably helpful science articles on the English Wikipedia for outreach, teacher training, and general science education" (slide shown at 29:35min in the video). However, the distribution of the set of articles was skewed towards the Good Article and Featured Article classes which constituted only 2% of the English Wikipedia at the time of investigation, and it did not include articles in the humanities (they come next).

Furthermore, the larger Wikipedias have a serious problem with vandalism: take an article of your choice and look in its history page for reverts - most of them will be about changes like this or worse. This is less of an issue with more popular topics for which large numbers of volunteers may be available to correct spammy entries but it is probably fair to assume that most researchers value their time too much to spend it on repeatedly correcting such information if it had already been correctly entered once. Other problems with covering scientific topics at Wikipedia include the notability criteria which have to be fulfilled to avoid an article being deleted, and the rejection of "original research" in the sense of not having been peer reviewed before publication. Peer review is indeed an important aspect of scholarly communication, as it paves the way towards the reproducibility that forms one of the foundations of modern science. Yet I know of no compelling reason to believe that it works better before than after publication (doing it beforehand was just a practical decision in times when journal space was measured in paper pages).

Fortunately, the Wikipedias are not the only wikis around, and amongst the more scholarly inclined alternatives, there are even a number of wiki-based journals, though usually with a very narrow scope and/ or a low number of articles. On the contrary, Citizendium, Scholarpedia (which has classical peer review and an ISSN and may thus be counted as a wiki journal, too), OpenWetWare and the Wikiversities are cross-disciplinary and structured (as well as sized, for the moment) such that vandalism and notability are not really a problem (with minor exceptions, real names are required at the first three, and anybody can write about anything, particularly their fields of expertise). None of these is even close to providing the vast amount of context existing in the English Wikipedia but they might perhaps if the latter were broken down to scholarly useful stuff, as discussed above. Out of these four wikis, only OpenWetWare and some Wikiversities (here counted as one) currently allow for original research to be published on their site - in the case of OpenWetWare, this is indeed the main purpose.

Further, a number of more specialized scholarly wikis exist (e.g. WikiGenes, the Encyclopedia of Earth, the Encyclopedia of the Cosmos, or the Dispersive PDE Wiki) which can teach us about the usefulness of wikis within specific academic fields. I will not dwell on details here but instead list a number of features I deem desirable for future scholarly wikis, derived from experience with existing ones. These include, in no particular order:

• search engines that integrate or otherwise compare favourably with major scholarly search engines on the web (the already mentioned Google Scholar and PubMed as well as, say, the BioText Search Engine that searches Open Access text and images)
  
• pan-disciplinary scope, with consistent disambiguation of specialist terms (mainly but not fully achieved at Citizendium)
• some system of peer review (basically, any wiki allows to leave comments, annotations or formal reviews on talk pages of users or articles but these ratings should be featured more prominently; templates like those visualizing article status at Citizendium may help with that); this may be as simple as disallowing individuals to add information to Citizendium when the only available support is their own non-reviewed research published at OpenWetWare - the real name policy will minimize misuse
  
• the uploadability of all kinds of media (including videos, which are blocked at the Wikipedias but allowed at Citizendium, for instance, and the scope of the Journal of Visualized Experiments) that traditionally (if you can call a habit that barely is a decade old a tradition already) went along with paper-based publications as "supporting online information" (which would be easily integrated in an all-online article with no sharp space limitations)
  
• stable versions for contents that has undergone peer review (like the Approved Articles at Citizendium), along with draft versions for anything else (including improvements to and updates of previous stable versions); like any non-protected page at the Wikipedias, these draft versions can serve as a playground, though a real-name policy would probably make it a more educational one
  
• a separate namespace for references (already in use at the Dispersive PDE Wiki and the French Wikipedia, in test at Citizendium); as a side line, this would open up ways for new citation metrics, via the What links here function
  
• attributability of contributions (automatically realized, though not in the traditional scholarly way, in any wiki with a real name policy like that at Citizendium, via the User contributions function; special arrangements exist at Scholarpedia and WikiGenes; OpenWetWare does allow nicknames but real names prevail; the Wikiversities have basically the same user name policy as the Wikipedias)
• easy download of selected sets of pages for local archiving by individual researchers
  
• licenses that allow unrestricted reuse and derivative work if the original source is properly acknowledged (typically CC-by-SA or the older GFDL, both of which are hopefully going to be compatible soon)
• resource-effective design (see also discussions on the energy use of the internet and individual websites)
• integration with the non-scholarly world (certainly achieved in the Wikipedias and Citizendium), particularly with students (cf. the Eduzendium initiative at Citizendium) and non-English contents
  
• automation of the formatting, as already common in non-wiki environments, e.g. with LaTeX templates (none of the wikis I know comes close to that, albeit templates are heavily used at the various Wikipedias and, to a lesser extent but in a more consistent manner, at Citizendium; they seem to be rather rarely used on smaller or more specialized wikis); the same applies to references, though automated wikificationhas already progressed considerably here, despite the lack of wiki export functions at publisher's sites (or of suitable XML-to-wiki converters for those who provide XML)
  

One of the most useful templates in use at Citizendium is that for subpages (open the Biology article in a separate window to see what this is about) :  

• The article's main page is a stable version, approved by an author with expertise in that field
• Next comes the Talk tab that leads to the discussion page, as per default in any wiki
  
• the Draft tab leads to the editable version (this only applies for articles that have already been approved; in others, the main page is editable)
• the Related Articles tab roughly corresponds to "see also" in the Wikipedias but is more usefully structured for navigation and somewhat replaces the categories which are heavily used in Wikipedia but only to a limited extent at Citizendium
• there are further subpages: Bibliography for further reading, External Links, Gallery, Video and so on

It is interesting to see that these individual subpages largely complement existing social networking tools and have thus the potential to replace them (or to be replaced by them), at least for scholarly purposes:

• the Bibliography subpage is a context-based alternative to CiteULike, Zotero, BibSonomy and other reference managers, possibly in conjunction with Open Library, scholarly search engines and tools like Scribd or Papers. One problem wikis cannot solve is that of access to paper-based research publications but due to the current spread of Green and Gold Open Access initiatives, this is likely to change in the next few years anyway.
  
• the External Links subpage is a context-based alternative to conventional social bookmarking as known from delicious and simpy 
• Additional subpages could be tailored to meet the needs of individual categories of articles (e.g. properties of chemical elements, genes, stellar constellations etc.) or more general scholarly needs (e.g. peer review, slides, code, protocols, or bot-generated transcripts from video lectures)

Besides, User pages may provide context-based alternatives to individual pages at different networking sites, and possibly even to blogs like this one, while the Recent changes page could turn into an alternative for friendfeed, with items on your Watchlist (if you are logged in) equivalent to friendfeed rooms or personal feeds you are subscribed to. For the record, this social networking component of Citizendium has already been discussed two years ago, prior to its official launch and thus at a time when many of its current structures and their implications were not known yet.

Finally, and importantly, the easy availability of context (once the system would be reasonably well adopted by scholarly communities, and the encyclopedic corpus thus reasonably complete) would make it more easy to guide expert attention and thus to identify obvious gaps in current knowledge (e.g. by means of an expert evaluation of items listed on the Most Wanted page), and science funders could then issue a call for research proposals on such topics (e.g. via a Calls subpage, InnoCentive, Mechanical Turk or by more traditional means). And while we are at it, I think science funders, job committees and review panels would profit from familiarizing themselves with the workings of wikis, particularly the aspects relevant to reliability, attribution, and outreach (your organization, company or university probably has a page on Wikipedia - take a look at it, along with its history and talk pages, and you will almost certainly find something to improve).

To sum up, the still fledgling Citizendium currently seems to be the closest match for a cross-disciplinary scholarly wiki anchored in the real world, and independent of whether it will allow original research to be posted in the future or not, this essential function in scholarly communication can be fulfilled by OpenWetWare (indeed, a similar separation of powers is one of the most healthy elements of most democracies). If widely adopted, this would entail a major shift in the way research is being done and communicated, towards what has come to be known as open science. As a side effect, commercial publishers would have to look for new things to publish, other than original research (non-commercial publishers like scholarly societies may, after the usual period of resistance, see more advantages than disadvantages in the groupware model). Reviews at different levels of expertise may be one option, and tutorials or other learning tools another but all this could be done via some intelligently structured set of groupware, too, depending on the incentives involved (in fact, such reviews are the scope of Scholarpedia). A side effect for researchers would be that they could use the author fees, page and figure charges and all the other money currently required to publish a paper for other purposes.

Of course, there are potential problems with such an enormous concentration of knowledge (e.g. for attacks and misuse, especially in relation to an international author identification that is currently being discussed). The obvious solutions are appropriate mirroring and otherwise transparency. Similar concerns would apply to a journal like PLoS ONE that does not have a scope in the traditional paper-limited sense mentioned above, yet two years after launch, it is doing pretty well, and my guess is that if it were to adopt a symbiosis with a suitable wiki in a way similar to the RNA Biology initiative, it may even do better.

As a next step, I wish to go into more detail concerning the relative merits of paper-based and wiki-based scholarly communication. So I started a Wikiversity page on wikis in scholarly communication and invite you to add to it (I chose Wikiversity such that those who object to real name policies may make their voice heard, too, and I think I can deal with spam should it arise there). This overview may also help in working out an ecological footprint scheme applicable to research, as described previously.

I dedicate this post to my granny who passed away last week.