peer review

How would you fund science?

Some days ago, science funding made an important step towards arriving in the web era: Fundscience.org a non-profit platform dedicated to opening up science funding to the eyes of the interested publicissued its first call for submissions of research proposals (deadline for submissions: April 1). Both the number of grants available in this first round (up to 3; yes, three) and their volume (up to US$ 50,000, or packages of 5,000 CPU cycles) are small in comparison to what established science funders have on offer but the call's conditions contain the seed of a new culture in science funding, one in which funding decisions are made less and less behind closed doors.

Fair play in academia - a test of the efficiency of non-public peer review

Yesterday was the deadline for submissions of "[p]roblems that, if solved, would advance the knowledge and capabilities in an area of your research". You can vote on them until Sunday, March 21 — two of the submitted problems will be turned into prizes (US$30k & 20k) for those coming up with a solution. My proposal (including the text contained in the supplementary file) is pasted in below:

Test the efficiency of public versus non-public peer review, both for research proposals and scholarly manuscripts. The results of this test would provide food for thought for the whole scientific community (which is based on non-public peer review) and the society at large (which ultimately pays the bills).

ESOF 2010 Proposal on Science 2.0

Next Euroscience Open Forum will take place on 2 to 7 of July, 2010 in Torino, Italy. I think it is the ideal forum to talk also about new science communciation tools.

That why I am proposing here a session on young researchers and Science 2.0. for the Career Programme of ESOF 2010.

We still need include names of interested experts to join the discussions with Second Life, Friendfeed or similar services in the proposal. The deadline for proposals for ESOF is September, 30.

The exact time within the frame of 2-7. July 2010 will be fixed later, taking into account the availabilities of the participants. It will be an one hour session. So is anybody interested to join the discussions?
w.

Wikipedia journal

This is a response to http://www.wittylama.com/2009/09/wikipedia-journal/ - a new (still hypothetical) initiative consisting of an Open Access journal (with ISSN and CC license) that publishes scholarly reviews that are peer-reviewed and ready to be pasted into Wikipedia.

I like this idea a lot. It is much like Scholarpedia (which contains commissioned but anonymously peer-reviewed articles and which has an ISSN but no coherent license), just has a broader scope and does not confine itself to the top-notch experts in the field. Your proposal, as mentioned above, also bears some resemblance to Citizendium (where the review process usually involves domain generalists rather than topic specialists, and it is non-anonymous; has CC license but no ISSN). Both operate stable versions that can be updated. The former allows for attribution, the latter not.

You also mentioned that a similar journal could be set up for original research (something that the Wikipedias, but also Scholarpedia and Citizendium have avoided so far), and in this regard, it is very close to the journal PLoS ONE (meant to be for all scientific disciplines, though currently with a bias towards the biomedical fields; has ISSN and CC license) and the recently launched PLoS Currents (which, in essence, uses Knol as a preprint server), which I have commented here.

I am also currently drafting a blog post for the Euroscientist on these matters at Wikiversity, to which everzone is welcome to contribute and where a number of related posts is referenced. To quote from just one of them: "science is already a wiki if you look at it a certain way. It's just a really, really inefficient one - the incremental edits are made in papers instead of wikispace, and significant effort is expended to recapitulate the existing knowledge in a paper in order to support the one-to-three new assertions made in any one paper."

Open-process academic publishing: Some more comments

This is a comment on Open-process academic publishing, which I have read until here.



Content
-Good framing of the discussion, though at places lacking in references
-On "discussions in comments", see here and here.
-If you do not comment in detail on the "different discursive universe", you might as well shorten or delete that phrase.
-Open-process publishing and reviewing advantages, (1)
--A good reference on the Atmospheric Chemistry and Physics model is here.
-Open-process publishing and reviewing advantages, (3)
--Plagiarism detection already works quite well now, some tools are listed here.
-Open-process publishing and reviewing advantages, (4)
--On speeding up the publication process, see here (my comment).
-Open-process publishing and reviewing advantages, (5)
--The readership and even reputation of open-process publishers may increase, but "journals" in the sense we know them may well cease to exist (in fact, already now there is but one journal — the scientific literature), since the open-process handling of submissions will naturally focus on the article level (as long as these exist) and later perhaps on individual submissions to the global knowledge system, and be this a single wiki edit (e.g. via tools like WikiTrust). On incremental publishing, see here and here and here.

-Internal benefits for journals, general
--given my reservations on the last point, it may be worth considering to exchange the term "journal" for something else in this section (I used "public research environment"), which will obviously affect other aspects of the phrasing
-Internal benefits for journals, (1)
--on the feedback loop between productivity and recognition, see here.
-Internal benefits for journals, (4)
--Karma system in use at Slashdot may be relevant for this section, see here.

-Modular process: stages and states
--These stages fit well with text-based disciplines, but there may be more components (overview here)


Typos and phrasing
-production work . Still,
-John Wilibanks
-what i think ought to done
-publish and perish devaluing model. Model
-argument even more focused that those in an average 8000 paper are
-on whose work the organization relies on
-in-dept texts
(yes, I would like to subscribe)
-or at to have




Reinventing academic publishing online

Such is the title of a two-part publication by Brian Whitworth and Rob Friedman. It provides for a fascinating assessment of the state-of-the-art of Information Science over the last ten years, much of it generalizable to other fields. In the following, I will list some quotes from the papers — I will try to comment on them later as time permits: Part I

What would science look like if it were invented today?

This is the first of two parts of a guest post for the Euroscientist, the blog of Euroscience.org. Part two is available here.

The Internet represents an opportunity to change this system, one which has created a 300-year-old, collective long-term memory, into something new and more efficient, perhaps adding in a current, collective short-term working memory at the same time. With new online tools, scientists could begin to share techniques, data and ideas online to the benefit of all parties, and the public at large.

Implementing Fantasy Science Funding

"Fantasy Science Funding" is an online game played by people with concrete ideas about science funding who are not currently in a position to put these ideas into practice. There are five rules to the game: 1 - choose a funding body whose funds you are managing in your fantasy, 2 - imagine how their funds could be distributed to the benefit of science, 3 - choose areas of science to be "fired" (i.e. whose funding should be decreased with respect to present state), 4 - choose areas of science to be "hired" (where funding should be increased with respect to now), 5 - blog about it.

Previous shindings that I am aware of were hosted by Duncan Hull, Björn Brembs and Cameron Neylon.

Research grant systems that encourage innovation

A recent paper entitled "Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant" suggests that there are opportunities to improve on currently prevailing research funding systems and gives concrete examples.

Abstract:

The journal scope in focus -- putting scholarly communication in context

Just imagine if all authors currently writing up manuscripts about a subject were instead to coordinate their efforts by collaborating on a single but detailed and balanced citable reference in which the topic would be described in and linked to all relevant contexts, updated as new research results pass peer review.

Since the advent of printed scholarly periodicals in the late Middle Ages, context in scientific communication has mainly been established by providing each of these publication venues (now collectively referred to as journals) with a scope, typically in terms of topics or methods covered, or with respect to a perceived threshold in newsworthiness.

Besides establishing context, the scope also defined the audience -- and thus indirectly the number of printed copies, their pricing and their distribution amongst individuals and institutions -- as well as criteria to be met by manuscripts in order to be considered for publication. Given the scope of a particular journal, consequently, knowledge about specialist terms (which may describe completely non-congruent concepts in different fields), methodologies, notations, mainstream opinions, trends or major controversies could reasonably be expected to be widespread amongst the audience, which reduced the need of redundantly repeating the same things all over and again. Interestingly, redundancy is still quite visible nonentheless, especially in the introductory, methods and discussion sections and the abstracts, often in a way characteristic of the authors (such that services like eTBLAST and JANE can make qualified guesses on authors of a particular piece of text, with good results if some of the authors have a lot of papers in the respective database, mainly PubMed, and if they have not changed their individual research scope too often in between).

Of course, there would be side effects: A manuscript well-adapted to the scope of one particular journal is often not very intelligible to someone outside its intended audience, which hampers cross-fertilization with other research fields (we will get back to this below). When using paper as the sole medium of communication, there is not much to be done about this limitation, and we got so used to it that few indeed would perceive it as a limitation at all. However, the times when paper alone reigned over scholarly communication have certainly passed.

So, in principle, the online version of a manuscript could link directly to any appropriate source of information (even blogs, for that matter, if no better source is available or accessible; see here for an example) but in current practice, linking is usually achieved paper-style, i.e. indirectly, via a list of references which itself is often not linked to online versions (let alone openly accessible ones) of the references in question, even though Uniform Resource Identifiers like DOI and SRef have been around for about a decade now, and International Standard Book Numbers longer still.

The above-mentioned hampered cross-field fertilization is crucial with respect to interdisciplinary research projects, digital libraries and multi-journal (or indeed cross-disciplinary) bibliographic search engines (e.g. Google Scholar), since these dramatically increased the likelihood of, say, a biologist stumbling upon a not primarily biological source relevant to her research (think shape quantification or growth curves, for instance). What options do we have to integrate these cross-disciplinary hidden treasures with the traditional intra-disciplinary background knowledge?

Interestingly, lack of context is also a consistent feature of most "Facebooks for scientists" (including ways.org which hosts this blog) - in fact, the whole set of scholarly pages on the www is the appropriate network for researchers but so far it is not optimally connected, particularly because formal scholarly communication has not yet fully hatched from the structures it had during the paper-based era (see also this nice overview of the current situation). Just imagine if all authors currently writing up manuscripts about a subject were instead to coordinate their efforts by collaborating on a single but detailed and balanced citable reference in which the topic would be described in and linked to all relevant contexts, updated as new research results pass peer review. Of course, this would shift the focus away from periodicals (and, in passing, render things like a journal's scope and Impact Factor superfluous), which is likely to meet resistance from the publishing establishment.

Groupware comes to mind in this regard, and wikis in particular: They allow to aggregate and inter-link diverse sets of knowledge in an online-accessible manner, basically for free. The by now classical example are the Wikipedias, and one scientific journal - RNA biology - has already announced that it requires an introductory Wikipedia article for papers it is to publish on RNA families, an idea that recently spurred an ongoing debate on the merits of such an initiative and of doing it with Wikipedia.

An investigation (video lecture by Bill Wedemeyer here, my brief annotation here) of the quality of a set of science articles in the English Wikipedia is currently being written up for classical paper-style publication but the preliminary results indicate that "[t]here is a subset of reliably helpful science articles on the English Wikipedia for outreach, teacher training, and general science education" (slide shown at 29:35min in the video). However, the distribution of the set of articles was skewed towards the Good Article and Featured Article classes which constituted only 2% of the English Wikipedia at the time of investigation, and it did not include articles in the humanities (they come next).

Furthermore, the larger Wikipedias have a serious problem with vandalism: take an article of your choice and look in its history page for reverts - most of them will be about changes like this or worse. This is less of an issue with more popular topics for which large numbers of volunteers may be available to correct spammy entries but it is probably fair to assume that most researchers value their time too much to spend it on repeatedly correcting such information if it had already been correctly entered once. Other problems with covering scientific topics at Wikipedia include the notability criteria which have to be fulfilled to avoid an article being deleted, and the rejection of "original research" in the sense of not having been peer reviewed before publication. Peer review is indeed an important aspect of scholarly communication, as it paves the way towards the reproducibility that forms one of the foundations of modern science. Yet I know of no compelling reason to believe that it works better before than after publication (doing it beforehand was just a practical decision in times when journal space was measured in paper pages).

Fortunately, the Wikipedias are not the only wikis around, and amongst the more scholarly inclined alternatives, there are even a number of wiki-based journals, though usually with a very narrow scope and/ or a low number of articles. On the contrary, Citizendium, Scholarpedia (which has classical peer review and an ISSN and may thus be counted as a wiki journal, too), OpenWetWare and the Wikiversities are cross-disciplinary and structured (as well as sized, for the moment) such that vandalism and notability are not really a problem (with minor exceptions, real names are required at the first three, and anybody can write about anything, particularly their fields of expertise). None of these is even close to providing the vast amount of context existing in the English Wikipedia but they might perhaps if the latter were broken down to scholarly useful stuff, as discussed above. Out of these four wikis, only OpenWetWare and some Wikiversities (here counted as one) currently allow for original research to be published on their site - in the case of OpenWetWare, this is indeed the main purpose.

Further, a number of more specialized scholarly wikis exist (e.g. WikiGenes, the Encyclopedia of Earth, the Encyclopedia of the Cosmos, or the Dispersive PDE Wiki) which can teach us about the usefulness of wikis within specific academic fields. I will not dwell on details here but instead list a number of features I deem desirable for future scholarly wikis, derived from experience with existing ones. These include, in no particular order:

  • search engines that integrate or otherwise compare favourably with major scholarly search engines on the web (the already mentioned Google Scholar and PubMed as well as, say, the BioText Search Engine that searches Open Access text and images)
  • pan-disciplinary scope, with consistent disambiguation of specialist terms (mainly but not fully achieved at Citizendium)
  • some system of peer review (basically, any wiki allows to leave comments, annotations or formal reviews on talk pages of users or articles but these ratings should be featured more prominently; templates like those visualizing article status at Citizendium may help with that); this may be as simple as disallowing individuals to add information to Citizendium when the only available support is their own non-reviewed research published at OpenWetWare - the real name policy will minimize misuse
  • the uploadability of all kinds of media (including videos, which are blocked at the Wikipedias but allowed at Citizendium, for instance, and the scope of the Journal of Visualized Experiments) that traditionally (if you can call a habit that barely is a decade old a tradition already) went along with paper-based publications as "supporting online information" (which would be easily integrated in an all-online article with no sharp space limitations)
  • stable versions for contents that has undergone peer review (like the Approved Articles at Citizendium), along with draft versions for anything else (including improvements to and updates of previous stable versions); like any non-protected page at the Wikipedias, these draft versions can serve as a playground, though a real-name policy would probably make it a more educational one
  • a separate namespace for references (already in use at the Dispersive PDE Wiki and the French Wikipedia, in test at Citizendium); as a side line, this would open up ways for new citation metrics, via the What links here function
  • attributability of contributions (automatically realized, though not in the traditional scholarly way, in any wiki with a real name policy like that at Citizendium, via the User contributions function; special arrangements exist at Scholarpedia and WikiGenes; OpenWetWare does allow nicknames but real names prevail; the Wikiversities have basically the same user name policy as the Wikipedias)
  • easy download of selected sets of pages for local archiving by individual researchers
  • licenses that allow unrestricted reuse and derivative work if the original source is properly acknowledged (typically CC-by-SA or the older GFDL, both of which are hopefully going to be compatible soon)
  • resource-effective design (see also discussions on the energy use of the internet and individual websites)
  • integration with the non-scholarly world (certainly achieved in the Wikipedias and Citizendium), particularly with students (cf. the Eduzendium initiative at Citizendium) and non-English contents
  • automation of the formatting, as already common in non-wiki environments, e.g. with LaTeX templates (none of the wikis I know comes close to that, albeit templates are heavily used at the various Wikipedias and, to a lesser extent but in a more consistent manner, at Citizendium; they seem to be rather rarely used on smaller or more specialized wikis); the same applies to references, though automated wikificationhas already progressed considerably here, despite the lack of wiki export functions at publisher's sites (or of suitable XML-to-wiki converters for those who provide XML)

One of the most useful templates in use at Citizendium is that for subpages (open the Biology article in a separate window to see what this is about) :  

  • The article's main page is a stable version, approved by an author with expertise in that field
  • Next comes the Talk tab that leads to the discussion page, as per default in any wiki
  • the Draft tab leads to the editable version (this only applies for articles that have already been approved; in others, the main page is editable)
  • the Related Articles tab roughly corresponds to "see also" in the Wikipedias but is more usefully structured for navigation and somewhat replaces the categories which are heavily used in Wikipedia but only to a limited extent at Citizendium
  • there are further subpages: Bibliography for further reading, External Links, Gallery, Video and so on

It is interesting to see that these individual subpages largely complement existing social networking tools and have thus the potential to replace them (or to be replaced by them), at least for scholarly purposes:

  • the Bibliography subpage is a context-based alternative to CiteULike, Zotero, BibSonomy and other reference managers, possibly in conjunction with Open Library, scholarly search engines and tools like Scribd or Papers. One problem wikis cannot solve is that of access to paper-based research publications but due to the current spread of Green and Gold Open Access initiatives, this is likely to change in the next few years anyway.
  • the External Links subpage is a context-based alternative to conventional social bookmarking as known from delicious and simpy 
  • Additional subpages could be tailored to meet the needs of individual categories of articles (e.g. properties of chemical elements, genes, stellar constellations etc.) or more general scholarly needs (e.g. peer review, slides, code, protocols, or bot-generated transcripts from video lectures)

Besides, User pages may provide context-based alternatives to individual pages at different networking sites, and possibly even to blogs like this one, while the Recent changes page could turn into an alternative for friendfeed, with items on your Watchlist (if you are logged in) equivalent to friendfeed rooms or personal feeds you are subscribed to. For the record, this social networking component of Citizendium has already been discussed two years ago, prior to its official launch and thus at a time when many of its current structures and their implications were not known yet.

Finally, and importantly, the easy availability of context (once the system would be reasonably well adopted by scholarly communities, and the encyclopedic corpus thus reasonably complete) would make it more easy to guide expert attention and thus to identify obvious gaps in current knowledge (e.g. by means of an expert evaluation of items listed on the Most Wanted page), and science funders could then issue a call for research proposals on such topics (e.g. via a Calls subpage, InnoCentive, Mechanical Turk or by more traditional means). And while we are at it, I think science funders, job committees and review panels would profit from familiarizing themselves with the workings of wikis, particularly the aspects relevant to reliability, attribution, and outreach (your organization, company or university probably has a page on Wikipedia - take a look at it, along with its history and talk pages, and you will almost certainly find something to improve).

To sum up, the still fledgling Citizendium currently seems to be the closest match for a cross-disciplinary scholarly wiki anchored in the real world, and independent of whether it will allow original research to be posted in the future or not, this essential function in scholarly communication can be fulfilled by OpenWetWare (indeed, a similar separation of powers is one of the most healthy elements of most democracies). If widely adopted, this would entail a major shift in the way research is being done and communicated, towards what has come to be known as open science. As a side effect, commercial publishers would have to look for new things to publish, other than original research (non-commercial publishers like scholarly societies may, after the usual period of resistance, see more advantages than disadvantages in the groupware model). Reviews at different levels of expertise may be one option, and tutorials or other learning tools another but all this could be done via some intelligently structured set of groupware, too, depending on the incentives involved (in fact, such reviews are the scope of Scholarpedia). A side effect for researchers would be that they could use the author fees, page and figure charges and all the other money currently required to publish a paper for other purposes.

Of course, there are potential problems with such an enormous concentration of knowledge (e.g. for attacks and misuse, especially in relation to an international author identification that is currently being discussed). The obvious solutions are appropriate mirroring and otherwise transparency. Similar concerns would apply to a journal like PLoS ONE that does not have a scope in the traditional paper-limited sense mentioned above, yet two years after launch, it is doing pretty well, and my guess is that if it were to adopt a symbiosis with a suitable wiki in a way similar to the RNA Biology initiative, it may even do better.

As a next step, I wish to go into more detail concerning the relative merits of paper-based and wiki-based scholarly communication. So I started a Wikiversity page on wikis in scholarly communication and invite you to add to it (I chose Wikiversity such that those who object to real name policies may make their voice heard, too, and I think I can deal with spam should it arise there). This overview may also help in working out an ecological footprint scheme applicable to research, as described previously.

I dedicate this post to my granny who passed away last week.


Funding revolutionary scientific research

A recent paper suggest that revolutionary research can be stimulated to a larger extent than currently by large-scale prizes for projects that have initiated such revolutions. This is especially relevant for young scientists, as they have to establish themselves in an appropriate niche within the scientific community, and one of the most promising (yet usually very risky, and thus not popular) strategies to create such niches is to remodel significant parts of the scientific landscape by questioning (with the support of relevant empirical or theoretical arguments). This also generally creates opportunities for a whole generation of scientists, and so the scientific community should have an interest in rewarding its revolutionaries.

The paper appeared as an editorial (Bruce G. Charlton and Peter Andras: Stimulating revolutionary science with mega-cash prizes) in Medical Hypotheses 70:709-713 (2008) and is available online via

Young Scientists Journal

An online journal entirely run by Young Scientists for Young Scientists has already published three issues, with more to come. For more information, or if you want to join in, take a look at
http://www.youngscientists.co.uk/ .