Research intelligence - It's peerless, but it could be better still

Credit: GettyFiring on all cylinders: The subcommittee would like to see publishers training editors, and funders ensuring researchers receive instruction in the fine points of peer review

Peer review may be the worst method of assessing scientific excellence, apart from all the other methods, but a committee of MPs sees plenty of scope to make it less bad.

That common spin on Winston Churchill’s verdict on democracy was trotted out by successive witnesses during the Commons Science and Technology Committee’s recent inquiry into peer review in scientific publications.

But the committee’s report, published last week, notes that there is “little solid evidence” of the efficacy of peer review. And while the MPs accept that it is indispensable, they call on publishers, research funders and research users such as industry and the government to work together to gather that evidence so that peer review “can be optimised and innovations introduced, and the impact of the common criticisms minimised”.

Impact explored

One issue that particularly interested the committee was the consideration of potential academic impact in the editorial decision-making that determines which papers are published in journals. The report concludes that such an assessment “requires subjective judgment” and is related in part to how topical the research is deemed to be. As a consequence, there is an “element of chance” at play in the determination of what gets published in a top-tier journal, so universities should be “cautious” about promoting people on the basis of the impact factor of the journals in which their work has been published.

“While we have been assured by research funders that they do not use (impact factors) as a proxy measure for the quality of research … representatives of research institutions have suggested that publication in a high-impact journal is still an important consideration when assessing individuals for career progression,” the report says.

The committee welcomes “new approaches that focus on carrying out a technical assessment prior to publication and making an assessment of impact after publication”.

Although it notes the potential of “repository-style” journals such as PLoS ONE that publish all submissions that meet a minimal technical quality standard, it emphasises that it is “important that a high quality of peer review is maintained”.

The committee encourages the “prudent” use of online tools for post-publication review. It is particularly enthusiastic about the “enormous opportunity” presented by social-networking sites to share links to papers that are interesting or potentially problematic.

The MPs also find the “transparency” of open peer review “attractive”, but they accept that its suitability may vary across fields.

The desire for transparency is apparent in the committee’s suggestion that pre-publication peer reviewers should have to ensure that papers contain enough information to allow other researchers to reproduce results. What’s more, funders and publishers should develop ways to reward efforts to make datasets reusable and should ensure that they are made available “in a timely manner”, the MPs say.

“The presumption must be that, unless there is a strong reason otherwise, data should be fully disclosed and made publicly available,” the report says.

This verdict prompted one witness at the hearings, Tracey Brown, managing director of the campaign group Sense About Science, to subsequently caution that data divorced from the “meaning and context” provided by papers was the most common source of public misunderstanding of science.

The committee is not convinced that all those involved in the peer-review process are performing their roles to the highest possible standards. To this end, the report urges publishers to develop common training standards that ensure that editors are “fully equipped for the central role that they play”. Funders should also see that all early-career researchers are trained in peer review, and publishers should help reviewers improve the quality of their efforts by circulating examples of good reviews.

The committee doubts that the relentless rise in research output has led to a “crisis” that threatens to overwhelm established reviewers, but it urges publishers to forge ahead with efforts to train new peer reviewers from rising research powers such as China. It also calls for greater recognition of peer reviewers by both publishers and institutional employers.

Integrity is a big concern

The MPs reserve their sharpest criticism for the oversight mechanisms for ensuring research integrity in the UK, which they describe as “confused” and “highly unsatisfactory”. Although they accept that “it is not the role of peer review to police research integrity and identify fraud or misconduct”, they point out that misconduct “damages peer review and science as a whole”.

Citing the disturbing findings of a 2002 poll by the US National Institutes of Health in which about a third of researchers surveyed admitted to having engaged in misconduct in the previous three years, the MPs express surprise that no research council has ever withdrawn funding from a researcher on the grounds of fraud.

“We recommend that…funders of research reassess the robustness of their procedures for dealing with allegations of research fraud or misconduct, to ensure that they are not falling through the cracks,” the report says.

The committee calls for the establishment of an independent regulator to oversee investigations into research misconduct. It accepts the assertion by Sir Mark Walport, director of the Wellcome Trust, that employers bear primary responsibility for ensuring research integrity. But the MPs believe that an external body is needed to “oversee the employer and make sure that they are doing the right thing”.

The report cites the view of Rick Rylance, chair of Research Councils UK, that there is “no appetite” in the research community for a regulatory body. But it does no more than note funders’ alternative proposal: a “concordat” outlining ethical principles that research organisations will be expected to endorse.

Nominate a leader

The committee calls on research institutions to appoint a specific individual to lead on research integrity and declares it “essential” that the outcome of any investigation into misconduct be published.

Funders are urged to “revisit” their decision not to implement the recommendation of the UK Research Integrity Futures Working Group, convened last year by RCUK and Universities UK, to establish an independent body that could advise research organisations and individuals on how to handle misconduct allegations.

The report notes that the UK Research Integrity Office wrote to the committee to contradict assertions from Professor Rylance that it dealt only with research integrity in the biosciences. In practice, the committee was told, UKRIO had offered advice in all subjects since its foundation in 2006. However, the MPs do not explicitly recommend that funders renew their funding for UKRIO, which ended last December.

James Parry, UKRIO’s acting head, told Times Higher Education that the body would continue to operate as an independent charity under a new funding model whereby “key users” such as universities, NHS organisations, charities and companies would be asked to pay a “modest” annual subscription.

He said a number of major research universities had already pledged support. “It is pleasing that they have all said they recognise that it will benefit their institution but also that supporting UKRIO is worthwhile in itself,” he said.

paul.jump@tsleducation.com

• http://tinyurl.com/3vxjd7s