‘There is a problem’: Australia’s top scientist Alan Finkel pushes to eradicate bad science

Editor’s note: A number of RealKM articles have looked at issues in regard to the quality of science, as well how incentives can negatively influence behaviour. Australia’s Chief Scientist Alan Finkel brings these two issues together in a critique of the role of incentives in the research system..

Alan Finkel, Office of the Chief Scientist

In the main, Australia produces high-quality research that is rigorous and reproducible, and makes a significant contribution towards scientific progress. But that doesn’t mean we can’t do it better.

In the case of the research sector here and abroad, we need to acknowledge that as good as the research system is, there is a problem.

There are a significant number of papers that are of poor quality, and should never have made it through to publication. In considering why this might be the case, I have found myself reflecting on the role of incentives in the research system.

Because incentives matter, as we have seen through the findings of the Royal Commission into the banking sector led by Kenneth Hayne.

The commission shone a light on how the sector incentivises its employees. And there are some incentives in the research community that, in my view, need to be looked at.

We may be inadvertently encouraging poor behaviour. And to ensure research remains high-quality and trustworthy, we need to get the incentives right.

Lessons from the banking Royal Commission

The commission showed that over the past decade or two, the banking sector moved from salary-based to bonus-based remuneration. But those bonuses have been mapping to the wrong values: to sales and profit instead of compliance with the law and net benefit to customers.

Similarly in the research sector, we can’t ignore that there are many incentives pushing some to cut corners and lower their standards.

The competition for funding is fierce and is increasing every day. The temptation to judge a researcher’s performance through simple metrics, such as the number of published research papers, is strong. These metrics are incentives that drive behaviour – not all of it good.

We all know of instances of poor research practice. Selective publication of results to support a hypothesis. HARKing: hypothesising after results are known. Manipulating data and research methods to achieve statistical significance.

If we can focus on improving the quality of research in general, we can achieve broad and long-lasting benefits. And I think the best way to do this is to look at the incentives.

Quality should trump quantity

Publication is a principal criterion for career advancement in the research sector. And I don’t want to change that. However, the institutionalisation of performance metrics has created incentives for researchers to publish as many papers as possible.

There shouldn’t be an incentive for a researcher to salami-slice their results into three or four separate publications, rather than one meaningful publication. If the purpose of publication is to share your results in a way that can be built on by other researchers, this kind of practice completely defeats that purpose.

One model that places the focus on quality over quantity is the “Rule of Five”. With this rule, a researcher’s performance for grant funding or promotion is judged on their best five publications over a five year period, accompanied by a description of its impact and the researcher’s individual contribution.

The exact number of publications or years isn’t important, as long as it is less than ten.

Of course, there are disciplinary differences that may need to be taken into account. But what matters is the emphasis on the significance of the research.

Researchers must undergo integrity training

Unlike other professions, there are no national competencies and no national recognition of education and training in research integrity. While many institutions in Australia do provide training programs for their PhD students, these programs vary in quality, content and reach.

And, to the best of my knowledge, no Australian institutions have a training requirement for their existing research workforce.

I strongly believe the overall quality of research in Australia would be strengthened by research integrity training for all researchers.

Training puts a spotlight on expectations for the whole community and encourages consistent behaviour. It also removes that old chestnut of plausible deniability: “Honest, officer, I didn’t know it was wrong!”

The training must be accredited, and must be high quality. It should not be a “tick the box” exercise. And if we circle back to incentives, the best way to encourage researchers to undertake the training is to make proof of training a requirement for obtaining a grant.

To those naysayers who say it will never happen, let me tell you that it already has. The Irish Health Research Board has recently implemented such a scheme.

‘Predatory, evil’ scholarly journals

Finally, I am concerned that the incentives in the research system are not just driving bad behaviour for researchers, but are also creating a market for criminals to enter scholarly publishing.

What is motivating the crooks is the pay-per-page system that has come with the introduction of open access publishing.

Now, open access publishing has many benefits and I support the move towards it. But I remain concerned that it has opened the door for predatory, evil, crooked journals.

It is just too easy to set up a journal and a website with a highfalutin title, and appropriate the biographies of leading researchers for the editorial board – without their knowledge or permission. Before you know it, huge numbers of papers are being published without any rigour.

And there are researchers who are knowingly paying to publish in journals that have no peer review, even though they claim to. Journals that have no ethics. Not even an editorial team to consider the submitted paper.

These researchers might just be naïve, but we have to acknowledge that the current incentives reward this behaviour.

While this is not a major problem in Australia, emerging research nations are really struggling with this.

In my conversations with senior research leaders around the world, they are looking for ways to improve performance metrics in a way that does not drive their researchers to these journals.

I propose a rigorous quality assurance system, designed to inform stakeholders that a particular journal’s processes for assessing a paper meets agreed publishing standards. I like to call it Publication Process Quality Assurance, or PPQA.

Compliance with PPQA would indicate to researchers, research institutions, libraries and granting agencies that the journal follows internationally accepted guidelines for the publication process.

Granting agencies are best placed to provide the incentive for researchers to only publish in PPQA-compliant journals by enforcing it through their grant application process.

Follow the money

You might have picked up by now a common thread; that in each of my three recommendations, I am looking to take the responsibility back to the granting agencies. It’s a concept referred to by others as “follow the money”.

If the granting agencies put in place these measures, they will ripple through into the research institutions and mitigate the ongoing risks of poor quality research.

It will change the culture and ensure that last century’s academic rigour continues for the 21st century research workforce and beyond.

This is an edited version of a speech Dr Finkel delivered in Melbourne on September 12 at the Peter Doherty Institute for Infection and Immunity, to mark its fifth anniversary.

Alan Finkel, Australia’s Chief Scientist, Office of the Chief Scientist

Article source: This article is republished from The Conversation under a Creative Commons license. Read the original article.

Header image: Australia’s Chief Scientist Alan Finkel. AAP.