Science’s credibility crisis: why it will get worse before it can get better

Andrea Saltelli, University of Bergen

Science’s credibility crisis is making headlines once more thanks to a paper from John P. A. Ioannidis and co-authors. Ioannidis, an expert in statistics, medicine and health policy at Stanford University, has done more than anyone else to ring the alarm bells on science’s quality control problems: scientific results are published which other researchers cannot reproduce.

When the crisis erupted in the media in 2013 The Economist devoted it’s cover to “Wrong Science”. Ionannidis’s work was an important part of the background material for the piece.

In previous papers Ioannidis had mapped the troubles of fields such as pre-clinical and clinical medical studies; commenting how, under market pressure, clinical medicine has been transformed to finance-based medicine.

In this new work he and co-authors target empirical economics research. They conclude that the field is diseased, with one fifth of the subfields investigated showing a 90% incidence of under-powered studies – a good indicator of low-quality research – and a widespread bias in favour of positive effects.

The field of psychology had gone through a similar ordeal. Brian Nosek, professor of psychology at the University of Virginia and his co-workers ran a replication analysis of 100 high-profile psychology studies and reported that only about one third of the studies could be replicated.

Several other instances of bad science have gained attention in the media. The problems in “priming research”, relevant to marketing and advertising, prompted Nobel Prize winner Daniel Kahneman to issue a publicised statement of concern about the wave of failed replication.

And a study on “power poses”, which claimed that body posture influences a person’s hormones level and “feelings of power” went first viral on TED when it was published – then again when its replication failed.

We are observing two new phenomena. On the one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.

Fixes

In his newest work Ioannidis sets out a list of remedies that science needs to adopt urgently. These include fostering a culture of replication, data sharing and more collaborative works that pool together larger data sets; along with pre-specification of the protocol including model specifications and the analyses to be conducted.

Ioannidis has previously proposed additional remedies to “fix” science, as have other investigators. The list includes better statistical methods and better teaching of statistics as well as measures to restore the right system of incentives at all stages of the scientific production system – from peer review to academic careers.

Important work is already being done by committed individuals and communities, among them Nosek’s Reproducibility Project, Ioannidis’ Meta-research innovation centre, Ben Goldacre’s alltrials.net and the activities of Retraction Watch. These initiatives – which attracted private funding – are necessary and timely.

But what are the chances that these remedies will work? Will this crisis be solved any time soon?

Methods, incentives and introspection

Ioannidis and co-authors are aware of the interplay between methods and incentives. For example, they say they’d refrain from suggesting that underpowered studies go unpublished, “as such a strategy would put pressure on investigators to report unrealistic and inflated power estimates based on spurious assumptions”.

This is a crucial point. Better practices will only be adopted if new incentives gain traction. In turn the incentives will have traction only if they address the right set of science’s problems and contradictions.

Ethics is a crucial issue in this respect. And here is where research effort is lacking. The broader field of economics is aware of its ethical problems after Paul Romer – now chief economist of the World Bank – coined the new term “Mathiness”, to signify the use of mathematics to veil normative premises. Yet there seem to be some hesitation to join the dots from the methodology to the ethos of the discipline, or of science overall.

The book Science on the Verge has proposed an analysis of the root causes of the crisis, including its neglected ethical dimension. The formulation of remedial measures depends on understanding what happened to science and how this reflects on its social role, including when science feeds into evidence based policy.

These analyses are indebted to philosophers Silvio O. Funtowicz and Jerome R. Ravetz, who spent several decades studying the science’s quality control arrangements and how quality and uncertainty impacted the use of science for policy.

Ravetz’s book “Scientific knowledge and its social problems” published in 1971 predicted several relevant features of the present crisis.

For Ravetz it is possible for a field to be diseased, so that shoddy work is routinely produced and accepted. Yet, he notes, it will be far from easy to come to accept the existence of such a condition – and even more difficult to reform it.

Reforming a diseased field or arresting the incipient decline of another will be delicate tasks, adds Ravetz, which calls for a

sense of integrity, and a commitment to good work, among a significant section of the members of the field; and committed leaders with scientific ability and political skill. No quantity of published research reports, nor even an apparatus of institutional structures, can do anything to maintain or restore the health of a field in the absence of this essential ethical element operating through the interpersonal channel of communication.

Ravetz emphasises the loss of this essential ethical element. In later works he notes that the new social and ethical conditions of science are reflected in a set of “emerging contradictions”. These concern the cognitive dissonance between the official image of science as enlightened, egalitarian, protective and virtuous, against the current realities of scientific dogmatism, elitism and corruption; of science serving corporate interests and practices; of science used as an ersatz religion.

Echoes of Ravetz’s analysis can be found in many recent works, such as on the commodification of science, or on the present problems with trust in expertise.

A call to arms?

Ioannidis and co-authors are careful to stress the importance of a multidisciplinary approach, as both troubles and solutions may spill over from one discipline to the other. This would perhaps be a call to the arms for social scientists in general – and for those who study science itself – to tackle the crisis as a priority.

Here we clash with another of science’s contradictions: at this point in time, to study science as a scholar would mean to criticise its mainstream image and role. We do not see this happening any time soon. Because of the scars of “science wars” – whose spectre is periodically resuscitated – social scientists are wary of being seen as attacking science, or worse helping US President Donald Trump.

Scientists overall wish to use their moral authority and association with Enlightenment values, as seen in the recent marches for science.

If these contradictions are real, then we are condemned to see the present crisis becoming worse before it can become better.

Andrea Saltelli, Adjunct Professor Centre for the Study of the Sciences and the Humanities, University of Bergen, University of Bergen

Article source: This article was originally published on The Conversation. Read the original article.