Unreliable research

How scientists pressed for time inadvertently made it seem like there was a global jellyfish boom

Flawed citation practices can perpetuate scientific ideas even before they've been fully established as true.

When is a jellyfish plague not (necessarily) a jellyfish plague? When time-poor scientists selectively cite the literature to make it look like the oceans are flooded with jellies – even when it’s far from clear that they really are.

What does scientists being in a rush have to do with jellyfish populations? Let’s start from the beginning.

The identification of patterns and trends in nature happens through the accumulation of consistent observations, published in scientific reports. Once observed, the emerging patterns are usually reported in narrative reviews, which often stimulates a flurry of research activity in that field.

Eventually, the purported patterns are formally tested using “meta-analyses” of the published literature, to either confirm the pattern and establish it as theory, or refute it.

This path from the primary observations to theory can be traced through a network of citations.

Science, however, is done by humans and citation practices are subject to errors of bias and accuracy. Citation practices that are biased in a particular direction have the potential to lead to the identification of false patterns and flawed theory.

Enter the jellies

In the 1990s and 2000s, reports began to appear in the scientific literature of increased jellyfish populations in some parts of the world’s oceans. Various reviews reported the possibility that jellyfish blooms might be increasing globally. Over time, these became increasingly assertive about the existence and extent of the trend, until researchers were asking what to do about the increasingly “gelatinous state” of the oceans worldwide.

The question of whether the global jellyfish boom was real or not was tested by two meta-analyses – which came to opposite conclusions. A 2012 study concluded that populations were increasing globally because they found evidence for increasing populations in 62% of large marine ecosystems tested (although low certainty was assigned to two-thirds of these). The following year, another study found that only 30% of populations were increasing. It concluded that jellyfish populations wax and wane over several decades.

So, in reality, the scientific community is still divided over whether there really has been a sustained global increase in jellyfish numbers.

What about perception?

We wanted to know whether the perception of a global increase in jellyfish blooms was at least partly due to poor citation practices in the scientific literature. Our research, published in Global Ecology and Biogeography, suggests that it was.

Citation practices can be flawed in several ways:

  • Unsupported citations are when authors cite sources that contain no evidence that could possibly support the author’s claim.
  • Selective citations happen when a paper is cited to support a claim but contrasting evidence provided in the same paper is ignored, or when authors choose to cite earlier papers that have since been refuted.
  • Ambiguous citations happen when an author’s sentence contains multiple phrases, but the citations used to support each phrase are clustered at the end of the sentence, preventing readers from telling which is which.
  • Empty citations are when authors cite a paper that cites another paper as evidence for the claim, rather than the original source (also called “lazy author syndrome”).

We comprehensively searched the literature for papers, published before the two meta-analyses, that issued statements regarding trends in jellyfish populations. We classified each statement according to its affirmation and direction (that is, whether it said jellyfish are “increasing”, “may be increasing”, “decreasing”, or “not sure”), as well as their geographic extent (global, multiple regions, or one region).

We then assessed the papers cited as evidence of the statement, to see whether the citations were accurate or whether they fell into one of the categories of flawed citations outlined above.

A (jelly)fishy tale?

Of 159 papers that had issued statements about trends in jellyfish, 61% claimed that populations were increasing (27% at the global scale and 34% in multiple regions) and 25% asserted that populations may be increasing. Only 10% of papers said the data was equivocal. Just one reported that populations were decreasing (but at a local scale).

Most concerning was that only 51% of papers cited were considered to provide unambiguous support for the statements made by the authors. Almost all statements based on unsupportive citations were those claiming that jellyfish were increasing globally (despite the fact that it would have been impossible to make any claims about global trends before the first global meta-analysis was published in 2012). And in all cases, selective citations were biased towards claims that jellyfish populations were increasing.

Pressure to publish in prestigious journals and win research funds may lead some scientists to make claims that reach beyond the evidence available. In most cases, however, citation errors are not overt attempts to distort the evidence. Rather, they probably arise because increasing academic workloads reduce the time available to evaluate papers accurately and to keep abreast of the almost exponential increase in the volume of literature being published.

As scientists, we need to ensure that our claims are always supported by robust evidence because it is apparent that poor citation practices – and, in particular, selective citation of the literature – can distort perceptions within a research field.

Kylie Pitt, Associate Professor, Griffith University; Carlos Duarte, Adjunct professor, King Abdullah University of Science and Technology; Cathy Lucas, Associate Professor, Marine Biology & Ecology Research Group (MBERG), University of Southampton; Charles Novaes de Santana, Postdoctoral research associate, University of Zurich; Marina Sanz-Martín, Researcher, Department of Global Change Research, University of Barcelona, and Rob Condon, Assistant Professor in Biological Oceanography, University of North Carolina Wilmington.

This article first appeared on The Conversation.

We welcome your comments at letters@scroll.in.
Sponsored Content  BY 

As India turns 70, London School of Economics asks some provocative questions

Is India ready to become a global superpower?

Meaningful changes have always been driven by the right, but inconvenient questions. As India completes 70 years of its sovereign journey, we could do two things – celebrate, pay our token tributes and move on, or take the time to reflect and assess if our course needs correction. The ‘India @ 70: LSE India Summit’, the annual flagship summit of the LSE (London School of Economics) South Asia Centre, is posing some fundamental but complex questions that define our future direction as a nation. Through an honest debate – built on new research, applied knowledge and ground realities – with an eclectic mix of thought leaders and industry stalwarts, this summit hopes to create a thought-provoking discourse.

From how relevant (or irrelevant) is our constitutional framework, to how we can beat the global one-upmanship games, from how sincere are business houses in their social responsibility endeavours to why water is so crucial to our very existence as a strong nation, these are some crucial questions that the event will throw up and face head-on, even as it commemorates the 70th anniversary of India’s independence.

Is it time to re-look at constitution and citizenship in India?

The Constitution of India is fundamental to the country’s identity as a democratic power. But notwithstanding its historical authority, is it perhaps time to examine its relevance? The Constitution was drafted at a time when independent India was still a young entity. So granting overwhelming powers to the government may have helped during the early years. But in the current times, they may prove to be more discriminatory than egalitarian. Our constitution borrowed laws from other countries and continues to retain them, while the origin countries have updated them since then. So, do we need a complete overhaul of the constitution? An expert panel led by Dr Mukulika Banerjee of LSE, including political and economic commentator S Gurumurthy, Madhav Khosla of Columbia University, Niraja Gopal Jayal of JNU, Chintan Chandrachud the author of the book Balanced Constitutionalism and sociologist, legal researcher and Director of Council for Social Development Kalpana Kannabiran will seek answers to this.

Is CSR simply forced philanthropy?

While India pioneered the mandatory minimum CSR spend, has it succeeded in driving impact? Corporate social responsibility has many dynamics at play. Are CSR initiatives mere tokenism for compliance? Despite government guidelines and directives, are CSR activities well-thought out initiatives, which are monitored and measured for impact? The CSR stipulations have also spawned the proliferation of ambiguous NGOs. The session, ‘Does forced philanthropy work – CSR in India?” will raise these questions of intent, ethics and integrity. It will be moderated by Professor Harry Barkema and have industry veterans such as Mukund Rajan (Chairman, Tata Council for Community Initiatives), Onkar S Kanwar (Chairman and CEO, Apollo Tyres), Anu Aga (former Chairman, Thermax) and Rahul Bajaj (Chairman, Bajaj Group) on the panel.

Can India punch above its weight to be considered on par with other super-powers?

At 70, can India mobilize its strengths and galvanize into the role of a serious power player on the global stage? The question is related to the whole new perception of India as a dominant power in South Asia rather than as a Third World country, enabled by our foreign policies, defense strategies and a buoyant economy. The country’s status abroad is key in its emergence as a heavyweight but the foreign service officers’ cadre no longer draws top talent. Is India equipped right for its aspirations? The ‘India Abroad: From Third World to Regional Power’ panel will explore India’s foreign policy with Ashley Tellis, Meera Shankar (Former Foreign Secretary), Kanwal Sibal (Former Foreign Secretary), Jayant Prasad and Rakesh Sood.

Are we under-estimating how critical water is in India’s race ahead?

At no other time has water as a natural resource assumed such a big significance. Studies estimate that by 2025 the country will become ‘water–stressed’. While water has been the bone of contention between states and controlling access to water, a source for political power, has water security received the due attention in economic policies and development plans? Relevant to the central issue of water security is also the issue of ‘virtual water’. Virtual water corresponds to the water content (used) in goods and services, bulk of which is in food grains. Through food grain exports, India is a large virtual net exporter of water. In 2014-15, just through export of rice, India exported 10 trillion litres of virtual water. With India’s water security looking grim, are we making the right economic choices? Acclaimed author and academic from the Institute of Economic Growth, Delhi, Amita Bavisar will moderate the session ‘Does India need virtual water?’

Delve into this rich confluence of ideas and more at the ‘India @ 70: LSE India Summit’, presented by Apollo Tyres in association with the British Council and organized by Teamworks Arts during March 29-31, 2017 at the India Habitat Centre, New Delhi. To catch ‘India @ 70’ live online, register here.

At the venue, you could also visit the Partition Museum. Dedicated to the memory of one of the most conflict-ridden chapters in our country’s history, the museum will exhibit a unique archive of rare photographs, letters, press reports and audio recordings from The Partition Museum, Amritsar.

This article was produced by the Scroll marketing team on behalf of Teamwork Arts and not by the Scroll editorial team.