Skip navigation
Skip navigation
Open Research will be down for maintenance between 8:00 and 8:15 am on Tuesday, December 1 2020.

Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?

Chuard, Pierre J. C.; Vrtílek, Milan; Head, Megan; Jennions, Michael D.

Description

There is increased concern about poor scientific practices arising from an excessive focus on P-values. Two particularly worrisome practices are selective reporting of significant results and ‘P-hacking’. The latter is the manipulation of data collection, usage, or analyses to obtain statistically significant outcomes. Here, we introduce the novel, to our knowledge, concepts of selective reporting of nonsignificant results and ‘reverse P-hacking’ whereby researchers ensure that tests produce a...[Show more]

dc.contributor.authorChuard, Pierre J. C.
dc.contributor.authorVrtílek, Milan
dc.contributor.authorHead, Megan
dc.contributor.authorJennions, Michael D.
dc.date.accessioned2019-02-19T02:10:48Z
dc.date.available2019-02-19T02:10:48Z
dc.identifier.issn1545-7885
dc.identifier.urihttp://hdl.handle.net/1885/156427
dc.description.abstractThere is increased concern about poor scientific practices arising from an excessive focus on P-values. Two particularly worrisome practices are selective reporting of significant results and ‘P-hacking’. The latter is the manipulation of data collection, usage, or analyses to obtain statistically significant outcomes. Here, we introduce the novel, to our knowledge, concepts of selective reporting of nonsignificant results and ‘reverse P-hacking’ whereby researchers ensure that tests produce a nonsignificant result. We test whether these practices occur in experiments in which researchers randomly assign subjects to treatment and control groups to minimise differences in confounding variables that might affect the focal outcome. By chance alone, 5% of tests for a group difference in confounding variables should yield a significant result (P < 0.05). If researchers less often report significant findings and/or reverse P-hack to avoid significant outcomes that undermine the ethos that experimental and control groups only differ with respect to actively manipulated variables, we expect significant results from tests for group differences to be under-represented in the literature. We surveyed the behavioural ecology literature and found significantly more nonsignificant P-values reported for tests of group differences in potentially confounding variables than the expected 95% (P = 0.005; N = 250 studies). This novel, to our knowledge, publication bias could result from selective reporting of nonsignificant results and/or from reverse P-hacking. We encourage others to test for a bias toward publishing nonsignificant results in the equivalent context in their own research discipline. There is concern that the scientific literature is biased towards reporting statistically significant results. By contrast, this Perspective article presents evidence for an unusual situation in which there is a systematic bias towards publishing non-significant findings.
dc.description.sponsorshipPC and MV were funded by Australia Awards-Endeavour Fellowships (IDs 6114_2017 and 6558_2018, respectively) awarded by the Australian Government (URL: https:// internationaleducation.gov.au/Endeavour% 20program/Scholarships-and-Fellowships/Pages/ default.aspx); and MH and MJ were funded by the Australian Research Council (DP160100285 and FT160100149) (URL: https://www.arc.gov.au/ grants/discovery-program)
dc.format7 pages
dc.format.mimetypeapplication/pdf
dc.language.isoen_AU
dc.publisherPublic Library of Science
dc.rights� 2019 Chuard et al.
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.sourcePLOS Biology
dc.subjectP-values
dc.subjectP-hacking
dc.subjectreverse P-hacking
dc.subjectsignificant results
dc.subjectnonsignificant result
dc.titleEvidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?
dc.typeJournal article
local.description.notesImported from PLOS
local.identifier.citationvolume17
dc.date.issued2019-01-25
local.identifier.ariespublicationu9511635xPUB1919
local.publisher.urlhttps://www.plos.org/
local.type.statusPublished Version
local.contributor.affiliationChuard, Pierre J. C., Division of Ecology and Evolution, CoS Research School of Biology, The Australian National University
local.contributor.affiliationVrtilek, Milan, Division of Ecology and Evolution, CoS Research School of Biology, The Australian National University
local.contributor.affiliationHead, Megan L., Division of Ecology and Evolution, CoS Research School of Biology, The Australian National University
local.contributor.affiliationJennions, Michael D., Division of Ecology and Evolution, CoS Research School of Biology, The Australian National University
dc.relationhttp://purl.org/au-research/grants/arc/DP160100285
dc.relationhttp://purl.org/au-research/grants/arc/FT160100149
local.bibliographicCitation.issue1
local.bibliographicCitation.startpagee3000127
local.identifier.doi10.1371/journal.pbio.3000127
dc.date.updated2019-01-27T09:05:30Z
dcterms.accessRightsOpen Access
dc.rights.licenseThis is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
CollectionsANU Research Publications

Download

File Description SizeFormat Image
01 Chuard P J C et al Evidence that nonsignificant 2019.pdf547.05 kBAdobe PDFThumbnail


This item is licensed under a Creative Commons License Creative Commons

Updated:  19 May 2020/ Responsible Officer:  University Librarian/ Page Contact:  Library Systems & Web Coordinator