![]() What are the consequences of these two different scenarios? A famous example of the consequences of uncorrected multiple simultaneous statistical tests is the finding of fMRI BOLD activation in a dead salmon when appropriate corrections for multiple tests were not performed ( Bennett et al., 2010 Bennett et al., 2009). Unless a reasonable justification exists for this difference between the two scenarios, this is troubling. In this scenario, with the same data, we have two published positive findings compared to the single positive finding in the previous scenario. A second researcher now performs the second test ( v 1 ∼ v 3, p = 0.04) and deems this a positive finding too because it is under a p < 0.05 threshold and they have only performed one statistical test. No correction is performed, and it is considered a positive finding (i.e. Instead, the researcher only performs one statistical test ( v 1 ∼ v 2, p = 0.001). In this case, both tests are published, but only one of the findings is treated as a positive finding.Īlternatively, let us consider a different scenario with sequential analyses and open data. Thus, the researcher chooses to apply a Bonferroni correction such that p < 0.025 is the adjusted threshold for statistical significance. In many cases, we expect the researcher to correct for the fact that two statistical tests are being performed. The analysis yields p-values of p = 0.001 and p = 0.04 respectively. Let us now imagine that one researcher performs the statistical tests to analyze the relationship between v 1 ∼ v 2 and v 1 ∼ v 3 and decides that a p < 0.05 is treated as a positive finding (i.e. Imagine there is a dataset which contains the variables ( v 1, v 2, v 3). Thus, we have identified three desiderata regarding open data and multiple hypothesis testing: Sharing incentiveīefore proceeding with technical details of the problem, we outline an intuitive problem regarding sequential statistical testing and open data. Further, in order to ensure data are still shared, the sequential correction procedures should not be antagonistic with current data-sharing incentives and infrastructure. Sequential correction procedures are harder to implement than simultaneous procedures as they require keeping track of the total number of tests that have been performed by others. Here we will also propose a third, α -debt, which does not maintain a constant false positive rate but allows it to grow controllably. There are several proposed solutions to address multiple sequential analyses, namely α -spending and α -investing procedures ( Aharoni and Rosset, 2014 Foster and Stine, 2008), which strictly control false positive rate. Simultaneous procedures correct for all tests at once, while sequential procedures correct for the latest in a non-simultaneous series of tests. However, as we discuss in this article, when performing hypothesis testing it is important to take into account all of the statistical tests that have been performed on the datasets.Ī distinction can be made between simultaneous and sequential correction procedures when correcting for multiple tests. At present, researchers reusing datasets tend to correct for the number of statistical tests that they perform on the datasets. However, researchers re-analyzing these datasets will need to exercise caution if they intend to perform hypothesis testing. The availability of open datasets will increase over time as funders mandate and reward data sharing and other open research practices ( McKiernan et al., 2016). The ability to explore pre-existing datasets in new ways should make research more efficient and has the potential to yield new discoveries ( Weston et al., 2019). Making data open will allow other researchers to both reproduce published analyses and ask new questions of existing datasets ( Molloy, 2011 Pisani et al., 2016). In recent years, there has been a push to make the scientific datasets associated with published papers openly available to other researchers ( Nosek et al., 2015).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |