1): this implies that, for an individual whose underlying true value is equal to the true group mean, the long-run distribution of measured values would overlap with the entire population distribution of true values, under ideal conditions. A reliability of 0.5 means that there is equal true and error-related variance ( Fig. Accordingly, a reliability of 1 means that all variability is attributable to true differences and there is no measurement error, while a reliability of 0 means that all variability is accounted for by measurement error. Hence, it yields information regarding the overall consistency of a measure, the distinguishability of individual measurements, as well as the signal-to-noise ratio in a set of data. Conceptually, reliability refers to the fraction of the total variance which is not attributable to measurement error ( Fleiss, 1986). If we consider a series of measurements of a particular steady-state attribute in a set of individuals, the variability between the measured values can be attributed to two components: inter-individual differences in the true underlying value of the attribute, and differences due to measurement error. In the assessment of individual differences, reliability is typically assessed using test-retest reliability, inter-rater reliability or internal consistency. In summary, these methods and tools will allow researchers to avoid performing costly studies which are, by virtue of their design, unlikely to yield informative conclusions. An R package called relfeas also accompanies this article for performing these calculations. In this way, the feasibility of a new study can be assessed during planning stages, and before collecting any new data. I present a new method and tools for using summary statistics from previously published test-retest studies to approximate the reliability of outcomes in new samples.

Reliability is usually assessed in validation studies using healthy participants, however these results are often not easily applicable to clinical studies examining different populations. This problem is exacerbated by the use of outcome measures whose accuracy is sometimes insufficient to answer the scientific questions posed. These constraints limit the possibility of large-sample experimental designs, and often lead to statistically underpowered studies.

Neuroimaging, in addition to many other fields of clinical research, is both time-consuming and expensive, and recruitable patients can be scarce.

Test of reliability in research