Objective or subjective test? The answer lies in its psychometric property! Free

The appropriateness, meaningfulness, and usefulness of a test-score interpretation indisputably depends on two aspects: the test’s standardized administration protocol and the test’s internal characteristics. The latter is termed a tool’s psychometric properties. This applies to “a test-score” obtained after a test administration for screening, diagnosing or appraising treatment efficacy.

That is, we are all familiar with the administration and scoring guidelines of a sleep study and its derivatives, such as the Multiple Sleep Latency Test or Maintenance of Wakefulness Test [1]. We might furthermore be aware of the perils when not applying those guidelines. Polysomnography is after all our gold standard. Standardized subjective tools adhere to the same level of quality assurance. Each standardized subjective tool has a test manual or guideline, detailing its administration, and scoring. Moreover, the extensive information on its psychometric properties (e.g. parameters of interest, structure, reliability, validity, norms, etc.) is always included in the manual or guideline. Whilst a sleep study is believed to be objective or providing unbiased facts, a subjective test such as a questionnaire is too often (mistakenly) seen as less reliable being based on opinions, perceptions, or conjectures. Exactly here we, as sleep experts, are at risk of theoretical, and clinical fallouts upon interpreting an “objective or subjective” test-score.

In this issue of SLEEP, Schokman et al. [2] report a two-stage systematic review and nicely summarize potential risks. In the first stage, an identification of outcome measures in publications and clinical trial protocols reporting treatment efficacy in patients with narcolepsy type 1 and 2, was pursued. And second, an evaluation of those outcome measures by applying the COSMIN guideline (Consensus-Based Standards for the Selection of Health Measurement) [3] was performed. COSMIN aims to improve our selection of tools applied and available in healthcare, and is an initiative of an internationally multidisciplinary team of researchers beyond the sleep field. But probably, the most recommendable aspect of Schokman et al. [2] two-stage systematic review is the elaborate quest for the validation studies of the outcome measures found for their objective: “measures of treatment efficacy in RCTs in people with narcolepsy.” Validation studies scrutinize tools for biases, or the errors that may exist upon obtaining a test-score. That is, studies investigating (and unfortunately applying) tools, objective, or subjective, may show several biases in administration (e.g. items of an adult version modified towards children, or providing a parental version to an adolescent, careless placing of electrodes, inappropriate test setting, etc.), psychometric property (e.g. developed for one population but applied in a different one, changing items without assessing the factor structure, or reliability, etc.), scoring (e.g. changing the response format, careless application of scoring rules, etc.) and interpretation (e.g. inappropriate clinical cutoffs, or diagnosing with a screening tool, etc.). Regretfully for the sleep field, Schokman et al. [2] concluded that few validation studies exist and those that exist show little validity and reliability.

Undoubtedly, more validation studies of sleep tools are needed [4]. Tools that are applied clinically or experimentally, as well as tools that are “objective or subjective,” but foremost valid, reliable and unbiased are vastly needed. After all, our gold standard polysomnography or less gold standard actigraphy need administration, scoring, and interpretation in diverse populations across their lifespan. Similarly, our neuroscientific tests (e.g. attention or vigilance tests) and particularly questionnaires applied in the field need further validation.

Another lesson that can be learned from the work by Schokman et al. [2], is that psychometric properties such as structural validity, internal consistency, measurement invariance, reliability, measurement error, criterion validity, construct validity, responsiveness are not equivalent to a mere content validity (i.e. it appears to us to measure a sleep issue). For these reasons, when embarking on a quest to measure a sleep issue, it is necessary to find the right tool, apply the tool rightly, and use a righteous interpretation of the test-score. In other words, the test’s psychometric properties must be checked!

Disclosure Statement

Financial disclosure: The authors have no financial conflicts of interest to disclose.

Nonfinancial disclosure: The authors declare no potential nonfinancial conflicts of interest.

References