Improving Prognostic Accuracy in Subjects at Clinical High Risk for Psychosis: Systematic Review of Predictive Models and Meta-analytical Sequential Testing Simulation

Abstract

Discriminating subjects at clinical high risk (CHR) for psychosis who will develop psychosis from those who will not is a prerequisite for preventive treatments. However, it is not yet possible to make any personalized prediction of psychosis onset relying only on the initial clinical baseline assessment. Here, we first present a systematic review of prognostic accuracy parameters of predictive modeling studies using clinical, biological, neurocognitive, environmental, and combinations of predictors. In a second step, we performed statistical simulations to test different probabilistic sequential 3-stage testing strategies aimed at improving prognostic accuracy on top of the clinical baseline assessment. The systematic review revealed that the best environmental predictive model yielded a modest positive predictive value (PPV) (63%). Conversely, the best predictive models in other domains (clinical, biological, neurocognitive, and combined models) yielded PPVs of above 82%. Using only data from validated models, 3-stage simulations showed that the highest PPV was achieved by sequentially using a combined (clinical + electroencephalography), then structural magnetic resonance imaging and then a blood markers model. Specifically, PPV was estimated to be 98% (number needed to treat, NNT = 2) for an individual with 3 positive sequential tests, 71%–82% (NNT = 3) with 2 positive tests, 12%–21% (NNT = 11–18) with 1 positive test, and 1% (NNT = 219) for an individual with no positive tests. This work suggests that sequentially testing CHR subjects with predictive models across multiple domains may substantially improve psychosis prediction following the initial CHR assessment. Multistage sequential testing may allow individual risk stratification of CHR individuals and optimize the prediction of psychosis.

Introduction

In the last 2 decades, a new research paradigm has supported the development of preventive interventions in individuals at clinical high risk (CHR) for psychosis.¹ Preventive intervention in CHR individuals for psychosis has unique and unprecedented potential in the history of psychiatry to alter the course of disabling illnesses such as schizophrenia (see meta-analyses of effective treatments in CHR individuals^2,3).

Effective preventive interventions for CHR individuals are limited by the ability to prognosticate psychosis onset from an initial CHR state. CHR psychometric instruments have excellent prognostic properties (AUC = 0.90),⁴ which is comparable to other preventive approaches in medicine.⁵ However, excellent prognostic performances are mainly mediated by an outstanding ability of the CHR instruments to rule out psychosis, ie, very low negative likelihood ratios and high sensitivity (SE), at an expense of their ability to rule in psychosis, ie, unsatisfactorily low positive likelihood ratios and only moderate overall specificity (SP).⁴ Specifically, the initial CHR testing can increase the probability of detecting risk of developing psychosis in subjects referred to high-risk services from 15% (pre-test risk) at approximately 3 years⁴ to a 26% probability of psychosis onset (post-test risk),⁴ mostly toward schizophrenia spectrum psychoses⁶ (for further details on pre- and post-test concepts please see Fusar-Poli et al^7,8). Consequently, there is a need to improve the ability to rule in heightened risk of subsequent psychosis, while preserving the outstanding ability to rule it out.⁴ Improved prediction would facilitate personalized interventions and minimize either unnecessary treatment (for the false positives) or lack of treatment (for the false negatives). To improve the limited positive predictive values (PPVs) delivered by psychopathology-based classifications associated with CHR instruments,⁹ models with biological, neurocognitive or environmental data have been developed. In fact, the use of predictive models¹⁰ along with sequential multistage testing¹¹ is common practice in preventive medicine to improve prognostic discrimination between individuals who will develop a certain condition and those who will not.

This study first presents a systematic review of predictive models used to improve prediction of psychosis onset in CHR. We systematically reviewed prognostic accuracy metrics (SE, SP, PPV, negative predictive value (NPV), for details see Fusar-Poli et al⁷) across clinical, biological, neurocognitive, environmental, and combined predictive models. In a second step, we sought to investigate the potential clinical utility of sequential 3-stage testing following an initial CHR assessment. We employed meta-analytical simulation analyses across different combinations of models and critically discussed the findings in light of risk stratification approaches.¹²

Methods

Search Strategy and Selection Criteria

A systematic search strategy identified relevant articles. Three investigators (MC, GR, AS) conducted a 2-step literature search. At a first step, the Web of Knowledge database by Thomson Reuters was searched, incorporating both the Web of Science and MEDLINE. The search was extended until October 2015. We used several combinations of the following keywords: “at risk mental state,” “psychosis risk,” “prodrome,” “prodromal psychosis,” “high risk,” “prognostic accuracy,” “sensitivity,” “specificity,” “psychosis prediction,” “psychosis onset,” and the name of each CHR assessment instrument. The second step involved using Scopus to search citations of previous systematic reviews on transition outcomes in CHR subjects and a manually searching the reference lists of retrieved articles. Articles identified through these 2 steps were then screened for the selection criteria on basis of abstract. The articles with potentially relevant abstracts were retrieved and assessed for eligibility.

Studies were included if the following criteria were fulfilled: (a) original articles, written in English; (b) inclusion of CHR subjects (ie, presence of attenuated psychosis symptoms [APS] or genetic risk and deterioration syndrome [GRD] or brief limited and intermittent psychotic symptoms [BLIPS] or brief intermittent psychosis syndrome [BIPS] or basic symptoms) according to international standard criteria¹; (c) inclusion of clinical, biological, neurocognitive, environmental, or combinations of predictors to distinguish CHR individuals who later developed psychosis from those who did not; (d) inclusion of appropriate predictive models, algorithms, or learning systems to predict the probability of transition to psychosis, such as regression (logistic, Cox proportional hazard model, least absolute shrinkage, and selection operator), support vector machines or greedy algorithms.^13–16 Exclusion criteria were: (a) abstracts, pilot datasets, reviews, articles in languages other than English; (b) inappropriate statistics (ie, use of mean differences or chi square tests); (c) studies testing the prognostic accuracy of the baseline CHR assessment as predictor (previously reviewed in Fusar-Poli et al⁴) (d) articles with overlapping datasets using the same predictor. Specifically, in case of multiple publications deriving from the same study population, we selected the articles reporting the largest, most recent data set. The search results were summarized according to the PRISMA guidelines¹⁷ (figure 1).

Recorded Variables

Data extraction was independently performed by 3 investigators (MC, GR, AS). The following variables were recorded from each article: author, year of publication, demographic characteristics of the CHR sample, predictor domain (clinical, biological, neurocognitive, environmental, combinations), cut-off of predictive variables, use of validation, type of CHR diagnostic instrument used, exposure to antipsychotics, follow-up time, predictive model and prognostic accuracy data (SE, SP, PPV, NPV). When prognostic accuracy data were not directly presented they were indirectly extracted from associated measures if possible. Additionally, we contacted all the corresponding authors to provide additional data when needed.

Meta-analytical Sequential Testing Simulations

Models Selection.

Using statistical probabilistic simulations based on Bayes’theorem,¹⁸ we estimated the theoretical PPV of a sequential 3-stage testing following the initial CHR assessment. Such testing included different combinations of 3 predictive models (eg, electroencephalography/clinical, magnetic resonance imaging, and blood markers). We restricted the simulations to 3 tests because more tests would be practically infeasible in clinical practice. Validation is of paramount importance if the estimation of the PPV has to work satisfactorily for individuals other than those from whose data the model was derived.^19,20 Therefore, we limited the potential combinations of tests to studies that had performed a validation of their models. Also, we did not mix together models that used the same type of predictive parameters, eg, we did not simulate combinations in which 2 of the assessments involved EEG.

Procedure.

See online supplementary methods 1 for mathematical details of these analyses, which were conducted using R software.²¹ Briefly, we simulated that an individual would first have a CHR baseline assessment, which would convert the “pre-CHR assessment probability of transition to psychosis” into a “post-CHR assessment probability of transition to psychosis”.⁷ The value of the latter would depend on the former, on the SE and SP of the CHR assessment, and on the result of the CHR assessment.⁴ If the CHR assessment was positive, the individual would then undergo a second test (eg, a structural MRI) which would convert the “post-CHR assessment/pre-MRI probability of transition to psychosis” into a “post-MRI probability of transition to psychosis.” Again, the value of the latter would depend on the former, on the SE and SP of the MRI test, and on the result of the MRI assessment. These steps were repeated for each of the 3-stage tests.

Following this strategy, we obtained probabilistic 3-stage sequential testing diagrams such as the one shown in figure 2, in which the x-axis shows the sequential tests and the y-axis the probability of transition to psychosis before and after knowing the results of each of the tests. Each bifurcation in the plot represents the update in the probability of transition to psychosis after knowing that the test yielded a positive result (ascending solid line) or after knowing that the test yielded a negative result (descending dashed line).

We focused on the combination that yielded the best PPV, as this would be the one to be further validated and potentially applied in clinical practice⁴ (see online supplementary methods 2 for details). However, in order to provide the whole range of results from this simulation work, we also present the following less advantageous scenarios.

Firstly, we reported the poorer global PPVs of all other combinations of tests. Secondly, we estimated the lower limit of the 95% confidence interval of the global PPV. This global interval combined the confidence interval of the pre-CHR assessment probability to transition,⁴ plus the 4 confidence intervals associated to the CHR assessment and the 3 subsequent tests.

Thirdly, we recalculated the global PPV assuming a degree of correlation between the tests, so that the SE and SP of a test would depend on the results of the previous tests, decreasing the contribution of the test to the global PPV. For example, we assumed that among individuals who will have a psychotic episode, those with a positive CHR assessment are more likely to have a positive MRI test.

Impact of CHR Subgroups.

Finally, we repeated the simulation for different CHR subgroups. Given that the BLIPS/BIPS shows the highest risk of transition to psychosis, which is comparable to other brief psychotic disorders coded in international manuals,²⁵ the GRD the lowest risk and the APS an intermediate risk,²⁶ we conducted a separate analysis to test the impact of the CHR subgroup on the final prognostic accuracy.

Theoretical Clinical Effectiveness of 3-Stage Sequential Testing.

We further assessed the theoretical clinical effectiveness of 3-stage sequential testing by estimating the number needed to treat (NNT) at each node, assuming a risk ratio for preventative treatments of 0.54 as reported in previous meta-analysis of RCTs in CHR patients.³

Results

Selection of Studies

The electronic and manual searches returned 1300 studies. After the screening of abstracts 112 full articles were retrieved for further evaluation (figure 1). Twenty-five of them met the inclusion criteria; 10 studies using clinical predictive models, 5 studies using biological models, 5 studies using neurocognitive models, 5 studies using environmental models, and 8 studies using combinations of predictive models across different domains. The details of the included studies are reported in table 1.

Clinical Predictive Models

The 10 studies testing prognostic accuracy of clinical predictive models are shown in table 2. These tested a wide range of clinical parameters including specific positive,^{27,31,32,38,40,42,46,48} negative^27,32,38 and basic symptoms,⁴⁰ a decline in social and global functioning^27,31,36,46 and the Strauss and Carpenter Prognostic Scale.³⁷

The highest PPV of 86% was achieved by using a model including measures of odd beliefs, marked impairment in role functioning, blunted affect, auditory hallucinations, and anhedonia/asociality.²⁷ This model yielded an SE of 84%, SP of 86%, and NPV of 84%. None of the clinical predictive models have been validated.

Biological Predictive Models

Five studies tested the prognostic accuracy of biological predictive models (table 2). The tested models referred to measures of gray matter volume,²³ electrophysiological markers,^39,44,47 and blood analytes.²⁴

The highest PPV of 83% was achieved using gray matter volumes as the predictive variable, which produced an SE of 76%, SP of 85%, and NPV of 78%.²³ This²³ and 2 other biological predictive models^24,47 have been cross-validated.

Neurocognitive Predictive Models

Five studies tested the prognostic accuracy of cognitive predictive models (table 2). Cognitive predictive models included measurements of IQ,^30,32,40 verbal memory,^30,35 executive functioning,³⁵ attention,³² processing speed,^30,35 and speech perception.²⁹

Including verbal and executive functioning in the predictive model, the highest PPV of 83% could be achieved accompanied with an SE of 75%, SP of 80%, and NPV of 71%.³⁵ Only this model³⁵ has been validated in this domain.

Environmental Predictive Models

The prognostic accuracy of environmental predictive models was tested in 5 studies (table 2). These predictive models comprised substance abuse,^31,43 unemployment,³³ urbanicity,³⁴ social-sexual aspects,³⁴ and social maladjustments.^34,38

The highest PPV (63%) was produced by combining measures of urbanicity, social-sexual aspects, and social-personal adjustment, a predictive model that revealed an SE value of 63%, SP of 88%, and NPV of 88%.³⁴ None of the environmental predictive models have been validated.

Combinations of Predictive Models

Eight studies combined different predictive models across domains to test prognostic accuracy (table 2). These studies combined variables from 2 of the predictive models domains,^{22,28,31,32,38,40,41,45} but no study considered variables from 3 domains.

The highest PPV (82%) resulted from a predictive model including disorganized communication, suspiciousness, verbal memory deficit, and decline in social functioning. This predictive model yielded an SE of 60%, SP of 97%, and NPV of 93%.⁴⁵ Excluding this predictive model, 3 other combined predictive models^22,32,41 have been validated.

Validated Models Used in the Sequential Testing Simulations

Seven models with validation procedures were used for the simulations.^{22–,24,32,35,41,47} Model details are reported in table 3.

Meta-analytical Sequential Testing Simulations

We conducted 13 simulations in total, the details of which are reported in online supplementary figure 1. The highest PPV was achieved by sequentially using a combined model (clinical + EEG²²) and 2 biological (structural MRI²³ and blood markers²⁴) models (figure 2). Specifically, PPV was estimated to be 98% for an individual with 3 positive tests, 71–82% for an individual with 2 positive complementary tests, 12%–21% for an individual with 1 positive complementary test, and 1% for an individual with no positive tests (figure 2). Accordingly, the NNT was 2 for those with 3 positive sequential tests, 3 for those with 2 positive tests, 11–18 for those with 1 positive test, and 219 for those with no positive tests (see online supplementary table 1 for results in the bounds of the CI of the risk ratio for preventive treatments). This suggests that 3-stage sequential testing can significantly impact effectiveness of preventative treatments in CHR samples. To demonstrate the worst case scenario, we additionally used the lower limit of the confidence interval, producing lower but still medium PPVs: 49% for an individual with 3 positive tests, and 24%–30% for an individual with 2 positive tests (see online supplementary figure 2). PPVs after assuming the strongest possible correlation between the tests yielded high (or medium to high) PPVs: 98% for an individual with 3 positive tests, and 55%–81% for an individual 2 positive complementary tests (see online supplementary figure 3).

PPVs were similar when the analysis was restricted to CHR individuals meeting APS criteria at baseline (high for 2 or 3 positive tests and low otherwise), but globally higher when the analysis was restricted to CHR individuals meeting BLIPS/BIPS criteria (high for 2 or 3 positive tests, still medium for 1 positive test, and low otherwise), and globally lower when the analysis was restricted to CHR individuals meeting GRD criteria (high for 3 positive tests but medium for 2 positive tests, and low otherwise) (see online supplementary table 2 and supplementary figure 4).

Discussion

To our knowledge, this is the first study to systematically review predictive models for psychosis onset in CHR and to test the theoretical clinical utility of a 3-stage sequential testing to improve psychosis prediction. Twenty-five original studies were retrieved, addressing clinical, biological, neurocognitive, environmental, or combinations of predictive models across different domains. The highest PPV across environmental predictive models was modest (63%),³⁴ whereas the highest PPVs in clinical,²⁷ biological,²³ neurocognitive,³⁵ and combined⁴⁵ predictive models were above 82%. Thirteen 3-stage sequential testing simulations based on probabilistic risk assessment were conducted. The best model showed that probability of transition in a CHR individual was 98% if the 3 tests were positive based on 1 combined (EEG + clinical)²² and 2 biological predictive models (structural MRI and blood markers),^23,24 71%–82% if only 2 tests were positive, 1%–21% with 1 positive test, and 1% with no positive sequential tests.

We focused on PPV to improve risk prediction in CHR samples. This is based on the findings of our previous work which indicated that CHR instruments have excellent prognostic accuracy to rule out true negatives, subjects who not go on to develop psychosis.⁴ By contrast, there is still a need to specifically improve the ability to rule in subsequent psychosis and to increase PPVs.⁴ We examined 25 studies encompassing different types of biological, neurocognitive, environmental, and combined predictive models. The environmental predictive models yielded modest PPVs (63% for the best model³⁴). This may reflect a poor discriminative power that may be caused by heterogeneity in the environmental factors entered in the models. Environmental predictive models were mostly based on general factors associated with psychotic disorders such as substance abuse,^31,43 urbanicity,³⁴ unemployment,³³ and social maladjustment^34,38 so it is plausible that their specificity to CHR pathophysiology is relatively poor. These models were also characterized by poor methodological quality as none had employed validation analyses to confirm their findings. Conversely, the models from the other domains (clinical, neurobiological, neurocognitive, combination) with the highest PPVs had values above 82%. An additional finding is that of nonsuperiority of combined predictive models (82% for the model delivering the highest PPV⁴⁵) as compared to the other models such as neurocognitive models (83% the highest PPV³⁵). Similar findings have been observed for dementia prediction in patients with mild cognitive impairment where the accuracy of combined models (neuropsychological testing, health screening, neuroimaging, genetics, and informant or patient reports) did not significantly exceed that of more parsimonious models.⁴⁹ A previous study by our group found that combining cognitive, genetic, and imaging methods did not substantially improve the discrimination between healthy controls and CHR individuals.⁵⁰ Therefore, here we simulated the potential clinical utility of a 3-stage sequential probabilistic testing to refine psychosis prediction. Probabilistic testing analyses are common in other areas of preventative clinical medicine. For instance, they have been successfully applied to discriminate patients with Alzheimer’s disease from other forms of dementia.⁵¹

Because our 3-stage sequential testing analysis is not based on original data but on statistical simulations, we have restricted it only to the validated predictive models. Measures of prognostic accuracy are extremely sensitive to the design of the study and studies without validation procedures can severely overestimate the indicators of test performance. Overall, only 7 studies included in the current review have employed a rigorous prognostic accuracy approach combining appropriate predictive modeling with internal validation. Importantly, from the models with the highest PPVs, only the biological²³ and neurocognitive³⁵ but not the combined model⁴⁵ underwent validation. We thus tested if PPV could be improved on top of the initial CHR baseline assessment⁴ by sequentially combining 3 validated predictive models in 13 different combinations (see online supplementary figure 1). Our probabilistic testing simulations identified the best theoretical model, which was based on 1 combinatory (EEG + clinical)²² and 2 biological predictive models (structural MRI³⁶ and blood markers^23,24) (figure 2). This model showed that at least 3 positive tests are required to reach a high PPV for the development of psychosis (98%) and 2 negative tests to have low probability (1%–21%). These findings provide a theoretical framework suggesting that sequential testing in CHR individuals may improve psychosis prediction by stratifying individual risk profiles. It is striking that all 3 predictive model tests that were used in the meta-analytical probabilistic assessment included biological measurements. It may be speculated that since biological predictors of psychosis map direct neurobiological processes associated with the development of the illness, they have a high PPV. The first model in our simulation includes clinical variables (premorbid functioning) together with the P300 event-related potential.²² Interesting in this context is the fact that a recent meta-analysis confirmed that event-related potentials such as the P300 or the mismatch negativity (a measure of prediction error dependent learning⁵²) may be used as promising markers to predict the onset of psychosis.⁵³ The second test in our simulation includes gray matter volume reductions in prefrontal cortices such as dorso- and ventromedial areas as well as the cingulate cortex, which have been widely implicated in CHR pathophysiology,²³ and are known to be involved in cognitive processing.⁵⁴ Impairments in cognitive performance are associated with the onset of psychosis and may be useful in predicting psychosis.^55,56 Finally, our third test includes a multiplex blood assay.²⁴ Most of the blood analytes were involved in the regulation of the hypothalamic-pituitary axis, oxidative stress and inflammation, all of which are abnormal in patients with schizophrenia.^57–60 Consistent with the hypothesis that inflammation, oxidative stress, and dysregulation of hypothalamic-pituitary axes may be prominent in the earliest stages of psychosis,²⁴ CHR subjects had elevated cortisol levels and increased hypothalamic⁶¹ and pituitary volumes.⁶² Overall, our findings indicate that measures of pathophysiological anomalies may complement baseline clinical assessments to stratify CHR individuals into different risk groups, which in turn may lead to personalized treatments to prevent transition to psychosis.¹² In a first sensitivity-preserving step, CHR psychometric instruments could be used to rule out subjects seeking help at high-risk services but who are unlikely to develop psychosis. In a second step, additional tests of objective pathophysiological measures could be sequentially applied to the CHR group, with the aim of increasing prognostic reliability. Multicomponent sequential testing will not only decrease the risk of offering unnecessary treatment to false positives, but may also inform the treatment for people who do go on to develop psychosis. This could improve the benefits associated with early detection and early intervention, while reducing the possible costs (eg, weight gain) associated with receiving unnecessary pharmacological treatment.

Limitations and Future Directions

Our sequential testing is theoretical and not based on original data. Future original investigations should test the generalizability of our approach. Collaborative studies between international multisite CHR projects such as PRONIA (www.pronia.eu/), PSYSCAN (www.psyscan.eu/) and NAPLS3 (http://campuspress.yale.edu/napls/) are being planned and they may deliver large scale databases needed to externally validate the stepwise assessment identified by the current analysis. Another issue that may have influenced our simulation results may be the presence of affective comorbidities that can impact both psychopathology⁶³ and neurobiology⁶⁴ of a CHR sample. There were no data to test this in our review. It is also possible that duration of follow-up might affect our simulations. However, all predictive models employed have provided prognostic accuracy data in the longer period of time (baseline CHR assessment at 38 months,⁴ neurocognitive assessment at 48 months,³⁵ combined assessment at 36 months,²² neuroimaging assessment at 52 months²³), when most transition to psychosis would have already occurred.⁶⁵ Furthermore, we did not investigate outcomes other than psychosis transition, such as functional status,⁶⁶ remission,⁶⁷ or treatment responses,⁶⁸ which are becoming a mainstream focus of CHR research. Different sequential testing approaches are likely to be needed depending on the specific outcome to be predicted. Finally, the cost-effectiveness of having patients undergo neuroimaging testing will need to be established if there is any likelihood of integrating imaging into routine use.⁶⁹

Conclusions

The use of a sequential testing approach that improves baseline clinical assessments with predictive models from different domains, especially biological data may deliver high prognostic accuracy for psychosis prediction in subjects undergoing CHR assessment. Although our findings are theoretical and must be validated on original data, such probabilistic multimodal and multistep testing might help to improve the ability of high-risk services to stratify personalized risk profiles.

Funding

This work was supported by the Swiss National Science Foundation (SNSF; No. P2ZHP3_155184 to A.S.), the Instituto de Salud Carlos III—Subdirección General de Evaluación and the European Regional Development Fund (personal grant Miguel Servet CP14/00041 and project PI14/00292 integrated into the National Plan for research, development and innovation) (J.R.) and in part by a 2014 NARSAD Young Investigator Award (P.F.P). P.F.P. was also supported by the National Institute for Health Research (NIHR), Mental Health Biomedical Research Centre at the South London and Maudsley, NHS Foundation Trust and King’s College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Acknowledgments

The authors have declared that there are no conflicts of interest in relation to the subject of this study.

References

Author notes