The Challenges of Validating in Precision Medicine: The Case of Excision Repair Cross‐Complement Group 1 Diagnostic Testing

Abstract

Personalized medicine relies upon the successful identification and translation of predictive biomarkers. Unfortunately, biomarker development has often fallen short of expectations. To better understand the obstacles to successful biomarker development, we systematically mapped research activities for a biomarker that has been in development for at least 12 years: excision repair cross‐complement group 1 protein (ERCC1) as a biomarker for predicting clinical benefit with platinum‐based chemotherapy in non‐small cell lung cancer. We found that although research activities explored a wide range of approaches to ERCC1 testing, there was little replication or validation of techniques, and design and reporting of results were generally poor. Our analysis points to problems with coordinating and standardizing research in biomarker development. Clinically meaningful progress in personalized medicine will require concerted efforts to address these problems. In the interim, health care providers should be aware of the complexity involved in biomarker development, cautious about their near‐term clinical value, and conscious of applying only validated diagnostics in the clinic.

Introduction

The movement into the personalized medicine era brings much promise for cancer care. This is especially true for treatment modalities that are highly burdensome or that entail narrow therapeutic indices. Here, treatment decisions based on biomarker status—that is, prospectively testable and clinically informative properties of a patient’s biological material—might spare patients the burden of toxic and ineffective therapy.

But despite advances in understanding the mechanisms of tumor survival, development of clinically useful biomarkers for predicting response to cancer therapies has proven challenging [1]. Progress toward a diagnostic for the excision repair cross‐complement group 1 (ERCC1) protein as a predictive biomarker for platinum‐therapy in advanced non‐small cell lung cancer (NSCLC) provides a prototypical illustration of the obstacles blocking the transformation of molecular insights into clinically useful applications.

Platinum‐based doublet chemotherapy is a pillar of advanced NSCLC treatment. Cisplatin and carboplatin, the two most commonly used platinum therapies, work by adding platinum adducts to individual nucleotides. This prevents DNA replication, resulting in cancer cell death. The ERCC1 protein is the rate‐limiting factor in the nucleotide excision repair (NER) pathway, which removes platinum adducts as part of normal cellular activity [2]. Because expression levels of ERCC1 vary from patient to patient, this makes it an attractive marker for potentially explaining differences in patient outcomes after platinum‐based treatment [3].

During a decade of clinical research, dozens of studies have investigated the prospect of correlating ERCC1 with clinical outcomes in advanced NSCLC patients after platinum therapy. Yet at best, only modest progress has been made. To better understand why, we systematically mapped the research portfolio and evolution of evidence in support of ERCC1 testing.

Materials and Methods

Literature Search

We searched Embase and Medline databases to identify studies that investigated a relationship between ERCC1 expression and clinical outcomes in advanced NSCLC patients receiving platinum therapy. Search terms included cancer type (“non‐small cell lung cancer,” “NSCLC”), biomarker (“excision repair cross‐complement group 1,” “ERCC1,” or “ERCC‐1”), and the therapy (“platinum,” “cisplatin,” “carboplatin”). We excluded nonempirical and non‐English reports. The database results were then supplemented with a hand search based on reviews and study references . All studies were screened first by title and abstract, and then by full text. Exclusion criteria at screening included (a) meta‐analysis, (b) abstract only (c) not original data, (d) study population (i.e., not advanced stage), and (e) did not report objective response (OR) or overall survival (OS) data.

Data Extraction

Data extraction was implemented with the Numbat Extraction Framework (available for free from http://github.com/bgcarlisle/Numbat) and included the following domains: (a) study characteristics, (b) design, (c) assay properties, (d) reporting quality, (e) OR data and whether marker status was significantly associated with OS, and (f) the stated utility of the marker as predictive or prognostic. All reports were independently extracted by Barsanti‐Innes and Hey, and all discrepancies were settled through deliberation. Following the recommendations by Simon et al. (2009) [8], we created a five‐point scale for study quality based on (a) use of control specimens, (b) successful determination of biomarker status for more than 66% of the specimens in study sample, (c) assessment of biomarker status blinded to clinical data, and (d) prospective definition for biomarker‐positive status.

Patterns of Research Activity

To assess the organization of research activities, we used Accumulating Evidence and Research Organization (AERO) graphing [9]. This is a graph‐theoretic method for visually representing the trajectory and structure of research in a scientific domain. Briefly, an AERO graph represents each experiment as a node arranged on the x‐axis by time and stratified by study properties on the y‐axis. Nodes are then color‐coded to represent the evolution of evidence. For our AERO analysis, we stratified by five levels: (a) assay method, (b) tissue type, (c) assay reagents, (d) prespecified cutoff value, and (e) drug regimen.

Results

Sample Characteristics

Twenty‐eight studies met our eligibility criteria; Figure 1 shows the PRISMA flow diagram. These reports spanned 12 years of published research; properties of our sample are shown in Table 1  . This study sample reflects 4,311 patient samples (2,295 tested for protein expression and 2,016 for mRNA) in total. Only 14% of the studies received financial support from commercial sponsors.

Methodological Quality

Two of 28 studies in our cohort (7%) used a prospective biomarker trial design. The others were retrospective analyses. Pathologists were named as coauthors in 68% of studies. The sample size of each study was generally small, ranging from 35 to 443 with a median of 101 specimen samples per study. Mean quality score was 2.6. Twelve investigations (43%) reported no more than two of our four quality‐score elements. Use of controls and the proportion of samples successfully analyzed were inadequately reported in 21% and 29% of studies, respectively. A prospective cutoff for biomarker status was unreported in 36% of studies, and blinding of pathologists determining biomarker status to clinical response went unreported in 39% of studies. We did not observe an obvious trend toward greater rigor in study design as studies advanced from hypothesis generation toward confirmatory testing. No study after 2010 adequately reported all four methodological practices. Also, fewer than half of studies discussed the analytical validity of their biomarker diagnostics (12 of 28; 43%).

Predictive Versus Prognostic Biomarker Potential

Most of the studies (82%) included only patients who had received platinum therapy. However, 7 of those 23 (35%) made claims of testing predictive marker utility, 3 (11%) made claims of testing prognostic utility, and 18 (64%) made a claim that conflated the prognostic and predictive utility of the marker.

Patterns of Research Activity

Tables 2 and 3 list the diversity of components used in the study designs of our sample. Even simple biomarker tests consist of a coordinated set of practices and conditions, including a definition of tissues to be analyzed, an assay method, and a scoring rule. We call these coordinated practices and conditions a “biomarker ensemble.” Figure 2 depicts all of the combinations of the five components defining our biomarker ensemble that are present in our sample. In total, 24 different combinations were investigated. However, only three of these combinations were ever replicated.

The precise content of these combinations is further complicated by parameters that have mixed or unreported populations. Of the studies in our sample, 39% enrolled patients from more than two different treatment regimens, which we classified as “pragmatic” regimens. Also, 39% of studies did not specify the type of tissue used for analysis, largely rendering these results uninterpretable. In each of the three replicated ensembles at least one parameter had such a mixed population (tissue type, pragmatic regimen, or both).

The two prospective studies in our sample applied diagnostic components that had been not been previously validated—the 18SrNA internal reference gene [21] and automated quantitative analysis technique [13]. These particular techniques were then never followed up in later research.

Only six studies in our sample used a prespecified numerical cutoff (i.e., did not use the median score in the sample population as the cutoff). However, two of the six studies that reported an externally valid cutoff also used a pragmatic patient population , and one did not specify the location of the tumor sample [13].

Finally, in 50% of the studies in our sample there was discordance between ERCC1’s predictive correlation with OR and OS (represented by yellow nodes in Fig. 2). Even where these outcomes were concordant within a study, there was no consistency across our sample. Twenty‐five percent of studies reported concordant positive outcomes, and 25% reported concordant negative outcomes.

Discussion

The state of uncertainty regarding the value of ERCC1 testing is, in many ways, representative of personalized medicine more generally [1]. Many commentators have cataloged impediments to the successful translation of biomarkers, extending from validity issues in basic science [38] to the large‐scale coordination of geographically and technologically disparate research centers . Accordingly, there are numerous sets of recommendations in the literature for improving the analytical validity of assay reagents and increasing necessary reporting for tumor marker studies . There have also been calls for implementing a registry system focused on tumor markers and establishing a system with clear milestones for determining biomarker utility .

Our analysis suggests other impediments as well. Diagnostic tests consist of a set of practices—such as assay protocols, tumor preservation methods, type of tissues analyzed, scoring criteria, and so on—that must be combined to unlock the clinical value of a given biomarker. Determining whether a biomarker diagnostic should be implemented for patient management requires optimizing each of these variables [48]. The ERCC1 biomarker translation trajectory shows how difficult and prolonged translation can be when poor study design and reporting combines with a haphazard approach to testing these different variables.

For example, of the 16 studies in our sample that measured ERCC1 protein expression, all but one of them used an antibody, 8F1, that may not exclusively target ERCC1. The antibody 8F1 first appears in our cohort in 2005 and is utilized for the next 9 years, despite concerns relating to its specificity emerging in 2007 [49]. One study showed that ERCC1 is not the principal antigen recognized by 8F1 and that this antibody is incapable of distinguishing between high and low ERCC1 expression in human cell lines [50]. More recently, ERCC1 protein has been shown to exist in four functionally distinct isoforms, with only one of them displaying the target NER capacity [51]. Given that stratification of marker‐positive and ‐negative groups may have been based on faulty reagents, this may mean that half of our sample produced uninformative results.

Similar validity issues confront studies evaluating mRNA. Reports in our sample do not appear to have clearly addressed the selection of the internal reference gene used to quantify the amount of ERCC1 mRNA. Only one study noted that they used a reference gene that had confirmed normal expression in tumor cells [20]. Also, the primers used during the quantitative reverse‐ transcriptase polymerase chain reaction process are not isoform specific, resulting in the same potential for misclassification that may have occurred with the protein evaluations [6].

Scoring criteria represent another dimension that confounds predictive marker validation. Of the studies in our sample, 75% used the median value of ERCC1 expression to stratify their patient populations in a binary fashion according to whether they exceeded or were lower than this value. Although specifying a median value cutoff may be useful in exploratory investigations, this approach lacks the capacity for externally valid inference and compromises development of a useful patient selection criterion for use in a clinical setting.

Controversy also exists about the comparability of molecular profiles between tumor sites . Studies contrasting ERCC1 protein expression between primary and metastatic tissues in NSCLC have shown that levelsare often unrelated . Similarly, ERCC1 mRNA levels have been shown to lack the correlation between tumor and peripheral samples [24]. Despite this, 18% of the studies in our sample combined results from patient specimens that came from disparate sites, and 46% did not report the location of the sample origin. Clinicians wishing to apply personalized medicine strategies—or advance the science—need to be aware that how and from where biospecimens are collected can dramatically influence the results of biomarker analyses.

Variation in patient drug regimen is another potential confounder and barrier to progress. Cisplatin and carboplatin, the two most common types of platinum therapy, while known to target DNA using similar mechanisms, are also known to have different toxicity and efficacy profiles [57]. Yet half of the studies in our sample did not limit patient eligibility on the basis of which drug they received. This further complicates the ability of researchers to isolate the effect of ERCC1 expression on the response to a particular treatment.

NSCLC platinum therapy is also typically given in combination with other drugs—often as a doublet with paclitaxel, gemcitabine, docetaxel, vinorelbine, irinotecan, or pemetrexed [58]. Of the studies in our sample, 79% compared patients who had been treated with different drug regimens, and 39% compared more than two different regimens. This introduces another potential confound in isolating the effect of ERCC1 in moderating the effectiveness of the platinum therapy.

Biomarkers can also serve different purposes in directing patient management. A predictive marker identifies a population that is more likely to respond favorably to a particular therapy, thereby directly influencing patient treatment selection. In contrast a purely prognostic marker means that when treatment is not taken into consideration, a biomarker‐selected population will have a better clinical outcome. Markers can be prognostic or predictive or have characteristics of both. To distinguish between these two types of markers, a study must compare patient data from both biomarker groups, in which sufficient patients from each designation receive the marker‐associated therapy as well as standard of care, as in a randomized clinical trial. Only five studies in our cohort included ERCC1 high‐ and low‐expressing patients and compared the difference in outcomes between platinum‐based and nonplatinum‐based regimens. The remaining 23 studies included only patients who received platinum therapy, making it difficult to determine a truly predictive or prognostic use for the ERCC1 diagnostic. This information is necessary to guide clinicians tasked with selecting appropriate patient treatments .

Finally, the low‐level of reporting quality, particularly among the more recent investigations, represents unethical research waste of scarce research and biospecimen resources. As Figure 2 shows, 12 studies failed to report more than two of our four methodological elements. This greatly compromises the interpretability of reports and renders findings of little use in systematic reviews or quantitative synthesis.

The difficulties being experienced by researchers evaluating ERCC1 as a biomarker for personalizing medicine has, unfortunately, not prevented the introduction of several commercially available diagnostics promoting the ability to use the marker in selecting patients for platinum therapy . A recent study evaluating the reliability of three such assays demonstrated large variability between the tests in determining ERCC1 expression levels in identical patient samples, and furthermore, none could distinguish responders from nonresponders to platinum therapy [66]. Furthermore, clinical guidelines have vacillated in their inclusion of ERCC1 as a useful biomarker. In 2011 a special National Comprehensive Cancer Network task force reported ERCC1 as a biomarker with emerging evidence [67], and in 2013 their clinical practice guidelines included it as a marker to predict the efficacy of platinum therapy in NSCLC patients [68]. However, the most recent guidelines make no reference to the marker [69]. Yet several of the tests remain commercially available.

Our analysis of the ERCC1 research program illustrates how, despite a wide exploration of biomarker ensemble parameters and very compelling biological and clinical rationales, there has been little or no convergence on an optimal, clinically useful technique for stratifying patients on the basis of ERCC1 marker status. This can be explained, in part, by a failure to address the many dimensions of uncertainty surrounding a biomarker test in a systematic fashion. In the ideal development trajectory, a range of plausible values for the key elements in the ensemble would be tested in an exploratory setting. Once optimal values are discovered, these would then be incorporated into large‐ scale, decisive tests in appropriate patient populations to determine the diagnostic’s utility.

The current, passive mechanisms for research coordination appear insufficient to motivate and support attainment of key translation milestones of optimizing each component. This underscores the need for novel social mechanisms to guide activities in this domain, both in the form of a top‐down regulatory body that can help direct investigators toward critical unanswered questions and in the form of better adherence to standards in methodology and reporting practices. One possibility for biomarker research moving forward is the establishment of a formalized tool for coordinating ongoing investigations. In our work we present the AERO diagram as a means of retrospectively illustrating past issues in the portfolio of ERCC1 research. However, by tracking the progressing accumulation and reporting quality of research, it could also provide a valuable prospective tool to improve biomarker development. In highlighting unexplored areas of research, or areas of conflict, an AERO diagram would help direct investigators to the most pressing areas of research. Particularly if this tool were given the support of agencies such as the National Cancer Institute or the National Human Genome Research Institute, this kind of official status could incentivize its use among individual researchers, improve research communication, and reduce research waste.

Furthermore, there is an important gatekeeping role for journal publishers and peer reviewers in vetting submitted articles to ensure their adherence to reporting guidelines such as REMARK (REporting recommendations for tumour MARKer prognostic studies). Clinicians and especially pathologists participating in biomarker research either directly or through the provision of biospecimens can play a role in this by being aware of past and present research in their area to ensure that resources are being used efficiently to answer the most pressing translational questions and by ensuring that their own publications, and those that they review, meet reporting standards.

Our study has some limitations. Our literature search will have excluded some studies that examined the assay methods in isolation from the particular clinical application of interest. We also note that our “quality scale” has not been validated as a metric. However, we do not think it likely that these limitations, if rectified, would substantially alter our analysis or conclusions.

Conclusion

Researchers, policy makers, and funding agencies often talk about biomarkers as though their predictive capacities reside in a particular molecular signature rather than the clinical and assay operations used to access and define that signature . Simple “biomarker speak” can obscure the challenges of both translating and implementing personalized cancer care.

Our findings reinforce three points that bear on the practice of personalized cancer care. First, clinicians should be mindful that the prognostic or predictive value of a biomarker is highly sensitive to the conditions under which a biomarker test is implemented. Realizing the promise of personalized cancer care requires testing and optimizing each of these conditions. Second, current research systems are not well suited to efficient and reliable testing of these conditions. Research on a biomarker is often pursued in parallel rather than in a coordinated fashion, greatly prolonging development. Physicians can advocate for a more methodical and coordinated approach to research. Finally, an awareness of the scientific and social complexities in translating prognostic or predictive biomarkers should motivate a more cautious appraisal of the near‐term potential of personalized medicine.

Acknowledgments

This study was conducted under the PACEOMICS project, funded by Genome Canada, Genome Quebec, Genome Alberta, and the Canadian Institute for Health Research.

Author Contributions

Conception/Design: Brianna Barsanti‐Innes, Spencer Phillips Hey, Jonathan Kimmelman

Collection and/or assembly of data: Brianna Barsanti‐Innes, Spencer Phillips Hey, Jonathan Kimmelman

Data analysis and interpretation: Brianna Barsanti‐Innes, Spencer Phillips Hey, Jonathan Kimmelman

Manuscript writing: Brianna Barsanti‐Innes, Spencer Phillips Hey, Jonathan Kimmelman

Final approval of manuscript: Brianna Barsanti‐Innes, Spencer Phillips Hey, Jonathan Kimmelman

Disclosures

The authors indicated no financial relationships.

References

Author notes