Failure to agree with the results is not justification to dismiss them: the value of a randomized clinical trial

Based on the results obtained from their randomized clinical trial, Pulmonary Metastasectomy in Colorectal Cancer (PulMiCC) [1, 2], the outcome of which ran counter to the widely held belief that resection of colorectal pulmonary metastases offered a significant survival advantage over observation, Williams et al. set out to assess published references to the trial. In this study, they attempted to answer a simple question: what was the reaction of clinicians to the PulMiCC trial? [3] To the authors likely lack of surprise, many of the references to the trial were either critical or, in some instances, dismissive. This, despite the fact that most of the papers that referred to the trial either did not include data to support their contention or, in a few cases, misstated the data or conclusions.

The oft-cited 1997 paper from the International Registry of Lung Metastases that included data on 5206 patients as well as other single institution reported data, has supported continued enthusiasm for resection of pulmonary metastatic disease [4]. Of note, no randomized controlled trials (RCTs) were included in that 1997 report since no such trial had ever been done or at least published. The idea for the first RCT of pulmonary metastasectomy for colorectal cancer was first proposed in 2006 but the trial was not launched until 2010 [4]. Between 2010 and 2016 over 500 patients initially consented to registration, 484 met inclusion criteria but only 93 actually were randomized to either resection or observation. We recognize that patients struggle to agree to randomization when the choice is between an operation or continued observation and when combined with the inherent bias most surgeons maintain that resection of metastatic disease is beneficial, likely contributed to the challenge in accrual. One can argue that selection bias alone is responsible for the advantage in survival when resecting metastatic disease. Ideally, prior to the initiation of any clinical trial, there exists equipoise, a state in which the risks and benefits of alternative treatments offered during the trial are balanced, so that no pre-existing advantage is known to exist for 1 treatment arm over the other. One must assume the originators of this unique, precedent-setting trial were of that mindset when the trial was proposed. The authors had to recognize they were ‘swimming upstream’ but to their credit, persisted despite the challenges in accruing patients.

As noted above, the authors found that despite limited or no data in published reports that cited the PulMiCC data, the majority of references to the trial were negative, as judged by 4 independent researchers, indicative of the preconceived views of most thoracic surgeons. Interestingly many of the authors of the papers dismissive or critical of the trial had participated in the trial. To shed some additional light on this observation, lets for the moment envision a Texan (and I am not picking on Texans here, it is just that guns and Texans often appear together) who fires multiple shots into a barn wall and then proceeds to paint a target around the closest cluster of bullet holes. The shooter then points to the bullet-ridden target as evidence of their expert marksmanship. The so-called ‘Texas Sharpshooter Fallacy’ rests on the desire to cherry pick data based on a predetermined conclusion or outcome. Instead of letting the facts lead to a logical conclusion, evidence is ignored or discounted that contradicts their inherent beliefs [5]. Surgeons, as a group, tend to be very certain of their own opinions despite evidence that may be contrary to their inherent biases. As the great Joel Cooper pointed out on a number of occasions, 4 thoracic surgeons in a room often result in 5 different opinions. Those dismissive of the trial created their own ‘target’ around the data and criticized the trial based on the small number of patients accrued, the length of time it took to accrue even that small number and their own anecdotal experience.

Another fallacy operational here, that of the ‘Bandwagon Fallacy’, states that just because a significant group believe a proposition is true does not necessarily make it true. Popularity alone is not enough to invalidate an argument, or in this case, to be dismissive of the results of this first in class, randomized controlled trial [5]. Whether one likes the results, and recognizing that those who choose to enter a randomized trial may not be representative of the population at large, one cannot simply ignore or dismiss the outcome of such a trial. Our profession is replete with examples of observational trials that yielded positive results that ultimately failed to be reproduced in a well-designed RCT. We must also recognize that efficacy demonstrated in a trial does not always translate into effectiveness when the intervention is released to the general population.

The ‘Tu quoque Fallacy’ (Latin for ‘you also’) states that instead of presenting contrary or dissenting evidence, an attempt is made to discredit an argument or outcome simply by criticizing it [5]. Despite the negative or dismissive references identified by this paper, I congratulate the authors of the PulMiCC for the grit they demonstrated in completing this trial, albeit with a smaller accrual than originally anticipated, which now provokes continued questions as to whether we should be resecting colorectal pulmonary metastatic disease.

REFERENCES