Reply to Lim: Concerning MADCAP plots Free

We thank Mr Lim [1] for highlighting issues associated with the interpretation of MADCAP plots [2] and welcome the opportunity to try to clarify matters. In partial mitigation, the first author (S.G.) started his research career in the field of geometry and perhaps this had undue influence on our choice of wording. In Euclidean geometry [3], if two lines are parallel, they are not ‘divergent’ and ‘divergence’ is not a measure of their distance apart, although in other contexts the word ‘divergence’ may be taken to mean this. We have used the word ‘diverge’ as in a dictionary definition ‘to separate and go in different directions’ [4].

Otherwise, Mr Lim’s opinion seems to be broadly in accord with ours. For a given risk score, the actual distance apart of two plots is not necessarily cause for concern and may indeed be a consequence of cumulative plots carrying ‘memory’. However, when judging a risk model there may be cause for concern if a discrepancy between the two plots persists over the entire range of risk scores, as in Fig. 1 of our paper. Of particular interest are those intervals where there is substantial difference in the direction of the two plots. Charting the distance between the two plots, as in Fig. 2 of our paper, highlights this (avoiding the ‘memory problem’). In the middle section the chart climbs consistently indicating divergence (i.e. ‘difference in direction’) and underestimation of mortality within this sub-range of risks.

Whether or not inaccuracies in risk estimates for low-risk cases are ‘minor’ is something of a matter of opinion and depends on context. If patients and their families were assured that an operation carries no risk of death when in fact the risk is known to be say 0.005%, this may be a ‘minor’ error, but equally it is inexcusable.

We accept that there are elements of subjectivity concerning the interpretation of MADCAP plots and their use is certainly not intended to provide a means for formal statistical testing but to complement such methods. That said, formal hypothesis testing may indicate an adequate p-value, and may appear to be more objective, but may give a falsely optimistic impression of the worth of a risk scoring system. As we say ‘the acid test is to consider what uses the risk model will have and whether it is fit for these purposes’ [2].

While an adequate MADCAP plot does not give proof positive that a risk scoring system is sound, and we never suggested that it did, plots that indicate systematic discrepancies, such as Figs. 1 and 2 of our paper, do suggest that further investigation would be wise.

We note that software for constructing MADCAP plots is available, free of charge, at www.ucl.ac.uk/operational-research.

References