Validation of the EORTC QLQ-C30 and QLQ-BN20, including WHO performance status interrater reliability, for evaluation of patients with intracranial meningiomas

Abstract

Background

The EORTC questionnaires QLQ-C30 and QLQ-BN20 are commonly used to evaluate health-related quality of life in patients with meningiomas but have not undergone a disease-specific validation. The study aimed to address this issue and to determine the interrater reliability of WHO performance status (PS) assessments in these patients.

Methods

This population-based study included prospectively enrolled intracranial meningiomas treated at Umeå University Hospital between October 14, 2010, and December 31, 2021, followed up until March 30, 2023. Patients were assessed by the EORTC questionnaires before and at 3 months after surgery. WHO PS categorized as high (0–1) or low (2–5) were evaluated for interrater reliability and used together with sick-leave status to determine the questionnaires’ clinical validity. Remaining psychometric properties of the questionnaires were analyzed by conventional methods.

Results

Of 513 eligible surgeries, 454 (88.5%) had responded to at least 1 questionnaire. WHO PS interrater agreement was 94.4%. The EORTC questionnaires’ ability to distinguish between clinically distinct groups was high. Items correlated better with their own scale than others (most r > 0.70). Items measuring various aspects of the same construct showed good internal consistency (nearly all α > 0.70). Questionnaire responsiveness to symptom changes over time was acceptable. Several scales displayed floor and ceiling effects.

Conclusions

The EORTC QLQ-C30 and QLQ-BN20 are overall valid instruments to evaluate patients with intracranial meningiomas but require awareness of certain limitations when specific functions and symptoms are addressed. WHO PS assessments can be applied to meningioma patients with high reproducibility between observers.

Meningiomas are the most prevalent of intracranial tumours¹ and may cause a wide spectrum of symptoms, ranging from neurological disabilities and seizures to psychiatric disorders and cognitive deficits. Patients also frequently experience nonspecific symptoms such as fatigue, sleep disturbance, and headache. While neurosurgical resection, in addition to tumor removal, aims to alleviate symptoms and increase well-being, many patients nonetheless sustain disabilities several years after treatment.^2,3 In addition, recent studies have demonstrated that patients treated for meningiomas report considerable limitations in long-term health-related quality of life (HRQoL).^3,4 However, large population-based reports on long-term HRQoL for patients with meningiomas remain scarce, in particular, studies using validated instruments.²

The European Organization for the Research and Treatment of Cancer Core Quality of Life Questionnaire (EORTC QLQ-C30) is a generally well-established tool for assessing HRQoL and symptoms among cancer patients.⁵ Combined with the brain cancer module QLQ-BN20, the instruments capture a wide range of functional deficits and symptoms specific for brain cancer patients.⁶ While specific tools to assess HRQoL in patients with meningiomas have been suggested,⁷ the EORTC forms remain among the most widely applied in meningioma research and were considered most suitable in a review of questionnaires to evaluate HRQoL in patients with meningiomas specifically.² Although the EORTC instruments have been validated in patients with brain tumors of different histological subtypes,^8–13 most have focused exclusively on malignant tumors such as gliomas or metastases, for which management, treatment, and clinical course differ substantially. An adequately powered and disease-specific validation of the EORTC questionnaires to evaluate HRQoL in patients with meningiomas has thus become increasingly warranted.^2,3,14

The present study aims to validate the EORTC QLQ-C30 and BN20 questionnaires for assessment of HRQoL and symptoms in patients specifically with intracranial meningiomas. To anchor the questionnaires to a preexisting measure of global functional status, the WHO performance status (PS) grading system, as well as working capacity estimated via sick-leave statistics, were used. Moreover, as the interrater agreement of PS ascertainment between healthcare professionals may in some settings be subject to variability,^15,16 and its applicability to meningiomas patients unestablished, a corresponding supplemental analysis was also included.

Methods

Study Design

This is a population-based validation study approved by the Regional Ethics Committee. The study cohort was composed of prospectively enrolled patients with intracranial meningiomas eligible for neurosurgical treatment at Umeå University Hospital between October 14, 2010, and December 31, 2021, with follow-up until March 30, 2023. Written informed consent was obtained from all participants before study inclusion. HRQoL was assessed by means of the EORTC core questionnaire (QLQ-C30) and brain-specific module (QLQ-BN20) prior to surgery, and at 3 months after treatment in connection to a routine outpatient follow-up visit. All patients who had undergone craniotomy for a histopathologically confirmed intracranial meningioma and completed at least 1 of the 2 questionnaires were included. Baseline characteristics were collected by scrutiny of medical records. All patients’ functional status were estimated according to the WHO PS grading system by retrospective chart review. The WHO PS estimations were matched to the time point of each HRQoL assessment and ascertained by 2 separate investigators (R.N. and K.H.) for all patients treated prior to December 31, 2020, and categorized as having either a high (PS 0–1) or a low (PS 2–5) PS. As patients treated during the year 2021 were included for study at a later time point, and the number of observations to estimate interrater reliability was already considered sufficient, WHO PS for these patients was determined by one of the investigators exclusively (K.H.). Data on sick leave was collected from the Swedish Social Insurance Agency for patients aged 18–60 years, and patients were considered being on sick leave if receiving any of the following financial aids: sickness benefit, preventive sickness benefit, rehabilitation benefit, occupational injury sickness benefit, early retirement benefit, sickness allowance, sickness compensation, or activity compensation. Patients’ sick-leave status was dichotomized as being on sick leave (fully or partially), or not, and was determined matched to the date on which the questionnaire had been completed for each patient individually.

Instrument

The EORTC QLQ-C30 questionnaire consists of 30 items organized into 5 functional scales (physical, role, emotional, cognitive, and social functioning), 3 multi-item symptom scales (fatigue, nausea and vomiting, and pain), 6 single-item symptom scales (dyspnea, insomnia, appetite loss, constipation, diarrhea, financial difficulties) as well as a global health status/QoL scale. The items are scored on a 4-point Likert scale ranging from “not at all” (1) to “very much” (4), except for 2 items assessing global health status/QoL scored on a 7-point linear analog scale.

The EORTC QLQ-BN20 is a supplemental questionnaire to be used in addition to the QLQ-C30 in patients with brain tumors and comprises 20 complementary items, each scored on the same 4-point scale used in the QLQ-C30. The QLQ-BN20 is organized into 4 multi-item symptom scales (future uncertainty, visual disorder, motor dysfunction, and communication deficit) and 7 single-item symptom scales (headaches, seizures, drowsiness, hair loss, itchy skin, weakness of legs, and bladder control).

Raw scores were calculated and linearly transformed into scale scores ranging from 0 to 100 for all scales in the questionnaires according to the EORTC QLQ-C30 and QLQ-BN20 respective scoring manuals.¹⁷ Higher scores on a functional scale and the global health status/QoL scale indicate higher levels of functioning and health status/QoL, respectively, whereas higher scores on a symptom scale represent more pronounced symptoms.

Validity

Psychometric properties of the questionnaires were evaluated by assessment of floor and ceiling effects, convergent and discriminant validity, internal consistency, clinical validity, responsiveness, and construct validity. To clarify, floor and ceiling effects are present when a substantial number of respondents report the lowest or highest scores possible on a specific item or scale, which may indicate that the symptom or experience measured could be of limited relevance to the patients or lack sensitivity to different levels of severity. Convergent validity determines the relationship between a scale and its associated items, whereas discriminant validity addresses the relationship between a scale and other questionnaire items that are considered conceptually unrelated. Low convergent validity would, for example, indicate that items do not correlate with their associated scale, and that the scope of the items is likely too wide or unspecific, while low discriminant validity would denote too much association between items and scales that are conceptually unrelated. Internal consistency measures the correlation between items within the same multi-item scale (ie, that items supposed to be associated display converging scores). Clinical validity and responsiveness evaluate the questionnaires’ ability to distinguish between well-defined groups of patients based on their clinical condition. Shortcomings in clinical validity would indicate that the instruments cannot differentiate between patients whose clinical status differs significantly, while low responsiveness would suggest a lack of ability to capture changes over time when the same construct is measured at different time points. Construct validity refers to how well an item or scale measures the specific concept it is intended to evaluate (eg, if the Physical function scale actually reflects the respondent’s physical function) and, in addition to convergent and discriminant validity, was evaluated by measuring the level of association between different scales of the QLQ-C30 and QLQ-BN20. Strong associations between conceptually different scales could for instance indicate that they reflect constructs that they are not intended to.^18,19

Statistical Analyses

The interrater reliability of WHO PS assessments was analyzed and presented with level of agreement as well as Cohen’s kappa coefficient (κ).²⁰ While the latter to some extent constitutes an arbitrary measure, interpretation adhered to generally established guidelines, in which a value of 0–0.6 would denote poor to moderate agreement, 0.6–0.8 would suggest substantial conformity, whereas a value greater than or equal to 0.8 would indicate near-perfect agreement.²¹

To ensure that the study was sufficiently powered to validate the EORTC questionnaires, a sample size calculation was performed prior to study start. In line with previous validation reports, a minimum of 10 observations per item was deemed sufficient,^8,22 and a sample size of at least 500 completed forms was thus considered necessary to evaluate the total of 50 items covered by the 2 EORTC questionnaires. It was subsequently stipulated that at least 50 patients with intracranial meningiomas had undergone surgery at the study hospital per year, hence corresponding to 100 questionnaires if all eligible patients had chosen to participate and completed both the preoperative and the postoperative forms. As the study period spanned over more than 11 years, at least 550 patients were assumed to have had been operated on during that time, resulting in a potential 1100 questionnaires having been distributed. A response rate as low as 50% would thus amount to 550 questionnaires completed and still be able to account for substantial missing and unforeseen dropouts. As the response rate was expected to be significantly higher than 50%, the number of completed questionnaires was thus expected to meet and likely exceed the required sample size of 500.

Mean value and standard deviation (SD) were calculated for all EORTC questionnaire scale scores. Item floor and ceiling effects were considered present if >15% of respondents had reported scores on the lower or upper limit.²³ Convergent validity was analyzed by means of Spearman correlation between each item and its own scale, for which a correlation coefficient (r) > 0.40 was considered significant. Discriminant validity was assessed by comparing the interconnectedness between items and scales, wherein an item was expected to correlate better (>2 SE) with its own scale, as when compared to the other scales. Internal consistency among items in the same multi-item scale was calculated with Cronbach’s alpha (α), where α > 0.70 was deemed acceptable.²⁴ Clinical validity was determined based on the questionnaires’ ability to distinguish between known groups of patients, that is, groups of patients categorized as high or low WHO PS, and between those with working capacity versus those on sick leave, respectively. Differences between groups were analyzed with the Mann–Whitney U test. Low WHO PS (PS 2–5) was hypothesized to be associated with reduced physical functioning, fatigue, insomnia, visual disorder, motor dysfunction, drowsiness, weakness of legs, and reduced global health status/QoL. Sick leave was hypothesized to be associated with reduced physical, emotional, and cognitive functioning, as well as pain, insomnia, financial difficulties, future uncertainty, visual disorder, motor dysfunction, headache, seizures, drowsiness, weakness of legs, and reduced global health status/QoL. Furthermore, to assess responsiveness of the questionnaires to changes over time, patients were classified as having worsened (indicated by a shift in WHO PS from high [0–1] to low [2–5]), or having remained stable or improved (denoted by an unchanged or any shift to a higher WHO PS). Shifts in PS were then compared with changes according to the questionnaires (indicated by a worsened, stable, or improved scale score) during the same time period using Fisher’s exact test. Finally, evaluation of construct validity was complemented by analyzing Spearman correlation between the scales of QLQ-C30 with those of QLQ-BN20, for which r > 0.40 was considered a meaningful correlation. While some specific constructs are included in both questionnaires, a few correlations were expected, whereas the overall aim of the 2 questionnaires is to supplement each other, and should thus result in mostly low to moderate correlations between scales. In line with previous validation studies, only complete case analyses were included.^8,10 All statistical analyses were conducted using the computer software Stata/BE version 17 (StataCorp LLC). Significance level was set at P < .05.

Results

Baseline Characteristics

Of a total of 577 surgeries screened, 513 were deemed eligible and had been queried for study inclusion, out of which 52 had declined study participation, with a total of 454 remaining and included in the study cohort. Patient selection and exclusion criteria are shown in Figure 1. Baseline characteristics including age, sex, tumor location, WHO grade, number of tumors resected during each surgery, and numerical order of the operation are provided in Table 1.

WHO PS Interrater Reliability

The interrater reliability analysis of WHO PS ascertainment indicated a substantial level of agreement (94.4%), κ = 0.76 (95% CI 0.69–0.83) (Table 2).

EORTC Questionnaires’ Descriptive Statistics

A total of 441 (97%) preoperative and 326 (72%) postoperative EORTC questionnaires were completed. Mean values and SD for the QLQ-C30 and QLQ-BN20 scales, including floor and ceiling effects, as well as WHO PS and sick-leave status are shown in Table 3. Due to administrative failure, 60 patients (13.6%) had completed the intended preoperative questionnaire first after surgery and were therefore excluded from the analysis concerning preoperative scale scores. In addition, as scale scores could not be calculated in patients with single items missing, the total number of observations varied slightly between different scales. Several items and scales demonstrated floor and ceiling effects, reflecting that many patients experienced pronounced impairments or did not experience certain symptoms or loss of specific functions at all.

Validity

The convergent validity of the questionnaires met the preset criteria |r| > 0.40 for item-own scale correlation except for item 5 (“Do you need help with eating, dressing, washing yourself, or using the toilet?”) and the Physical functioning scale (|r| = 0.35).

All items correlated better with their own scale compared to the other scales. Correlation coefficients were >2 SE higher for all item-own scale correlations compared to item-other scale correlations, and thus fulfilled the predetermined criterion for discriminant validity. Moreover, the internal consistency of the multi-item scales met the preset threshold (α > 0.70) for all scales except for Nausea and vomiting (α = 0.56) (Table 4).

All scale scores were significantly different between patients with high and low PS. Patients with high PS reported higher scores for the Global health status/QoL scale and Functional scales corresponding to higher health status/QoL and level of functioning, respectively, and lower scores for the symptom scales corresponding to a lower level of symptomatology. All results were thus in line with the clinical hypotheses stipulated a priori. For sick-leave status, scale scores differed significantly between those with working capacity and those on sick leave for all scales except Constipation, Diarrhea, Seizures, Hair loss, and Itchy skin. In similarity with patients with high PS, patients who were not on sick leave reported higher scores for the Global health status/QoL scale and Functional scales, and lower scores for the symptom scales. The results were thus all in agreement with the stipulated hypotheses except for the Seizures scale (Table 5).

In the responsiveness analyses based on shifts or stability in WHO PS through surgery, significant differences were observed for all scales except Dyspnea, Insomnia, Diarrhea, Communication deficit, Seizures, Hair loss, Itchy skin, and Bladder control. For the Headaches scale, the mean score decreased, indicating a reduced level of symptomatology among patients with worsened WHO PS, while for Financial difficulties the mean score increased, indicating an increased experience of financial difficulties among patients with stable or improved WHO PS. For every other scale, patients with worsened WHO PS reported on average reduced health status/QoL and functioning, and an increased level of symptomatology (Table 5).

Correlations between scales of QLQ-C30 and scales of QLQ-BN20 are shown in Supplementary Table 1 (Appendix). All correlations except one (Seizures—Constipation) were statistically significant. However, correlation coefficients were with few exceptions higher for correlations between the functional and multi-item scales than for correlations involving symptom or single-item scales. As expected, some correlations between symptom scales were observed, for example, Fatigue—Drowsiness (|r| = 0.71), Pain—Headache (|r| = 0.56), Physical functioning—Weakness of legs (|r| = 0.53).

Discussion

The EORTC questionnaires QLQ-C30 and QLQ-BN20 are among the most widely employed and advocated to assess HRQoL in patients with meningiomas but, prior to this report, had not undergone a disease-specific validation. While the instruments demonstrate high clinical and overall validity, there are limitations researchers need to be aware of when the questionnaires are applied to meningioma patients specifically. Several items display floor and ceiling effects, which may indicate a lack of sensitivity to levels of severity of specific functions and symptoms, and for which complementary measures are likely necessary. Also originally developed to study malignant brain tumors, some items were expected to be of limited relevance to patients with meningiomas but this seems an acceptable drawback considering the high response and completion rate found. Moreover, a high concordance of WHO PS assessments between 2 separate investigators was shown and suggests that WHO PS categorized binarily can be a useful measure with substantial interrater agreement when applied to patients with intracranial meningiomas.

There are several limitations to this study. First, although the number of unique observations (n = 767) greatly exceeded the stipulated 500 responses required, it could be argued that the number of unique responders did not meet the preset sample size. However, review of the statistical analyses and results does not indicate that lack of statistical power had a significant impact on the study, as the absolute majority of analyses and all clinical hypotheses reached formal statistical significance or were distinctly negative. Second, this is a single-center study using only the Swedish version of the EORTC questionnaires. Thus, cultural and linguistic diversity are not accounted for and may theoretically reduce the external validity of the results. However, previous validation studies on the EORTC QLQ-C30 and QLQ-BN20 that excluded patients with meningiomas nonetheless included culturally diverse samples of patients with brain cancer from several different countries, of which none reported issues in terms of generalizability within these domains,^5,8 suggesting that translation and applicability across different languages and cultures would unlikely be of major concern for patients with meningiomas specifically. Third, due to administrative failure, a non-negligible number of patients responded to the preoperative form first after surgery but were, as a consequence, excluded from all analyses concerning preoperative scale scores. While this may, on the one hand, be considered a drawback, the figure is in line with a corresponding Scandinavian report on HRQoL in patients with diffuse gliomas using the EORTC instruments.²⁵ Importantly, on the other hand, it highlights the current study’s population-based design, which is a significant strength that contrasts with similar validation reports that have been based on highly selected randomized controlled trials,⁸ or in which the patient recruitment process is not reported,^9–11,13 resulting in significant limitations in terms of evaluating questionnaire acceptability and external validity. Furthermore, WHO PS was estimated by retrospective chart review exclusively. For most patients, scrutiny of medical records rendered sufficient information to determine functional status; however, medical notes were in some cases summary, thus resulting in estimations that were likely less accurate. Still, a substantial level of agreement between observers was demonstrated.

The population-based design is a major strength of the current study and shows that 9 out of 10 patients were willing to respond to the questionnaire. Although attitudes toward research participation may differ between different countries and cultures, this finding nevertheless indicates a high acceptability of the EORTC QLQ-C30 and QLQ-BN20 questionnaires among patients with intracranial meningiomas. The adequate sample size is also an important methodological strength and contrasts with most similar validation reports, wherein insufficient statistical power prevented firm and detailed conclusions to be drawn.^10–12 Additionally, as previous studies have found varying interrater concordance in PS assessment between different diagnoses,²⁶ an interrater reliability analysis of the WHO PS was included and showed a substantial level of agreement, thus substantiating its applicability. The subsequent use of WHO PS as a validated anchor in addition to data on sick leave to assess the clinical validity of the EORTC instruments adds to the robustness of the report.

Convergent and discriminant validity met the preset criteria for nearly all analyses, indicating an adequate representation and categorization of items into scales. Surprisingly, the only exception was the correlation between item 5 (“Do you need help with eating, dressing, washing yourself, or using the toilet?”) and the Physical functioning scale. While this may intuitively appear as a false-negative result, it could also reflect that the patients’ perception of capability of performing activities of daily living is not solely dependent on their physical functioning but also cognitive and mental functioning to a significant (and possibly even greater) extent.²⁷ Moreover, except for Nausea and vomiting, the internal consistency was high for all multi-item scales, hence demonstrating high uniformity between items within the same multi-item scale. One possible explanation for the Nausea and vomiting internal inconsistency could be that these symptoms are addressed separately by the items included in the multi-item scale, and that several patients might have experienced nausea as an isolated symptom without vomiting. Another explanation could be that these symptoms are infrequent in patients with meningiomas, a notion further supported by the vast floor effects observed within these domains, whereas especially in contrast to malignant tumors, more often eligible for chemotherapy, and thus with a much higher likelihood of suffering from these symptoms.²⁸ In addition, none of the scales evaluated demonstrated very high internal consistency (generally α > 0.95), which indicates that items supposed to evaluate a common construct managed to do so effectively by adding information rather than by replicating each other.

Results of the clinical validity analyses indicated a high level of concordance between the questionnaires and WHO PS and sick-leave status. The questionnaires were also responsive to changes in WHO PS, as scale score changes were in line with shifts in PS for most scales. These results altogether indicate that the questionnaires are susceptible to clinically and functionally important differences between patients, including their progression over time. For some symptom scales, however, score changes were not statistically significantly different between patients classified according to shifts in PS. Although functional deficits and specific symptoms are expected to correlate to some extent, the nonsignificant changes in mean scores observed for some specific symptoms may reflect that the WHO PS grading system primarily mirrors the patient’s overall functional status rather than the influence of certain symptoms. Furthermore, the mean score difference of the Physical functioning scale was surprisingly low (0.28) among patients with stable or improved WHO PS. However, since this group mostly composed of patients with stable WHO PS, few observations may be a possible explanation of this finding.

Several scales of the QLQ-C30 correlated considerably with those of QLQ-BN20. Some correlations were expected due to potential overlapping between certain scales, for example, Pain—Headache. However, correlations between more conceptually different scales were also observed, for example, Motor dysfunction—Cognitive functioning (|r| = 0.45), possibly reflecting a greater complexity of neurological deficits in patients experiencing these symptoms.

Although no adequately powered study has validated the QLQ-C30 and QLQ-BN20 questionnaires for meningioma patients specifically, the instruments have previously been evaluated in studies on patients with brain tumors,^8–13 2 of which included meningiomas.^10,13 While our findings in general conform to the results of these studies, pronounced floor effects suggest some of the questionnaire items are likely of limited relevance when applied to patients with intracranial meningiomas specifically. As an expected observation, this item redundancy could nevertheless be viewed as an acceptable limitation considering the high response rate and completion rate of the questionnaires. However, item ceiling effects were also found and may indicate a lack of questionnaire sensitivity to the full range of severity within these domains specifically. Awareness of these limitations is thus required when the EORTC instruments are applied to meningioma patients, while supplemental analyses are likely necessary to be able to stratify between patients when these outcomes are of particular interest. Moreover, in contrast to several previous studies, the internal consistency for the Cognitive functioning scale met the preset threshold to be considered significant in the current report.^10,12,13,22 This observation may reflect the higher internal validity of the present report wherein meningiomas were studied exclusively.

There are today several generic and brain tumor–specific questionnaires available to study patients with meningiomas, out of which the EORTC questionnaires have been considered one of the most appropriate to evaluate meningioma patients specifically.² Nonetheless, there has also in line with the current study findings been increasing awareness that generic instruments may lack sensitivity to nuances of specific symptoms and functions,^7,29 which have resulted in recent efforts to develop and validate meningioma-specific questionnaires.⁷ While these disease-specific questionnaires seem promising, healthy reference populations, cross-cultural validation, and translation across different languages are still lacking at this stage and may suggest that the EORTC questionnaires will likely continue to play an important role in studies on meningiomas. Importantly, however, researchers need to be aware of strengths of limitations of different questionnaires already at the time of study conceptualization and design in order to be able tailor the study approach to the research question at hand, and for which the current study aims to provide sufficient details for evaluation of the EORTC instruments for studies on meningiomas specifically.

In conclusion, the present study shows that the EORTC QLQ-C30 and QLQ-BN20 questionnaires are generally valid instruments to assess HRQoL and symptoms in patients with intracranial meningiomas but require awareness of possible limitations when specific functions and symptoms are evaluated. In addition, WHO PS assessment based on chart review had substantial interrater agreement when patients were categorized as either having a high or a low PS, suggesting that dichotomized WHO PS could be a useful measure in studies on patients with intracranial meningiomas.

Conflict of interest

None declared.

Funding

Umeå Clinical Neuroscience Research Fund; the Regional agreement between Umeå University and Västerbotten County Council (ALF) [RV-979700]; the Foundation for Medical Research in Skellefteå.

Ethics approval statement

The study was approved by the Regional Ethics Committee at Umeå University (DNR 2016-200-31, 2021-06594-02).

Permission to reproduce material from other sources

No material from other sources was included in this article.

Authorship statement

Conceptualization: A.T.B., K.H.; Investigation: R.N., E.S., K.H.; Data analysis: R.N., K.H.; Methodology: A.T.B., R.L.S., K.H.; Writing—editing and critical review of manuscript: R.N., A.T.B., R.L.S., P.L., K.H.; Supervision: K.H.

Data availability

Upon reasonable request, data and methodology, including STATA software code, can be shared. However, access to such data will be subject to external review by the Swedish Ethical Review Board and Umeå University concerning data sharing according to the European Union general data protection regulation (GDPR).

References