Artificial intelligence and consistency in patient care: a large-scale longitudinal study of mammographic density assessment

Abstract

Objectives

To assess whether use of an artificial intelligence (AI) model for mammography could result in more longitudinally consistent breast density assessments compared with interpreting radiologists.

Methods

The AI model was evaluated retrospectively on a large mammography dataset including 50 sites across the United States from an outpatient radiology practice. Examinations were acquired on Hologic imaging systems between 2016 and 2021 and were interpreted by 39 radiologists (36% fellowship trained; years of experience: 2-37 years). Longitudinal patterns in 4-category breast density and binary breast density (non-dense vs. dense) were characterized for all women with at least 3 examinations (61 177 women; 214 158 examinations) as constant, descending, ascending, or bi-directional. Differences in longitudinal density patterns were assessed using paired proportion hypothesis testing.

Results

The AI model produced more constant (P < .001) and fewer bi-directional (P < .001) longitudinal density patterns compared to radiologists (AI: constant 81.0%, bi-directional 4.9%; radiologists: constant 56.8%, bi-directional 15.3%). The AI density model also produced more constant (P < .001) and fewer bi-directional (P < .001) longitudinal patterns for binary breast density. These findings held in various subset analyses, which minimize (1) change in breast density (post-menopausal women, women with stable image-based BMI), (2) inter-observer variability (same radiologist), and (3) variability by radiologist’s training level (fellowship-trained radiologists).

Conclusions

AI produces more longitudinally consistent breast density assessments compared with interpreting radiologists.

Advances in knowledge

Our results extend the advantages of AI in breast density evaluation beyond automation and reproducibility, showing a potential path to improved longitudinal consistency and more consistent downstream care for screened women.

Introduction

Breast density refers to the amount of fibroglandular (“dense”) tissue within the breast. Breast density tends to gradually decrease with age, parity, and tamoxifen use, while increases in breast density tend to occur when body mass index (BMI) decreases, during pregnancy, or with hormonal replacement therapy.^1–3 Dense breast tissue can mask underlying cancers, making mammography less sensitive.⁴ Mammographic breast density is also an established independent risk factor for breast cancer, with women with dense breasts being at higher risk for breast cancer than women at otherwise similar risk levels who have less dense breasts.⁵

The most commonly used breast density assessment method in the clinical setting is visual grading, wherein breast density is classified by interpreting radiologists based on the American College of Radiology (ACR) Breast Imaging Reporting and Data System (BI-RADS^Ⓡ).⁶ The BI-RADS^Ⓡ density classification includes 4 categories of breast density (a, almost entirely fatty; b, scattered areas of fibroglandular density; c, heterogeneously dense, which may obscure small masses; and d, extremely dense, which lowers the sensitivity of mammography). Higher breast density categories have been consistently associated with increasing levels of breast cancer risk.^7,8 Moreover, inclusion of breast density in clinical risk prediction models, such as the Tyrer-Cuzick model, has been shown to improve predictive accuracy.⁹ However, BI-RADS^Ⓡ density classification is a subjective process with substantial inter- or intra-radiologist discrepancies¹⁰ that may have critical implications for a woman’s downstream care. In particular, inconsistent density assessments over time can lead to women receiving contrary recommendations for supplemental imaging with ultrasound or MRI. Furthermore, discrepancies in density assessments over time may interfere with the identification of longitudinal density trajectories associated with increased levels of breast cancer risk.^11–13

To enhance reproducibility and robustness in breast density assessment, various artificial intelligence (AI) models have been proposed to automatically classify mammographic images into BI-RADS^Ⓡ density categories.^14–19 Despite their promising performances,²⁰ so far breast density AI models have been assessed primarily in studies using a single examination for each woman. Therefore, their potential to enhance consistency in breast density assessment over time remains largely unexplored.

This study aims to address this question by using a large, multi-site mammographic screening cohort to assess whether use of a commercially available breast density AI model¹⁷ (WRDensity v1.1; Whiterabbit.ai, Santa Clara, CA) could result in more longitudinally consistent BI-RADS^Ⓡ breast density assessments for women compared with interpreting radiologists.

Methods

Study dataset

In this institutional review board-approved, Health Insurance Portability and Protection Act-compliant study under a waiver of consent, we retrospectively analysed a cross-sectional cohort of women who underwent at least 3 breast cancer screening examinations between January 1, 2016 and June 11, 2021 at an outpatient radiology practice (Onsite Women’s Health), using 50 sites across the United States (Figure 1). All women with breast implants were excluded. Examinations that were not acquired on Hologic imaging systems (Selenia Dimensions; Hologic, Inc.) or that did not have at least 1 mediolateral oblique (MLO) and 1 craniocaudal (CC) view per breast were also excluded. All screening examinations were performed with a combination of digital breast tomosynthesis (DBT) with digital mammography (DM) or 2D synthetic mammography (SM) images.

Images from standard acquisition angles (CC and MLO) obtained at mammographic screening examinations were used in our analysis. Age at screening was available for all examinations in electronic medical records. BMI at the time of each screening examination was also retrieved from the electronic medical record; when unavailable, BMI was approximated using image-based measures of breast fat and thickness (see Supplemental Materials).

Breast density assessment

The screening examinations in our dataset were interpreted by 1 of 39 board-certified radiologists who specialized in breast imaging and who had between 2 and 37 years of experience (36% with fellowship training). BI-RADS^Ⓡ density categories, assigned to each screening examination by the interpreting radiologist, were extracted from a structured mammography information system. The screening examinations were re-analysed for breast density using an AI model that provides an ACR BI-RADS^Ⓡ breast density category from DM or SM images. The AI density model was previously developed using deep learning based on over 600 000 DM and SM images from a large academic breast cancer screening practice.¹⁷ Screening examinations with indications of failed processing by the AI density model were excluded (Figure 1).

Statistical analysis

Longitudinal breast density patterns were characterized for all women as either constant, descending, ascending, or bi-directional, using 4-category breast density (BI-RADS^Ⓡ categories a-d) and binary breast density assessments (non-dense: BI-RADS^Ⓡ a and b; dense: BI-RADS^Ⓡ c and d). Given the short time window of our dataset (5 years), constant longitudinal patterns in breast density are the most physiologically likely to occur, while bi-directional longitudinal patterns in breast density are likely due to intra- and inter-reader variability for radiologists or limited robustness of the AI model. Also, Boyd et al.²¹ previously showed small decreases in area percent breast density over a 5-year period, especially for women not experiencing menopause. Therefore, our study focused primarily on constant and bi-directional longitudinal breast density patterns, and differences between the AI density model and interpreting radiologists were assessed using McNemar paired proportion hypothesis testing.²² In the main analysis, we assessed differences in the overall dataset, and in sub-analyses, we assessed differences in 4 overlapping subgroups of our dataset consisting of (1) post-menopausal women (women older than 55 years at first examination in the dataset), (2) women with stable image-based BMI (ie, proportional change of BMI <5%²³) between examinations (see Supplemental Materials), (3) women with examinations interpreted by the same radiologist, and (4) women with examinations interpreted by fellowship-trained radiologists. All statistical analyses were performed by using software (Python 3.8, python.org; Stata 17; StataCorp, College Station, TX, United States), and a P-value of .05 or less indicated statistical significance. Because of the large sample size, we had substantial power to detect differences in longitudinal breast density patterns in the main and subgroup analyses.

Results

Our study dataset was composed of 214 158 screening examinations (92.9% DM; 7.1% SM) from 61 177 women (mean age, 55.6 years; standard deviation, 10.3 years) (Table 1). A total of 37 156 women had 3 screening examinations (60.7% of our dataset), 17 618 women had 4 screening examinations (28.8%), and 6403 women had 5 or 6 screening examinations (10.5%). The overall BI-RADS^Ⓡ breast density distributions across all examinations exhibit small but detectable differences (P < .001) for the AI density model (a: 9.5%, b: 48.7%, c: 36.1%, d: 5.7%) and the interpreting radiologists (a: 9.0%, b: 45.1%, c: 38.1%, d: 7.7%).

The AI density model produced 10.4% fewer bi-directional (P < .001) and 24.2% more constant (P < .001) longitudinal patterns in 4-category breast density compared to interpreting radiologists (Figure 2A). The AI density model also produced 6.3% fewer bi-directional (P < .001) and 13.8% more constant (P < .001) longitudinal patterns for binary breast density (Figure 2B). Examples of longitudinal breast density patterns detected by the radiologists and the AI density model are shown in Figures 3 and 4. For many women, the interpreting radiologists and the AI model agreed upon a constant longitudinal pattern of breast density in both 4-category and binary density contexts (Figure 5); however, most disagreements occurred when the AI model identified a constant longitudinal density pattern and interpreting radiologists did not (Figure 5). Detailed changes in the proportions of the 4 breast density categories and in the proportion of dense vs non-dense breasts across consecutive screenings are shown in Figure 6.

Similar conclusions held for all subgroups of our cohort, with the AI model consistently producing fewer bi-directional and more constant longitudinal density patterns compared to interpreting radiologists (Tables 2 and 3). Interestingly, when examinations were reviewed by the same radiologist, by fellowship-trained radiologists, or when focusing on women with anticipated minimal changes in their breast parenchymal patterns over the 5-year window of our study (ie, women with stable image-based BMI), the interpreting radiologists produced fewer bi-directional and more constant longitudinal density patterns relative to their performance on the full dataset, yet still demonstrated less consistency compared to the AI model for these subgroups (Tables 2 and 3).

Discussion

The potential of AI to enhance consistency in clinical breast density assessment over time is largely unexplored. Our study addressed this question by assessing whether use of a previously validated AI model¹⁷ could result in more longitudinally consistent BI-RADS^Ⓡ breast density assessments for women compared with interpreting radiologists. Our data from a large, multi-site screening cohort of over 61 000 women, each with 3-6 mammographic screening examinations acquired over a 5-year period, showed that the AI model provides more longitudinally consistent breast density assessments compared to radiologists. This was seen with both 4-category breast density and binary breast density (dense vs non-dense) assessments, where, compared to radiologists, the AI model produced significantly more constant breast density patterns over time (4-category breast density: 81.0% vs 56.8%, P < .001; binary breast density: 89.9% vs 76.1%, P < .001) and significantly fewer bi-directional breast density patterns over time (4-category breast density: 4.9% vs 15.3%, P < .001; binary breast density: 2.5% vs 8.8%, P < .001). These same findings held in various sub-analyses focusing on examinations reviewed by the same radiologist/fellowship-trained radiologists or focusing on women with anticipated minimal changes in their breast parenchymal patterns over the 5-year window of our study (ie, post-menopausal women and women with stable image-based BMI). With a national breast density law in place²⁴ and breast density playing a key role in breast cancer risk assessment and supplemental screening recommendations,²⁵ having more consistent breast density assessment over time could lead to more consistent downstream care, in particular for dense-breasted and high-risk women.

Besides its substantial clinical impact, our study could also benefit future research on longitudinal breast density changes and their associations with breast cancer risk. Previous studies^11,13,26,27 have consistently shown that longitudinal changes in BI-RADS^Ⓡ breast density are more strongly associated with breast cancer risk than a single measure of BI-RADS^Ⓡ breast density, as well as the fact that a reduction in BI-RADS^Ⓡ breast density to a lower category reduces breast cancer risk relative to a density that remains stable or increases. However, the limited ability of these studies to decouple actual longitudinal density changes from longitudinal density changes due to inter- or intra-radiologist discrepancies^10,28 is a major drawback. This drawback could be addressed with more longitudinally consistent BI-RADS^Ⓡ breast density assessments provided by AI models, shedding more light into the identification of longitudinal density trajectories associated with increased levels of breast cancer risk.

A strength of our study was access to a large dataset, with a wide variety of clinics and interpreting radiologists, while being homogeneous in vendor and imaging system (Selenia Dimensions; Hologic, Inc.). Another major strength of our study was the use of a well-validated breast density AI model that supports both DM and the newer SM format acquired with DBT, which allows generalizability of our results to the current standard of breast cancer screening in the United States.²⁹ Finally, in assessing consistency in longitudinal breast density assessment in both 4-category and binary breast density (dense vs non-dense) contexts, our study offers preliminary evidence about potential critical implications of inconsistent density assessments over time for a woman’s downstream care.

Certain limitations should also be noted. We did not have access to more extensive demographic data, such as self-reported race/ethnicity. Therefore, although estimation from clinic zip codes suggests that our study cohort was racially diverse (White: 71%; Black: 21%; Asian: 4%; Other: 4%), we could not stratify our analyses by race. Moreover, we did not have universal access to BMI collected during attendance at screening or individual breast cancer risk assessments. To partially address this limitation, we approximated BMI using image-based breast fat and thickness data, previously shown to provide a suitable alternative to clinically acquired weight and BMI.^30–32 Last, compared to DM, the SM portion of our dataset was much smaller. In future studies, we aim to validate our findings in larger screening cohorts with SM. Future evaluations will also involve stratifications by race, as well as inclusion of BMI and other established breast cancer risk factors towards studying potential implications of differences in longitudinal breast density patterns on women’s risk trajectories.

Maintaining consistent AI performance over diverse patient populations, varying imaging settings, and long time horizons is a major challenge across several imaging AI applications, including AI models for mammography. For instance, previous studies have shown substantial effects of patient characteristics on AI performance in breast cancer detection with DM and DBT,^33–35 variable performances of different AI software for screening mammography,³⁶ as well as AI temporal quality degradation in dynamic clinical settings that involve multiple vendors, imaging systems, and AI models.³⁷ Therefore, our future work will focus on assessing the durability of our findings over diverse patient populations, multiple mammographic imaging vendors/systems, and advancements in AI models that provide BI-RADS or continuous breast density metrics. Future prospective studies on clinically used breast density AI models could also elucidate whether interpreting radiologists would more consistently assess breast density over time with assistance of AI, which was not tested in our study.

In conclusion, in a screening cohort of over 61 000 women, each with multiple mammographic screening examinations of a single vendor acquired over a 5-year period, the breast density AI model produced more longitudinally consistent BI-RADS^Ⓡ breast density assessments for women compared with interpreting radiologists, with fewer bi-directional and more constant assessments over time. Our results extend the advantages of AI in BI-RADS^Ⓡ breast density evaluation beyond automation and reproducibility,²⁰ showing a potential path to improved longitudinal consistency and more consistent downstream care for screened women.

Supplementary material

Supplementary material is available at BJR|Artificial Intelligence online.

Funding

Support for this study was provided by the Alvin J. Siteman Cancer Center through the Foundation for Barnes Jewish Hospital, the Fashion Footwear Charitable Foundation of New York, Inc., the Barnard Trust, the Susan G. Komen Foundation (CCR231011994), the Department of Defense (HT94252410101), and the National Cancer Institute of the National Institutes of Health (R01CA286120). This work was also partially supported by funding from Whiterabbit.ai, Inc. via a research contract (principal investigator: A.G.).

Conflicts of interest

Washington University in St. Louis (WU) has equity interests in Whiterabbit.ai, Inc. and may receive royalty income and milestone payments from a “Collaboration and License Agreement” with Whiterabbit.ai, Inc. to develop a technology evaluated in this research. These agreements are managed by the WU Institutional COI Committee. The following authors analysed and controlled the data in this work: E.E.T., A.G., D.C., T.P.M., and R.M.H. In addition, the following authors are employed by and/or have equity interests in Whiterabbit.ai, Inc.: D.C., T.P.M., and R.M.H.

References