Machine learning–based gait analysis to predict clinical frailty scale in elderly patients with heart failure

Abstract

Aims

Although frailty assessment is recommended for guiding treatment strategies and outcome prediction in elderly patients with heart failure (HF), most frailty scales are subjective, and the scores vary among raters. We sought to develop a machine learning–based automatic rating method/system/model of the clinical frailty scale (CFS) for patients with HF.

Methods and results

We prospectively examined 417 elderly (≥75 years) with symptomatic chronic HF patients from 7 centres between January 2019 and October 2023. The patients were divided into derivation (n = 194) and validation (n = 223) cohorts. We obtained body-tracking motion data using a deep learning–based pose estimation library, on a smartphone camera. Predicted CFS was calculated from 128 key features, including gait parameters, using the light gradient boosting machine (LightGBM) model. To evaluate the performance of this model, we calculated Cohen’s weighted kappa (CWK) and intraclass correlation coefficient (ICC) between the predicted and actual CFSs. In the derivation and validation datasets, the LightGBM models showed excellent agreements between the actual and predicted CFSs [CWK 0.866, 95% confidence interval (CI) 0.807–0.911; ICC 0.866, 95% CI 0.827–0.898; CWK 0.812, 95% CI 0.752–0.868; ICC 0.813, 95% CI 0.761–0.854, respectively]. During a median follow-up period of 391 (inter-quartile range 273–617) days, the higher predicted CFS was independently associated with a higher risk of all-cause death (hazard ratio 1.60, 95% CI 1.02–2.50) after adjusting for significant prognostic covariates.

Conclusion

Machine learning–based algorithms of automatically CFS rating are feasible, and the predicted CFS is associated with the risk of all-cause death in elderly patients with HF.

Graphical abstract

Open in new tabDownload slide

Introduction

Heart failure (HF) is increasingly common worldwide despite the advances in medical therapy and devices, leading to a social problem of a HF pandemic.¹ It is also a leading cause of hospitalization associated with poor clinical outcomes and high medical costs.^2,3

Frailty is a geriatric syndrome with vulnerability to stressors caused by the accumulation of various physiological deficits.^4,5 Many studies have shown a relationship between frailty and HF. For instance, 15–51% of patients with HF meet the diagnostic criteria for frailty, and the presence of HF also promotes frailty.^6,7 Notably, frailty is a poor prognostic factor in patients with HF. In older adults (≥70 years of age) with HF, frailty assessed by the Cardiovascular Health Study definition for frailty phenotype (Fried’s criteria) was independently associated with 1-year mortality [hazard ratio (HR) 2.13, 95% confidence interval (CI) 1.07–4.23].^8,9 Other studies also showed that frailty assessed by the Short Physical Performance Battery was significantly associated with 15 months mortality.^10,11

One of the most widely used frailty assessment tools is the clinical frailty scale (CFS).^12,13 The advantage of the CFS is that it is easy to use, provides an intuitive assessment, and requires no additional cost or effort for assessment. In addition, the frailty index (FI) can also be used for the assessment of frailty.¹⁴ As with the CFS, the FI is also associated with clinical outcomes in patients with HF.¹⁵ Nevertheless, it is necessary to evaluate at least 70 items in the FI, leading to a complexity of the frailty assessment. Although it is worthwhile to assess frailty in patients with HF, there is no consensus regarding how frailty should be measured. Therefore, this lack of consensus may be a major reason for not assessing frailty in clinical practice. In fact, the concordance rate between several frailty assessment tools is not sufficient.^12,16 Furthermore, there are probably several inter-rater value errors, even in commonly used tools such as the CFS. The concordance rate of CFS assessment among specialists was reported to be high; however, it was considerably low between specialists and residents.^17–19 These discordances and errors of commonly used frailty assessment tools could be caused by their subjective characteristics; therefore, the development of emerging objective tools for the assessment of frailty is warranted.

In the last decade, deep learning and machine learning techniques have shown promising image and video analysis capabilities in the field of medicine.^20–22 Concerning motion analysis, these techniques can quickly and cost-effectively quantify and provide objective parameters of human movement such as gait. Gait analysis, including gait speed, gait width, and step width, has been used to evaluate patients’ frailty.^9,14,23 Previous studies have shown that these parameters can be assessed by walking video analysis.^24,25 However, there might be many more features that have not been exhaustively explored to characterize the gait and thus to predict the patients’ general condition. Accordingly, we aimed to develop and explore the feasibility of a machine learning–based automatic rating system for the CFS by focusing on walking video analysis in patients with HF.

Methods

Study design

This was a multi-centre (seven centres), observational, prospective study that included patients aged ≥75 years who were diagnosed with symptomatic chronic HF based on the diagnostic criteria from the current HF clinical guidelines.^26,27 Patients with septic shock, acute coronary syndrome, acute pericarditis, receiving a left ventricular assist device or heart transplantation, open-heart surgery scheduled within 1 month, and the inability to walk independently were excluded. The included patients were divided into derivation (those from centre A: Hokkaido University Hospital) and validation (those from centre B: Japan Community Healthcare Organization Hokkaido Hospital, centre C: Japanese Red Cross Kitami Hospital, centre D: Otaru Kyokai Hospital, centre E: Sunagawa City Medical Centre, centre F: Hakodate National Hospital, and centre G: Kushiro City General Hospital) cohorts. This study was approved by the Institutional Review Board of each centre and is registered under the Japanese UMIN Clinical Trials Registration (UMIN000043390). The study conformed to the principles outlined in the Declaration of Helsinki. All the patients provided written informed consent to participate in the study.

Study population

We examined 424 consecutive patients (derivation cohort, 194 patients; validation cohort, 230 patients) between January 2020 and October 2023. Patients whose walking parameters could not be appropriately obtained from the video in the validation cohort (n = 7) were excluded. Ultimately, 194 patients in the derivation cohort and 223 in the validation cohort were examined in this study (Figure 1).

Clinical frailty scale rating

In general, the CFS consists of the following ratings: 1 (very fit), 2 (well), 3 (managing well), 4 (vulnerable), 5 (mildly frail), 6 (moderately frail), 7 (severely frail), 8 (very severely frail), and 9 (terminally ill).¹³ Nevertheless, CFS 1 and 2 are rated in patients with asymptomatic disease, and CFS 7 and over are rated in patients with an inability to walk without the assistance of caregivers.¹³ Thus, in the present study, all patients were evaluated as having CFS 3–6 because our study included only symptomatic patients with HF and excluded those who could not walk without the assistance of caregivers. We referred the CFS assessment, based on the obtained walking videos and clinical information, to 10 trained cardiologists certified by the Japanese Circulation Society (JCS). To minimize the inter-observer variability of CFS rating, the CFS evaluations by each rater were integrated using the modified Delphi method, which is a useful approach for the aggregation of varied opinions.²⁸ We adopted their final assessment as the actual CFS. To compare the machine learning model with human raters, we selected three independent non-trained cardiologists who were post-graduate students without JCS certification.

Development of gait analysis system

The gait analysis system begins with camera calibration, followed by enabling the extraction of key points from the human body and anomaly detection. Furthermore, after the extraction and selection of numerical parameters from the time series data of key points, the system enables the derivation of the final features for CFS prediction.

We developed an L-shaped walking course to obtain the patients’ gait parameters, as shown in Figure 2. Two blue markers were placed on the floor as distance reference points. The protocol for the walking video recording was as follows (see Supplementary material online, Video S1). After the patient sat on a chair, we started recording. The patient stood up from the chair and walked along the yellow line, turning to the right at the first marker (2 m away) and walked continuously, stopping at the second marker (further 2 m away), and we stopped the recording. Through this system, we could obtain ∼10–20 s of walking video, including a lateral view of the walk, turning motion, and frontal view of the walk. We recorded the walking video using a single smartphone camera (iPod touch®, 7th generation, Apple Inc., USA, memory capacity 128 GB, 30 frames per second at 1080p resolution), which was placed 4 m horizontally away from the second marker and 1 m vertically away from the floor.

The locations of the two markers were recognized using a single-shot multi-box detector (SSD), which is a deep learning–based object recognition technology.²⁹ Specifically, the SSD was pre-trained on ImageNet, using the transfer learning technique, to recognize the marker appearances. The distance between the chair and first marker was 2 m. Camera calibration was performed according to the location of the marker.

All walking videos were converted to keypoint data by OpenPose®, a deep learning–based pose estimation library.³⁰ OpenPose® is a deep learning–based keypoint detection algorithm for the human body. The keypoint data include the location and accuracy of the pose estimation for each frame of an input video. We used the BODY_25 model, which tracks the following 25 key points: the nose, neck, midhip, both sides of the shoulder, elbow, wrist, hip, knee, ankle, eye, ear, big toe, small toe, and heel. The original output data from OpenPose® are the estimation results for every frame; thus, the before/after frame relationship is not considered. Therefore, there are several misdetections in the left-foot and right-foot estimations. To solve this problem, an anomaly detection algorithm was used to detect the misestimations, and the data were corrected using spline interpolation from the previous and following values (see Supplementary material online, Figure S1). Specifically, the frame that includes the joint position change, which is >50 pixels in a positive direction or 35 pixels in a negative direction from the previous location, was defined as an abnormal frame. The location data of the bilateral foot (i.e. locations of the hip, knee, ankle, big toe, small toe, and heel) were deleted in the abnormal frame, and spline interpolation from the previous and following values was performed. From the locations of the 25 points and 2 floor markers, 123 features were calculated after camera calibration, and 5 features (age, sex, height, weight, and use of walking sticks) were added from the clinical information. Ultimately, 128 features were used for model training (see Supplementary material online, Table S1).

Model development

In the derivation cohort, we used 128 features and developed a machine learning–based automatic rating system using the light gradient boosting machine (LightGBM) regressor model based on the extremely randomized trees model.^31,32 The model development process is shown in Figure 1. First, we constructed a preliminary model using the derivation dataset to calculate the SHapley Additive exPlanations (SHAP) values.³³ SHAP values >0 indicate a positive association and <0 indicate a negative association with frailty. The mean absolute SHAP value of each feature was used as the variable of importance. Second, the derivation datasets were split into 80% training and 20% test sets, and five-fold cross-validation was performed to estimate the temporary reliabilities between the actual and predicted CFSs. Third, feature selection was performed using Optuna®, an open-source hyper-parameter optimization framework (Preferred Networks, Inc., Tokyo, Japan). After the first hyper-parameter optimization, five-fold cross-validation and hyper-parameter optimization were repeated 50 times to explore the best feature selection. Finally, the LightGBM model was trained using the best-selected features in the derivation dataset and was tested in the validation dataset. To validate the internal reliabilities, the internal dataset was split into 90% training (as an internal dataset) and 10% test sets (as an external dataset), and the internal reliabilities were then obtained by 10-fold cross-validation. The LightGBM model output is a continuous value [frailty level (FL)], and the predicted CFS is the integer value of the FL. All estimation models were built using Python 3.6.

To protect patient privacy, we developed an offline system from video capture to CFS prediction, because the walking videos included patient privacy. The camera and small portable computer (Jetson AGX Xavier^TM, NVIDIA®, USA) were connected with a wired cable, and the walking videos were uploaded from the camera to a portable computer without the internet. All processing (camera calibration, OpenPose®, anomaly detection algorithm, SSD, deletion of patient’s walking videos after converting the video to a key point data, and the LightGBM model for predicting the CFS) were performed on this computer, at each centre. Although the present study was a multi-centre study, patients’ personal information was not disclosed outside the study centres; thus, patient privacy was protected.

Outcome measures

The primary outcome of interest was the agreement between the actual and predicted CFSs. The secondary outcome of interest was all-cause death stratified by the predicted CFS and FL.

Statistical analysis

Continuous variables are presented as mean ± standard deviation when normally distributed and as median and inter-quartile range (IQR) when non-normally distributed. The differences between the two groups were compared using unpaired t-test or Mann–Whitney U-test for continuous variables and using χ² test or Fisher’s exact test for dichotomous variables, when appropriate. The agreement between the actual and predicted CFSs was represented by Cohen’s weighted kappa and intraclass correlation coefficient (ICC), and 95% CIs were calculated using the bootstrap method. According to the magnitude of weighted kappa and ICC, the interpretation was as follows: ≤0.20, none; 0.21–0.39, minimal; 0.40–0.59, weak; 0.60–0.79, moderate; 0.80–0.90, strong; and >0.90, almost perfect.³⁴ Cumulative incidence curves for all-cause death were generated to assess the prognostic significance of the predicted CFS. To evaluate the influence of the predicted CFS on all-cause death, we constructed four multivariable Cox proportional hazard models [Model 1: adjusted for age and sex; Model 2: adjusted for the Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC) risk score; Model 3: adjusted for the MAGGIC risk score and N-terminal pro-brain natriuretic peptide (NT-pro BNP); and Model 4: adjusted for the MAGGIC risk score, NT-pro BNP, and serum albumin]. The MAGGIC risk score was calculated as follows.³⁵ Total points were calculated based on age (every 10 years), male sex, body mass index (every 1 kg/m² increase, up to 30 kg/m²), current smoking status, diabetes, systolic blood pressure (every 10 mmHg increase), New York Heart Association functional class, left ventricular ejection fraction (LVEF; every 5% increase, up to 40%), chronic obstructive pulmonary disease, HF duration >18 months, creatinine (10 up to 350 μmol/L), angiotensin-converting enzyme inhibitor/angiotensin II receptor blocker use, and beta-blocker use. A simple integer score was derived with a maximum score of 57. We tested the proportional-hazards assumption on the basis of Schoenfeld residuals after fitting Cox regression models. All tests were two-tailed, and a P-value <0.05 was considered statistically significant. All analyses were performed using STATA® MP64 version 16 (StataCorp, College Station, TX, USA).

Results

Baseline characteristics and gait parameters

Baseline characteristics of 194 patients from the derivation cohort and 223 patients from the validation cohort are summarized in Table 1 and Supplementary material online, Table S2. Age was not significantly different (mean 82.3 vs. 82.5 years) between the derivation and validation cohorts. Patients from the derivation cohort were predominantly females [111 (57.2%) vs. 83 (37.2%)] compared with those from the validation cohort. Patients from the derivation cohort had a lower prevalence of ischaemic heart disease [23 (11.9%) vs. 61 (27.6%)], atrial fibrillation [94 (48.5%) vs. 130 (59.4%)], and smoking [68 (35.1%) vs. 118 (54.1%)]; lower haemoglobin, sodium, and creatinine levels; and higher NT-pro BNP level than those from the validation cohort. Regarding medications, patients from the derivation cohort were less frequently prescribed diuretics [117 (60.3%) vs. 166 (75.5%)], sodium-glucose transport protein-2 inhibitors [21 (10.8%) vs. 42 (19.1%)], and mineralocorticoid receptor antagonists [62 (32.0%) vs. 100 (45.5%)] compared with those from the validation cohort.

The gait parameters obtained using the proposed application are listed in Table 2. Patients with higher CFS had higher total gait time, cornering time, stance phase, and prevalence of using walking sticks and lower gait speed, gait width, ankle swing speed and momentum, and wrist swing speed and momentum than those with lower CFS (Figure 3A–F).

Model performances

In the preliminary training, 45 parameters were selected according to the SHAP values to calculate the FL and estimate the CFS (see Supplementary material online, Figure S2). Fifteen most important features that the model used for prediction are depicted in Figure 4. In the derivation dataset, the LightGBM model showed an excellent agreement between the actual and predicted CFSs (weighted kappa 0.866, 95% CI 0.807–0.911; ICC 0.866, 95% CI 0.827–0.898; Figure 5A and Table 3). In the validation dataset, the LightGBM model also showed good agreement between the actual and predicted CFSs (weighted kappa 0.812, 95% CI 0.752–0.868; ICC 0.813, 95% CI 0.761–0.854; Figure 5B and Table 3). The LightGBM model achieved a higher agreement rate than the non-trained cardiologists (Table 3). None of the patients had a two-stage or over-estimation error in the derivation and validation datasets.

Survival analysis

During a median follow-up period of 391 (IQR 273–617) days, all-cause death occurred in 33 patients. Multivariable Cox regression analyses revealed that higher predicted CFS and FL were independently associated with all-cause death, even after adjusting for prognostic models (Table 4).

Discussion

In this prospective study, we developed a machine learning–based automatic rating system to evaluate the CFS in elderly patients with HF. In addition, we evaluated the feasibility of the rating system by comparing it with the contemporary CFS rating system provided by trained cardiologists. We found that our automatic rating system using the LightGBM model has a strong reliability in comparison with trained cardiologists. Furthermore, the system proved to be as useful as a non-trained cardiologist for evaluating the CFS. We also found that higher predicted CFS and FL were independently associated with a risk of all-cause death, even after adjusting for prognostic covariates and a mortality prediction model.

In terms of the predicted CFS, our LightGBM model showed a strong agreement with the gold standard trained cardiologists (derivation cohort: weighted kappa 0.866 and ICC 0.866; validation cohort: weighted kappa 0.812 and ICC 0.813). Interestingly, these values were higher than those of the non-trained cardiologists (derivation cohort: weighted kappa 0.740–0.867 and ICC 0.740–0.868; validation cohort: weighted kappa 0.675–0.777 and ICC 0.669–0.774). Overall, our results indicate a good performance of the LightGBM model automatic rating system for estimating the CFS. One of the reasons for achieving good reliability of the predicted CFS was that we improved the training data (i.e. actual CFS) as much as possible using the modified Delphi method. The Delphi method is commonly used to obtain broad consensus among experts by determining the level of agreement on a given topic.³⁶ To date, the definition of frailty is not clear and can differ in many aspects of developmental background, detailed components, and phenotype. Therefore, the Delphi method is suitable for developing high-quality training data. Notably, the validity of the content of a Delphi consensus will depend on the appropriate choice of the participating experts.³⁷ Experts who participated in the study were well experienced in the field of cardiovascular medicine and were certified by the JCS. In addition, the level of participation was high: all the experts participated in each round, which supports the validity of the study as well as the final results.

For the development of the CFS automatic rating system, we found 15 key walking parameters, including gait speed, total gait time, ankle swing speed, and cornering time, based on the Shapley analysis without special cameras and/or body markers. Previous studies have shown that walking parameters can be analysed from the walking videos using machine learning models.^24,38,39 However, in these studies, some special cameras and/or body markers were required; thus, the machine learning system is costly and available only in limited facilities. On the other hand, Vafadar et al.²⁵ developed a marker-less system using a deep learning–based human pose tracking system that can estimate the spatiotemporal gait parameters within the range of the minimum detectable changes obtained using the marker-based reference system with special cameras. Furthermore, Park et al.²² developed a machine learning–based automated rating system for the two cardinal symptoms of Parkinson’s disease (i.e. resting tremor and bradykinesia) and confirmed the high reliability of their system compared with that of the manual gold standard ratings given by trained specialists.

In the present study, higher predicted CFS and FL were independently associated with a higher risk of all-cause death, even after adjusting for key prognostic confounders and a representative mortality prediction model. Several previous studies showed that frailty was a powerful predictor of morbidity and mortality, regardless of the frailty tool used (e.g. CFS, the Fried criteria, timed get-up-and-go test) and independent of age, comorbidities, HF symptoms, and severity.^40–43 Sze et al.⁴³ also found that the risk of hospital admissions and death increases as frailty status worsens. Given these information, the guidelines note that the treatment of frailty in HF should be multi-factorial and targeted to its main components and may include physical rehabilitation with exercise training and nutritional supplementation.²⁶ A supervised, exercise-based, cardiac rehabilitation programme is also recommended in patients with HF and frailty.²⁶ Most of the previous frailty indicators are based on classifications. Our automatic rating system can output not only the predicted CFS as a categorical value but also the FL as a continuous value. Evaluating frailty with continuous values may detect trajectories in frailty in a more sensitive manner than with classification indicators. Furthermore, the FL as detailed scores of the CFS (continuous value) would make it easier to compare before and after clinical intervention such as physical rehabilitation and nutritional supplementation.

There are several potential limitations of the present study that should be acknowledged. First, the walking video clips are two-dimensional images of the three-dimensional gait motion; thus, there may be several technical errors. Second, the actual CFS was determined by 10 trained cardiologists using the modified Delphi method. Because the Delphi method converges opinions based on majority voting, the results may be influenced by the tendencies and values of the evaluators. Third, the sample size used in our analyses, particularly the number of patients rated as CFS/LightGBM 5 or 6, was relatively small. Thus, the accuracy of the system may be limited. In addition, there were several differences in baseline patient characteristics between the derivation and validation cohorts. Fourth, our current methodology requires ∼5 min for estimating the FL and may be time consuming. Further studies developing an application that allows all processes, including walking video recording, prediction of the FL, and presenting the results output on a smartphone device, are required. Finally, owing to the relatively short-term follow-up (median 391 days), the association with long-term prognosis could not be investigated. Therefore, a prospective study with larger sample size is warranted to confirm the clinical implications of our CFS automatic rating system.

Conclusions

In this study, we verified the feasibility of using a machine learning, vision-based automated rating system for estimating the CFS in patients with HF. We also confirmed the high reliability of this system, which was comparable with that of manual ratings by trained cardiologists. Furthermore, the automatically rated CFS was associated with the risk of all-cause death. The expansion into a more distributive system, confirmation of the prognostic implication of the automatically rated CFS, and its changes during clinical intervention such as physical rehabilitation would allow its use in clinical practice.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health.

Acknowledgements

The authors are grateful for the CFS labelling team (Masaya Watanabe, MD, PhD, Kiwamu Kamiya, MD, PhD, Taro Temma, MD, PhD, Takuma Sato, MD, Kazunori Omote, MD, PhD, Atsushi Tada, MD, PhD, Taro Koya, MD, PhD, Suguru Ishizaka, MD, PhD, Yuta Kobayashi, MD, PhD, Daishiro Tatsuta, MD, Yutaro Yasui, MD, Sho Kazui, MD, Jiro Koya, MD) and the contributions of all the investigators, clinical research co-ordinators, and data managers.

Funding

This work was supported by a grant from the Japan Agency for Medical Research and Development (22hma322007h0001, 23hma322007h0002, T.N.) and grants from the Hokkaido University Hospital (T.N.), the Fukuda Foundation for Medical Technology (T.N.), the Suzuken Memorial Foundation (T.N.), and the CASIO Science Promotion Foundation (T.N.).

Data availability

The data underlying this article cannot be shared publicly due to the privacy of individuals who participated in the study. The data will be shared on reasonable request with the corresponding author. No new data were generated or analysed in support of this research.

References

Author notes