Abstract
Preterm infants are at high risk of neuromotor disorders. Recent advances in digital technology and machine learning algorithms have enabled the tracking and recognition of anatomical key points of the human body. It remains unclear whether the proposed pose estimation model and the skeleton-based action recognition model for adult movement classification are applicable and accurate for infant motor assessment. Therefore, this study aimed to develop and validate an artificial intelligence (AI) model framework for movement recognition in full-term and preterm infants.
This observational study prospectively assessed 30 full-term infants and 54 preterm infants using the Alberta Infant Motor Scale (58 movements) from 4 to 18 months of age with their movements recorded by 5 video cameras simultaneously in a standardized clinical setup. The movement videos were annotated for the start/end times and presence of movements by 3 pediatric physical therapists. The annotated videos were used for the development and testing of an AI algorithm that consisted of a 17-point human pose estimation model and a skeleton-based action recognition model.
The infants contributed 153 sessions of Alberta Infant Motor Scale assessment that yielded 13,139 videos of movements for data processing. The intra and interrater reliabilities for movement annotation of videos by the therapists showed high agreements (88%–100%). Thirty-one of the 58 movements were selected for machine learning because of sufficient data samples and developmental significance. Using the annotated results as the standards, the AI algorithm showed satisfactory agreement in classifying the 31 movements (accuracy = 0.91, recall = 0.91, precision = 0.91, and F1 score = 0.91).
The AI algorithm was accurate in classifying 31 movements in full-term and preterm infants from 4 to 18 months of age in a standardized clinical setup.
The findings provide the basis for future refinement and validation of the algorithm on home videos to be a remote infant movement assessment.
Introduction
Early identification and intervention of infants who are at risk for or have developmental disorders is an important global health policy and action. The Global Research on Developmental Disabilities Collaborators of the United Nations reported the prevalence rate of any developmental disorders among children aged younger than 5 years as 8.4%. 1 Although there is a rare prevalence survey of developmental disabilities in Taiwan, the number of children with developmental disabilities referred for early intervention gradually increased from 16,167 in 2009 to 26,471 in 2019, with an incidence estimate of 3% to 5%.2 The annual data in Taiwan from 2014 to 2019 revealed that children younger than 2 years of age were less likely referred for early intervention (17.4%–26.6%) than those older than 2 years of age (73.4%–82.6%).2 Moreover, the referral for early intervention was mostly via hospitals (34.4%–45.2%) and was less likely via infant care centers (7.7%–11.9%) and by guardians (8.1%–10.0%). These reports have highlighted the urgent need of a feasible early developmental assessment in the community.
The potential impact of motor development on the emergence of abilities in other domains in children at an older age has already been demonstrated in the case of some key motor skills.3 Attainment of motor skills may afford an infant opportunities to explore the environment and hence to develop perceptual, motor, cognitive, social, and emotional behaviors.4 Accumulating evidence has supported the associations between early motor function and later cognitive, motor, and language outcomes.5,6 Furthermore, preterm infants account for 10% of newborns and are at risk for developmental disorders, with motor delay being the leading problem.7 Therefore, early motor assessment of high-risk infants such as preterm infants is essential to help identify those who will have developmental disorders.
Contemporary infant motor assessments are based on the concepts of developmental sequence, variability in attainment of milestones, and motor performance that entail the evaluation of an evolutionary process and qualitative changes of motor behaviors over time.8 Because of this predictability, the sequential pattern of motor development provides an observable way to evaluate motor skills and to detect motor delay.
Infant motor assessments consist of diagnostic and screening tests.9 Diagnostic tests are administered by health care professionals using standardized procedures and test stimuli that offer comparability of standard scores across age groups and settings.10,11 However, parents often do not have access to these tests beyond hospitals, and the short window of opportunity for predictive value passes by, resulting in a missed opportunity for early identification and treatment. For example, the examiner-administered Alberta Infant Motor Scale (AIMS)8 was found to be reliable12 and valid in classifying the longitudinal motor development of 342 preterm infants during the first year of age that predicted cognitive and motor outcome at 24 months.13 Nevertheless, their early AIMS scores were predictive of age of walking attainment until 9 months of age or older.14 Although screening tests are often based on parental observation that are inexpensive and easy to use, parents have varied in knowledge and interpretation of child development that the results tend to have good specificity but poor sensitivity.15 For examples, the parent-report measures of the Early Motor Questionnaire were found to have high correlations with the examiner-administered motor assessments in children aged 3 to 24 months.16 On the other hand, the maternal measures of the Ages and Stages Questionnaires showed overidentification of motor delay in full-term and preterm infants, particularly for those aged younger than 13 months, when compared with the diagnostic assessment results.17 The mothers’ AIMS scores for their preterm infants were found to be higher than the AIMS scores based on assessments by physical therapists.18 These challenges in the current evaluation system have called for the need to explore innovative approaches that will complement existing infant motor assessment with clinical feasibility and accuracy.
Digital technology has increasing penetration in modern societies and has expedited utilization in the health care system. In particular, artificial intelligence (AI) enhances learning capacity and provides decision support systems at scales that are transforming the future of health care.19 AI employs machine learning with artificial deep neural networks that mimic the operation of the human brain using multiple layers of artificial neuronal links to generate automated predictions from training datasets.20 Deep learning technology has received increased interest as an alternative for the recognition of symptoms, classification, diagnosis, and prediction of outcomes in clinical practice.21
To incorporate AI technology into clinical application, explicit model development and validation of potential algorithms are essential.22 Unlike the algorithms for medical image classification that involve still images, the algorithm for movement assessment must fulfill the purpose of movement tracking, recognition, and classification, which is much more complex in data processing and analysis. The recent deep high-resolution representation learning for human pose estimation (HRNet) for skeleton-based action tracking in adult individuals is an intriguing innovation.23 Furthermore, several works, such as the spatial temporal graph convolutional networks24 and PoseConv3D,25 have been developed for skeleton-based action recognition. Such movement tracking and classification technologies have novel applications for gait analysis in stroke survivors26 and movement assessment in infants.27 Reich et al28 recently applied OpenPose for motion capture from 2-dimensional (2D) red–green–blue images and a shallow multilayer neural network that accurately discriminated fidgety movements from nonfidgety movements in 51 full-term infants with typical development on the basis of biweekly video records from 4 to 16 weeks of age at the laboratory using the annotation results of experienced General Movement Assessment assessors as the gold standards (accuracy = 0.88). Whether the new AI models for human movement classification are applicable and accurate for movement tracking and recognition in infants aged older than 4 months through incremental machine learning on video recordings requires investigation.
To achieve the ultimate goal of using AI technology for remote infant motor assessment by caregivers at home, 2 steps of work are necessary. The first is to collect more clean and standard data for the model to learn as a foundation and the second is to transfer or fine-tune the foundation model on home videos. In our current work, that is, the first step, data are collected from a standardized clinical setup with the assistance from a physical therapist to engage and guide the infant to perform certain movements. This is to ensure that the collected data have high quality and cover more diverse movements, in contrast to the home setup where performed movements may not be diverse, and the data are highly likely to be of low quality due to camera shaking, improper shooting angle, and unintentional panning. Under controlled scenarios in a standardized clinical setup, occlusion by objects or adults is also minimized, whereas infants performing their movements at home are often surrounded by family members in variable backgrounds, leading to frequent object occlusion.
Furthermore, to build a good AI model, it is better to include more data with diverse coverage, hence, the more cameras the better. However, as more data require more time and resources to train, the 5-camera option is a reasonable and optimal number which could still achieve the diversity and coverage. A separation of 45 degrees between camera positions could capture more diverse views and help alleviate occlusion issues. For example, the infant’s legs are occluded by his body in 1 camera view (view 2), whereas his body parts are captured better in the remaining camera views (Suppl. Figure). The Nanyang Technological University red–green–blue + depth dataset is a popular benchmark dataset for action recognition that also employs a 5-camera setup.29,30
Therefore, this study aimed to use a deep machine learning approach to develop and validate an AI algorithm for early motor assessment in full-term and preterm infants in a standardized clinical setup using 5 camera views. The purpose of this study was 3-fold: to collect useful data from video recordings of the movements of full-term and preterm infants during the AIMS assessment from 4 to 18 months of age (corrected for prematurity) under a standardized clinical setup using 5 camera recordings; to establish a reliable procedure for movement annotation by pediatric physical therapists; and to develop a pose estimation and action recognition model framework that learned from the annotated movement video data. We consider this study an incremental research effort as to establish an AI model and machine learning for infant movement recognition in a standardized clinical setup that will be refined and validated for classifying infant data collected at home and ordinary clinical setup in the future.
Methods
Participants
This observational study recruited full-term and preterm infants aged 4 to 18 months from National Taiwan University Children’s Hospital, Taipei, Taiwan, from November 2020 to April 2022. Full-term infants were recruited from the well-baby follow-up clinic if their gestational age was 37 to 42 weeks, if their birth body weight was >2500 g, and if they had no congenital/genetic abnormalities. Preterm infants were recruited from the neonatal follow-up clinic if their gestational age was <37 weeks, if their birth body weight was <2500 g, and if they had no congenital/genetic abnormalities. The parents signed the consent form before participating in the study. This study was approved by the Human Rights Review Committee at the study hospital (202010031RINB and 202012089RINB).
The estimation of the sample size for the AI model was based on the experience of Shahroudy et al29 and followed the common practice in the development of machine learning algorithms to split the dataset randomly into training and validation sets with a ratio of 8:2 according to the Pareto principle.31 To consider establishing an red–green–blue + depth human action recognition model for the 58 movement items and the attrition rate of 20% in our follow-up study, the projected number of infants was estimated to be 125.
Data Collection
The infants were prospectively examined with the AIMS assessment at the laboratory bimonthly from 4 to 18 months of age until walking attainment. The parents were first interviewed for their child’s and sociodemographic data.
The infants were administered the AIMS assessment by 1 of 2 pediatric physical therapists (A and B) on a green mattress marked with a red-lined square (2 × 1.4 m) for play and motion capture (Fig. 1). Each infant was required to wear tight clothes with colors different from the mattress for better visibility. The assessment procedure was simultaneously recorded by 5 video cameras (ie, Samsung Galaxy A71, Samsung Electronics, Suwon, South Korea; Sony HDR CX-405, Sony Corporation, Minato City, Tokyo, Japan; Sony NEX VG-10, Sony Corporation, Minato City, Tokyo, Japan; Sony HDR CX-380, Sony Corporation, Minato City, Tokyo, Japan; and Samsung Galaxy A71, Samsung Electronics, Suwon, South Korea) installed on tripods surrounding the mattress at a height of 0.6 m and a distance of 1.5 m from the square (Fig. 2A). Multiple camera views were used for data augmentation and occlusion prevention in the estimation of 3-dimensional human poses from 2D poses.32 Although the model could benefit from more training data using more camera views, the challenges were an increase in the annotation cost and effort, and the computing power to process more data. Since some body parts may be occluded in video recordings, 5 cameras with different angles were optimal to prevent data loss.33 If 1 view contained occluded body parts, then the other views that revealed clear body parts could be used. A slating mechanism using a clapper board was employed in video recordings to synchronize the start and end times of 5 camera views, by creating a distinctive sound and visual signal. This was necessary because the cameras had different start times in recording. All video frames that lied before the start slating and after the end slating were trimmed in postprocessing.
Context setting for infant motor assessment in the laboratory.
(A) Camera setup and (B) joints legend for the infant key point detection model. d = distance
During the AIMS assessment, the therapist led the infant to move into various positions within the red-lined square, and 1 parent sat near the infant to interact with the child. The therapist encouraged the infant to perform the movements more than once to enhance the success of later movement capture, and the whole procedure took 30 minutes. In completion of the AIMS assessment, the therapist scored the individual items for the infant to obtain the subscale scores and total score that the latter was further converted into percentile score at corrected age according to the manual.8 Motor delay was defined when an infant’s percentile score was below the 10th percentile.
Measurement
The AIMS is a norm-referenced assessment that examines gross motor development in infants aged 0 to 18 months.8 The assessment contains 58 movements that emphasize the attainment of gross motor skills, postural alignment, weight bearing of the body, and antigravity movement of the limbs in prone (21 movements), supine (9 movements), sitting (12 movements), and standing (16 movements) positions. Each movement item is scored as 1 (passed) and 0 (failed). The item scores are added into the total score that is converted into percentile rank. The sequence and age at emergence of the 58 movements have been found to be similar in the past 20 years in Canadian infants.34
Both the AIMS examiners had 2 years of clinical experience and undertook a training course on the administration that required their scorings to achieve agreement with an experienced physical therapist’s (S.-F.J.) results with inconsistency of no more than 2 items in the total scale before participation in the study. Their interrater reliability on 16 infants was high for the total score (intraclass correlation coefficient [ICC] = 0.99; P < .05; standard error of measurement [SEM] = 0.53).
Data Processing
The data processing of infants’ movement videos consisted of the following 3 steps:
Selection of Movement Video Recordings
All the AIMS assessment videos with the 5 video recordings of the individual infant at 1 assessment age were aligned for the start and end times with the excess parts trimmed to ensure the same duration. To protect personal privacy, each infant was assigned an identification number in the files with the name blinded. A server with high-speed central processing units and graphics processing units was used for data processing and storage.
All edited movement videos were then annotated by 1 of 3 pediatric physical therapists (A, B, and C) using the labeling tool developed by our research team to annotate the start and end times and the presence of the movement. The therapists were required to preview the video file with 5 camera views and label the movements according to the AIMS scoring criteria and the labeling guidelines (Suppl. Table). In most videos, the number of movements noted was identical across the camera views. However, more movements were noted in certain camera views than in the others because the infant’s body parts occluded with an object or person. Furthermore, the movement was coded as “perfect presence” when the infant’s features met all the AIMS scoring criteria but was coded as “acceptable presence” when the infant’s features met some of the scoring criteria with the body alignment not distinctly observed from the view angle. For example, an infant was recorded creeping from the frontal view leading to difficulty in determining whether the lumbar posture was flat for “reciprocal creeping (1)” or was in lordosis for “reciprocal creeping (2).” In such a case, the movement was coded as “perfect presence” in “reciprocal creeping (1)” and “acceptable presence” in “reciprocal creeping (2).” “Acceptable presence” was only considered in labeling the following movement sets for their similarity in most of the features: forearm support (1) versus (2), 4-point kneeling (1) versus (2), reciprocal creeping (1) versus (2), and support standing (2) versus (3).
Three pediatric physical therapists (A, B, and C) served as the annotators who had 2, 2, and 20 years of clinical experience in pediatric care, respectively. The therapists underwent a training course requiring therapists B and C to achieve agreement of >0.85 with the experienced annotator’s (A) results in the overall AIMS assessment and the labeling of individual movements before participation in the study. The intrarater (ICC = 0.99; P < .05; SEM = 2) and interrater (ICCs = 0.98–0.99; all Ps < .05; SEMs = 2.58–2.83) reliabilities of the overall AIMS assessment over the video recordings of 3 infants at 4, 7, and 16 months of age, respectively, were excellent for the total score. The intrarater reliability for therapist A in annotating all the movements from the 5 camera views in the 3 infants was excellent (agreement = 100%; κ = 1; P < .05). The interrater reliability for the 3 therapists in annotating the movements of the 16-month-old infant from 5 camera views was excellent (agreement = 100%; κ = 1; P < .05). The interrater reliability for annotating the movements of the 4- and 7-month-old infants was excellent in 4 camera views (all agreements = 100%; κ = 1; P < .05) and good in 1 camera view (agreements = 87.8% and 90.9%; κ = 0.62 and 0.63; P < .05).
Establishment of the Pose Estimation Model
The anatomic information is crucial for determining the infant’s movement. Therefore, a pose estimator was developed to predict the skeletal key points. The predicted key points were then fed to the movement recognition model as input. The annotated video files were sliced on the basis of the annotation of the start and end frames, and then the HRNet model23 was used as the infant key point detection model. The input to the model was a 2D red–green–blue image (ie, from the frame of a video). The output consisted of key points with their respective anatomic locations, encompassing 17 joints following the Microsoft Common Objects in Context (Microsoft Corp, Redmond, WA) described by Lin et al35: nose, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, left knee, left ankle, right hip, right knee, right ankle, left eye, right eye, left ear, and right ear (Fig. 2B). We fine-tuned our model using a small amount of annotated infant images to gain higher performance. Instead of training from scratch, the model may benefit from learning on an adult dataset, and this made the model converge faster.
To learn a more robust model, given a frame of a video, we first cropped the frame into an image that contained the infant only. This was to ensure that the model did not focus on objects or backgrounds that were not of interest. A person detector model named YOLOv536 was used to detect the bounding boxes of the “persons” inside a given image. The data were not labeled for the purpose of person detection; therefore, we employed transfer learning that used the pretrained neural network weights on the Microsoft Common Objects in Context dataset. Since we were only interested in predicting the infant, not the therapist and/or parent who appeared in the frame, we performed finer-grain detection using 2 discriminating labels: infant and adult. To achieve this, a subset from the Microsoft Common Objects in Context dataset that had “person” as its label was extracted. We manually annotated this subset using the 2 labels. A total of 1890 images were labeled. The person detector was fine-tuned by training the model on the labeled subset. All frames from the input videos were preprocessed by cropping the image region that only had the infant before feeding the frames into the HRNet model.
Training and Validation of the Action Recognition Model
The PoseConv3D model25 was used for movement recognition because of its robustness against potential noise from pose estimation and its good capability for spatiotemporal feature learning. It received an input of a sequence of 2D key points produced by the pose estimator on a video slice and converted them into a heat map using a Gaussian map. All the heat maps for each frame were stacked together sequentially to form a 3-dimensional heat map volume. Computation efficiency was further enhanced by cropping the heat map frames into the region of interest (which contained the subject) for the 2D heat maps and by performing uniform sampling for the third dimension (ie, the temporal dimension) to reduce the sequence length. Given a list of target classes (movements), PoseConv3D learned to predict the correct class from the key point sequence input. The annotations categorized as “perfect presence” and “acceptable presence” were used as the ground-truth labels.
Statistical Analysis
The infant’s basic and demographic data were analyzed using mean and SD for continuous variables and number and percentage for categorical variables. The intra- and interrater reliabilities of the AIMS assessment and movement annotation were examined using the ICC37 for continuous variables and the Cohen kappa38 for categorical variables. ICC values of <0.5, 0.5 to 0.75, 0.75 to 0.9, and > 0.90 indicated poor, moderate, good, and excellent reliabilities, respectively. Kappa values of 0.01 to 0.20, 0.21 to 0.40, 0.41 to 0.60, 0.61 to 0.80, and 0.81 to 1.00 indicated poor, fair, moderate, good, and excellent agreements, respectively. The action recognition model was tested for its validity in classifying the individual movements and the whole against the therapists’ annotated results using accuracy, recall, precision, and the F1 score. Recall refers to sensitivity, which is the proportion of the movements classified as presence by the action recognition model, and the results were the same as the therapists’ annotation of “perfect presence.” Precision refers to the positive predictive value of the model.39 The F1 score is the harmonic mean of precision and recall. For each item, accuracy was calculated as TP/(TP + TN + FP + FN), recall was calculated as TP/(TP + FN), precision was calculated as TP/(TP + FP), and the F1 score was calculated as 2PR/(P + R), where TP was the number of true-positive results, TN was the number of true-negative results, FP was the number of false-positive results, FN was the number of false-negative results, P was precision, and R was recall. The standards for achieving acceptable validity were set at >0.8 for all of the indexes.40 The statistical analyses were conducted using Statistics Analysis System software (version 9.4; SAS Institute Inc, Cary, NC, USA) with the alpha level set at .05.
Role of the Funding Source
The funders played no role in the design, conduct, or reporting of this study.
Results
Sample
This study included 30 full-term infants and 54 preterm infants with comparable proportions of boys (47% and 50%, respectively), parental occupation as professional and technical (mothers: 70% and 61%, respectively; fathers: 87% and 93%, respectively), and middle to high family income (97% and 100%, respectively). The full-term infants’ mean gestational age was 38.5 weeks (SD = 0.9 weeks), and the mean birth body weight was 3016.7 g (SD = 383.6 g); the preterm infants’ mean gestational age was 32.3 weeks (SD = 3.0 weeks), and the mean birth body weight was 1627.5 g (SD = 588.1 g). The infants returned 1 to 3 times for the AIMS assessment during 4 to 18 months of age. The longitudinal assessments were treated as independent in subsequent analysis. Their AIMS subscale scores, total score, and percentage of motor delay are illustrated in Table 1. The percentage of motor delay ranged from 0% to 1.6% across ages.
Alberta Infant Motor Scale Results for Full-Term and Preterm Infants Whose Video Records Were Available for Machine Learninga
| Age, mo | Full-Term (n = 30) | Preterm (n = 54) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case | Prone | Supine | Sitting | Standing | Total | Delay | Case | Prone | Supine | Sitting | Standing | Total | Delay | |
| 4 | 22 | 5.6 (1.2) | 5.7 (1.4) | 2.4 (0.9) | 1.9 (0.3) | 15.6 (2.2) | 0 (0) | 22 | 5.1 (1.2) | 6.0 (1.2) | 2.7 (0.5) | 1.7 (0.5) | 15.5 (2.0) | 0 (0) |
| 5 | 3 | 6.7 (2.5) | 7.3 (1.5) | 3.3 (0.6) | 2.3 (0.6) | 19.7 (4.2) | 1 (1.6) | 5 | 8.4 (3.6) | 6.6 (2.5) | 3.4 (1.7) | 1.8 (0.8) | 22.2 (11.2) | 2 (2.3) |
| 6 | 13 | 11.5 (2.5) | 8.7 (0.6) | 6.8 (1.6) | 2.8 (0.6) | 29.8 (3.9) | 0 (0) | 14 | 10.1 (2.3) | 8.0 (1.3) | 5.6 (2.0) | 2.4 (0.5) | 26.2 (4.2) | 0 (0) |
| 7 | 5 | 13.8 (2.2) | 9.0 (0.0) | 9.0 (1.9) | 3.0 (0.7) | 34.8 (4.3) | 0 (0) | 8 | 15.6 (3.3) | 9.0 (0.0) | 9.8 (2.8) | 3.0 (1.2) | 37.4 (6.2) | 0 (0) |
| 8 | 6 | 20.7 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 7.5 (1.4) | 49.2 (1.6) | 0 (0) | 7 | 14.0 (4.1) | 9.0 (0.0) | 9.1 (2.0) | 3.6 (2.2) | 35.7 (7.2) | 1 (1.2) |
| 9 | 0 | 3 | 17.0 (3.6) | 9.0 (0.0) | 8.0 (3.6) | 5.3 (3.2) | 39.3 (10.1) | 1 (1.2) | ||||||
| 10 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.8 (0.5) | 52.8 (0.5) | 0 (0) | 10 | 19.7 (2.0) | 9.0 (0.0) | 11.9 (0.3) | 5.3 (2.0) | 45.9 (3.7) | 0 (0) |
| 11 | 3 | 21.0 (0.0) | 9.0 (0.0) | 11.7 (0.6) | 11.7 (2.1) | 53.3 (2.5) | 0 (0) | 4 | 20.5 (0.6) | 7.5 (1.7) | 12.0 (0.0) | 8.3 (2.9) | 48.3 (3.0) | 0 (0) |
| 12 | 4 | 20.8 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 14.3 (2.9) | 56.0 (3.4) | 0 (0) | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.3 (0.6) | 52.3 (0.6) | 0 (0) |
| 13 | 0 | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.7 (3.8) | 53.7 (3.8) | 0 (0) | ||||||
| 14 | 1 | 21.0 | 9.0 | 12.0 | 15.0 | 57.0 | 0 (0) | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.0 (0.0) | 53.0 (0.0) | 2 (2.3) |
| 15 | 0 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 15.0 (1.2) | 57.0 (1.2) | 2 (2.3) | ||||||
| 16 | 0 | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 16.0 (0.0) | 58.0 (0.0) | 0 (0) | ||||||
| Age, mo | Full-Term (n = 30) | Preterm (n = 54) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case | Prone | Supine | Sitting | Standing | Total | Delay | Case | Prone | Supine | Sitting | Standing | Total | Delay | |
| 4 | 22 | 5.6 (1.2) | 5.7 (1.4) | 2.4 (0.9) | 1.9 (0.3) | 15.6 (2.2) | 0 (0) | 22 | 5.1 (1.2) | 6.0 (1.2) | 2.7 (0.5) | 1.7 (0.5) | 15.5 (2.0) | 0 (0) |
| 5 | 3 | 6.7 (2.5) | 7.3 (1.5) | 3.3 (0.6) | 2.3 (0.6) | 19.7 (4.2) | 1 (1.6) | 5 | 8.4 (3.6) | 6.6 (2.5) | 3.4 (1.7) | 1.8 (0.8) | 22.2 (11.2) | 2 (2.3) |
| 6 | 13 | 11.5 (2.5) | 8.7 (0.6) | 6.8 (1.6) | 2.8 (0.6) | 29.8 (3.9) | 0 (0) | 14 | 10.1 (2.3) | 8.0 (1.3) | 5.6 (2.0) | 2.4 (0.5) | 26.2 (4.2) | 0 (0) |
| 7 | 5 | 13.8 (2.2) | 9.0 (0.0) | 9.0 (1.9) | 3.0 (0.7) | 34.8 (4.3) | 0 (0) | 8 | 15.6 (3.3) | 9.0 (0.0) | 9.8 (2.8) | 3.0 (1.2) | 37.4 (6.2) | 0 (0) |
| 8 | 6 | 20.7 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 7.5 (1.4) | 49.2 (1.6) | 0 (0) | 7 | 14.0 (4.1) | 9.0 (0.0) | 9.1 (2.0) | 3.6 (2.2) | 35.7 (7.2) | 1 (1.2) |
| 9 | 0 | 3 | 17.0 (3.6) | 9.0 (0.0) | 8.0 (3.6) | 5.3 (3.2) | 39.3 (10.1) | 1 (1.2) | ||||||
| 10 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.8 (0.5) | 52.8 (0.5) | 0 (0) | 10 | 19.7 (2.0) | 9.0 (0.0) | 11.9 (0.3) | 5.3 (2.0) | 45.9 (3.7) | 0 (0) |
| 11 | 3 | 21.0 (0.0) | 9.0 (0.0) | 11.7 (0.6) | 11.7 (2.1) | 53.3 (2.5) | 0 (0) | 4 | 20.5 (0.6) | 7.5 (1.7) | 12.0 (0.0) | 8.3 (2.9) | 48.3 (3.0) | 0 (0) |
| 12 | 4 | 20.8 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 14.3 (2.9) | 56.0 (3.4) | 0 (0) | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.3 (0.6) | 52.3 (0.6) | 0 (0) |
| 13 | 0 | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.7 (3.8) | 53.7 (3.8) | 0 (0) | ||||||
| 14 | 1 | 21.0 | 9.0 | 12.0 | 15.0 | 57.0 | 0 (0) | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.0 (0.0) | 53.0 (0.0) | 2 (2.3) |
| 15 | 0 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 15.0 (1.2) | 57.0 (1.2) | 2 (2.3) | ||||||
| 16 | 0 | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 16.0 (0.0) | 58.0 (0.0) | 0 (0) | ||||||
Data are presented as mean (SD) for raw scores and numbers (percentages) for infants with motor delay.
Alberta Infant Motor Scale Results for Full-Term and Preterm Infants Whose Video Records Were Available for Machine Learninga
| Age, mo | Full-Term (n = 30) | Preterm (n = 54) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case | Prone | Supine | Sitting | Standing | Total | Delay | Case | Prone | Supine | Sitting | Standing | Total | Delay | |
| 4 | 22 | 5.6 (1.2) | 5.7 (1.4) | 2.4 (0.9) | 1.9 (0.3) | 15.6 (2.2) | 0 (0) | 22 | 5.1 (1.2) | 6.0 (1.2) | 2.7 (0.5) | 1.7 (0.5) | 15.5 (2.0) | 0 (0) |
| 5 | 3 | 6.7 (2.5) | 7.3 (1.5) | 3.3 (0.6) | 2.3 (0.6) | 19.7 (4.2) | 1 (1.6) | 5 | 8.4 (3.6) | 6.6 (2.5) | 3.4 (1.7) | 1.8 (0.8) | 22.2 (11.2) | 2 (2.3) |
| 6 | 13 | 11.5 (2.5) | 8.7 (0.6) | 6.8 (1.6) | 2.8 (0.6) | 29.8 (3.9) | 0 (0) | 14 | 10.1 (2.3) | 8.0 (1.3) | 5.6 (2.0) | 2.4 (0.5) | 26.2 (4.2) | 0 (0) |
| 7 | 5 | 13.8 (2.2) | 9.0 (0.0) | 9.0 (1.9) | 3.0 (0.7) | 34.8 (4.3) | 0 (0) | 8 | 15.6 (3.3) | 9.0 (0.0) | 9.8 (2.8) | 3.0 (1.2) | 37.4 (6.2) | 0 (0) |
| 8 | 6 | 20.7 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 7.5 (1.4) | 49.2 (1.6) | 0 (0) | 7 | 14.0 (4.1) | 9.0 (0.0) | 9.1 (2.0) | 3.6 (2.2) | 35.7 (7.2) | 1 (1.2) |
| 9 | 0 | 3 | 17.0 (3.6) | 9.0 (0.0) | 8.0 (3.6) | 5.3 (3.2) | 39.3 (10.1) | 1 (1.2) | ||||||
| 10 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.8 (0.5) | 52.8 (0.5) | 0 (0) | 10 | 19.7 (2.0) | 9.0 (0.0) | 11.9 (0.3) | 5.3 (2.0) | 45.9 (3.7) | 0 (0) |
| 11 | 3 | 21.0 (0.0) | 9.0 (0.0) | 11.7 (0.6) | 11.7 (2.1) | 53.3 (2.5) | 0 (0) | 4 | 20.5 (0.6) | 7.5 (1.7) | 12.0 (0.0) | 8.3 (2.9) | 48.3 (3.0) | 0 (0) |
| 12 | 4 | 20.8 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 14.3 (2.9) | 56.0 (3.4) | 0 (0) | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.3 (0.6) | 52.3 (0.6) | 0 (0) |
| 13 | 0 | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.7 (3.8) | 53.7 (3.8) | 0 (0) | ||||||
| 14 | 1 | 21.0 | 9.0 | 12.0 | 15.0 | 57.0 | 0 (0) | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.0 (0.0) | 53.0 (0.0) | 2 (2.3) |
| 15 | 0 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 15.0 (1.2) | 57.0 (1.2) | 2 (2.3) | ||||||
| 16 | 0 | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 16.0 (0.0) | 58.0 (0.0) | 0 (0) | ||||||
| Age, mo | Full-Term (n = 30) | Preterm (n = 54) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case | Prone | Supine | Sitting | Standing | Total | Delay | Case | Prone | Supine | Sitting | Standing | Total | Delay | |
| 4 | 22 | 5.6 (1.2) | 5.7 (1.4) | 2.4 (0.9) | 1.9 (0.3) | 15.6 (2.2) | 0 (0) | 22 | 5.1 (1.2) | 6.0 (1.2) | 2.7 (0.5) | 1.7 (0.5) | 15.5 (2.0) | 0 (0) |
| 5 | 3 | 6.7 (2.5) | 7.3 (1.5) | 3.3 (0.6) | 2.3 (0.6) | 19.7 (4.2) | 1 (1.6) | 5 | 8.4 (3.6) | 6.6 (2.5) | 3.4 (1.7) | 1.8 (0.8) | 22.2 (11.2) | 2 (2.3) |
| 6 | 13 | 11.5 (2.5) | 8.7 (0.6) | 6.8 (1.6) | 2.8 (0.6) | 29.8 (3.9) | 0 (0) | 14 | 10.1 (2.3) | 8.0 (1.3) | 5.6 (2.0) | 2.4 (0.5) | 26.2 (4.2) | 0 (0) |
| 7 | 5 | 13.8 (2.2) | 9.0 (0.0) | 9.0 (1.9) | 3.0 (0.7) | 34.8 (4.3) | 0 (0) | 8 | 15.6 (3.3) | 9.0 (0.0) | 9.8 (2.8) | 3.0 (1.2) | 37.4 (6.2) | 0 (0) |
| 8 | 6 | 20.7 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 7.5 (1.4) | 49.2 (1.6) | 0 (0) | 7 | 14.0 (4.1) | 9.0 (0.0) | 9.1 (2.0) | 3.6 (2.2) | 35.7 (7.2) | 1 (1.2) |
| 9 | 0 | 3 | 17.0 (3.6) | 9.0 (0.0) | 8.0 (3.6) | 5.3 (3.2) | 39.3 (10.1) | 1 (1.2) | ||||||
| 10 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.8 (0.5) | 52.8 (0.5) | 0 (0) | 10 | 19.7 (2.0) | 9.0 (0.0) | 11.9 (0.3) | 5.3 (2.0) | 45.9 (3.7) | 0 (0) |
| 11 | 3 | 21.0 (0.0) | 9.0 (0.0) | 11.7 (0.6) | 11.7 (2.1) | 53.3 (2.5) | 0 (0) | 4 | 20.5 (0.6) | 7.5 (1.7) | 12.0 (0.0) | 8.3 (2.9) | 48.3 (3.0) | 0 (0) |
| 12 | 4 | 20.8 (0.5) | 9.0 (0.0) | 12.0 (0.0) | 14.3 (2.9) | 56.0 (3.4) | 0 (0) | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 10.3 (0.6) | 52.3 (0.6) | 0 (0) |
| 13 | 0 | 3 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.7 (3.8) | 53.7 (3.8) | 0 (0) | ||||||
| 14 | 1 | 21.0 | 9.0 | 12.0 | 15.0 | 57.0 | 0 (0) | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 11.0 (0.0) | 53.0 (0.0) | 2 (2.3) |
| 15 | 0 | 4 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 15.0 (1.2) | 57.0 (1.2) | 2 (2.3) | ||||||
| 16 | 0 | 2 | 21.0 (0.0) | 9.0 (0.0) | 12.0 (0.0) | 16.0 (0.0) | 58.0 (0.0) | 0 (0) | ||||||
Data are presented as mean (SD) for raw scores and numbers (percentages) for infants with motor delay.
Selection of Movement Recordings
The 84 participating infants returned for a total of 158 AIMS assessment sessions during 4 to 16 months of age; video recordings were available in 153 assessment sessions for subsequent processing and were not available in 5 sessions because of camera failure in 1 and infant crying in 4. Although 1 video recording in each session had 5 camera views, 153 video recordings yielded 765 videos for trimming and annotation. The annotated video files were further sliced into 13,139 data samples, with the number of data samples ranging from 0 to 1301 per movement (Tab. 2).
Number of Data for Each Movement of the Alberta Infant Motor Scalea
| Movement | No. of Data |
|---|---|
| Prone | |
| Prone lying (1) | 0 |
| Prone lying (2) | 0 |
| Prone prop | 35 |
| Forearm support (1) | 482 |
| Prone mobility | 65 |
| Forearm support (2) | 509 |
| Extend arm support | 386 |
| Rolling prone to supine without rotation | 45 |
| Swimming | 59 |
| Reach from forearm support | 296 |
| Pivoting | 324 |
| Rolling prone to supine with rotation | 17 |
| Four-point kneeling (1) | 234 |
| Propped side lying | 24 |
| Reciprocal crawling | 173 |
| Four-point kneeling to sitting or half-sitting | 777 |
| Reciprocal creeping (1) | 373 |
| Reaching from extended arm support | 121 |
| Four-point kneeling (2) | 51 |
| Modified 4-point kneeling | 63 |
| Reciprocal creeping (2) | 284 |
| Supine | |
| Supine lying (1) | 0 |
| Supine lying (2) | 71 |
| Supine lying (3) | 160 |
| Supine lying (4) | 450 |
| Hands to knees | 222 |
| Active extension | 19 |
| Hands to feet | 122 |
| Rolling supine to prone without rotation | 175 |
| Rolling supine to prone with rotation | 159 |
| Sitting | |
| Sitting with support | 502 |
| Sitting with propped arms | 242 |
| Pull to sit | 387 |
| Unsustain sitting | 3 |
| Sitting with arm support | 143 |
| Unsustained sitting without arm support | 37 |
| Weight shift in unsustained sitting | 20 |
| Sitting without arm support (1) | 1301 |
| Reach with rotation in sitting | 40 |
| Sitting to prone | 93 |
| Sitting to 4-point kneeling | 898 |
| Sitting without arm support (2) | 645 |
| Standing | |
| Supported standing (1) | 50 |
| Supported standing (2) | 469 |
| Supported standing (3) | 186 |
| Pull to stand with support | 203 |
| Pulls to stand/stands | 307 |
| Supported standing with rotation | 226 |
| Cruising without rotation | 129 |
| Half-kneeling | 97 |
| Controlled lowering through standing | 420 |
| Cruising with rotation | 35 |
| Stands alone | 213 |
| Early stepping | 212 |
| Standing from modified squat | 129 |
| Standing from quadruped position | 110 |
| Walks alone | 215 |
| Squat | 131 |
| Movement | No. of Data |
|---|---|
| Prone | |
| Prone lying (1) | 0 |
| Prone lying (2) | 0 |
| Prone prop | 35 |
| Forearm support (1) | 482 |
| Prone mobility | 65 |
| Forearm support (2) | 509 |
| Extend arm support | 386 |
| Rolling prone to supine without rotation | 45 |
| Swimming | 59 |
| Reach from forearm support | 296 |
| Pivoting | 324 |
| Rolling prone to supine with rotation | 17 |
| Four-point kneeling (1) | 234 |
| Propped side lying | 24 |
| Reciprocal crawling | 173 |
| Four-point kneeling to sitting or half-sitting | 777 |
| Reciprocal creeping (1) | 373 |
| Reaching from extended arm support | 121 |
| Four-point kneeling (2) | 51 |
| Modified 4-point kneeling | 63 |
| Reciprocal creeping (2) | 284 |
| Supine | |
| Supine lying (1) | 0 |
| Supine lying (2) | 71 |
| Supine lying (3) | 160 |
| Supine lying (4) | 450 |
| Hands to knees | 222 |
| Active extension | 19 |
| Hands to feet | 122 |
| Rolling supine to prone without rotation | 175 |
| Rolling supine to prone with rotation | 159 |
| Sitting | |
| Sitting with support | 502 |
| Sitting with propped arms | 242 |
| Pull to sit | 387 |
| Unsustain sitting | 3 |
| Sitting with arm support | 143 |
| Unsustained sitting without arm support | 37 |
| Weight shift in unsustained sitting | 20 |
| Sitting without arm support (1) | 1301 |
| Reach with rotation in sitting | 40 |
| Sitting to prone | 93 |
| Sitting to 4-point kneeling | 898 |
| Sitting without arm support (2) | 645 |
| Standing | |
| Supported standing (1) | 50 |
| Supported standing (2) | 469 |
| Supported standing (3) | 186 |
| Pull to stand with support | 203 |
| Pulls to stand/stands | 307 |
| Supported standing with rotation | 226 |
| Cruising without rotation | 129 |
| Half-kneeling | 97 |
| Controlled lowering through standing | 420 |
| Cruising with rotation | 35 |
| Stands alone | 213 |
| Early stepping | 212 |
| Standing from modified squat | 129 |
| Standing from quadruped position | 110 |
| Walks alone | 215 |
| Squat | 131 |
Movements in bold type were considered for machine learning. The numbers in the parentheses indicate levels of movement patterns that are defined by the Alberta Infant Motor Scale (AIMS) assessment.
Number of Data for Each Movement of the Alberta Infant Motor Scalea
| Movement | No. of Data |
|---|---|
| Prone | |
| Prone lying (1) | 0 |
| Prone lying (2) | 0 |
| Prone prop | 35 |
| Forearm support (1) | 482 |
| Prone mobility | 65 |
| Forearm support (2) | 509 |
| Extend arm support | 386 |
| Rolling prone to supine without rotation | 45 |
| Swimming | 59 |
| Reach from forearm support | 296 |
| Pivoting | 324 |
| Rolling prone to supine with rotation | 17 |
| Four-point kneeling (1) | 234 |
| Propped side lying | 24 |
| Reciprocal crawling | 173 |
| Four-point kneeling to sitting or half-sitting | 777 |
| Reciprocal creeping (1) | 373 |
| Reaching from extended arm support | 121 |
| Four-point kneeling (2) | 51 |
| Modified 4-point kneeling | 63 |
| Reciprocal creeping (2) | 284 |
| Supine | |
| Supine lying (1) | 0 |
| Supine lying (2) | 71 |
| Supine lying (3) | 160 |
| Supine lying (4) | 450 |
| Hands to knees | 222 |
| Active extension | 19 |
| Hands to feet | 122 |
| Rolling supine to prone without rotation | 175 |
| Rolling supine to prone with rotation | 159 |
| Sitting | |
| Sitting with support | 502 |
| Sitting with propped arms | 242 |
| Pull to sit | 387 |
| Unsustain sitting | 3 |
| Sitting with arm support | 143 |
| Unsustained sitting without arm support | 37 |
| Weight shift in unsustained sitting | 20 |
| Sitting without arm support (1) | 1301 |
| Reach with rotation in sitting | 40 |
| Sitting to prone | 93 |
| Sitting to 4-point kneeling | 898 |
| Sitting without arm support (2) | 645 |
| Standing | |
| Supported standing (1) | 50 |
| Supported standing (2) | 469 |
| Supported standing (3) | 186 |
| Pull to stand with support | 203 |
| Pulls to stand/stands | 307 |
| Supported standing with rotation | 226 |
| Cruising without rotation | 129 |
| Half-kneeling | 97 |
| Controlled lowering through standing | 420 |
| Cruising with rotation | 35 |
| Stands alone | 213 |
| Early stepping | 212 |
| Standing from modified squat | 129 |
| Standing from quadruped position | 110 |
| Walks alone | 215 |
| Squat | 131 |
| Movement | No. of Data |
|---|---|
| Prone | |
| Prone lying (1) | 0 |
| Prone lying (2) | 0 |
| Prone prop | 35 |
| Forearm support (1) | 482 |
| Prone mobility | 65 |
| Forearm support (2) | 509 |
| Extend arm support | 386 |
| Rolling prone to supine without rotation | 45 |
| Swimming | 59 |
| Reach from forearm support | 296 |
| Pivoting | 324 |
| Rolling prone to supine with rotation | 17 |
| Four-point kneeling (1) | 234 |
| Propped side lying | 24 |
| Reciprocal crawling | 173 |
| Four-point kneeling to sitting or half-sitting | 777 |
| Reciprocal creeping (1) | 373 |
| Reaching from extended arm support | 121 |
| Four-point kneeling (2) | 51 |
| Modified 4-point kneeling | 63 |
| Reciprocal creeping (2) | 284 |
| Supine | |
| Supine lying (1) | 0 |
| Supine lying (2) | 71 |
| Supine lying (3) | 160 |
| Supine lying (4) | 450 |
| Hands to knees | 222 |
| Active extension | 19 |
| Hands to feet | 122 |
| Rolling supine to prone without rotation | 175 |
| Rolling supine to prone with rotation | 159 |
| Sitting | |
| Sitting with support | 502 |
| Sitting with propped arms | 242 |
| Pull to sit | 387 |
| Unsustain sitting | 3 |
| Sitting with arm support | 143 |
| Unsustained sitting without arm support | 37 |
| Weight shift in unsustained sitting | 20 |
| Sitting without arm support (1) | 1301 |
| Reach with rotation in sitting | 40 |
| Sitting to prone | 93 |
| Sitting to 4-point kneeling | 898 |
| Sitting without arm support (2) | 645 |
| Standing | |
| Supported standing (1) | 50 |
| Supported standing (2) | 469 |
| Supported standing (3) | 186 |
| Pull to stand with support | 203 |
| Pulls to stand/stands | 307 |
| Supported standing with rotation | 226 |
| Cruising without rotation | 129 |
| Half-kneeling | 97 |
| Controlled lowering through standing | 420 |
| Cruising with rotation | 35 |
| Stands alone | 213 |
| Early stepping | 212 |
| Standing from modified squat | 129 |
| Standing from quadruped position | 110 |
| Walks alone | 215 |
| Squat | 131 |
Movements in bold type were considered for machine learning. The numbers in the parentheses indicate levels of movement patterns that are defined by the Alberta Infant Motor Scale (AIMS) assessment.
Training and Validation of the Action Recognition Model
The 58 movements were ranked according to the number of data samples, with a median of 175. Because including movements with few data samples might decrease the model quality because of limited information for recognizing movement features,41 movements with a sufficient number of data samples that exceeded the median (range = 186–1301) were included for machine learning. Furthermore, 5 movements (“rolling supine to prone with rotation,” sitting with arm support,” “cruising without rotation,” “standing from modified squat,” and “squat”) with data samples close to the median (129, 129, 131, 143, and 159, respectively) were also selected because of their developmental significance in various positions and ages. Machine learning was therefore conducted over a total of 31 movements, including 9 out of 21 prone movements (42.8%), 3 out of 9 supine movements (33.3%), 7 out of 12 sitting movements (58.3%), and 12 out of 16 standing movements (75%).
The number of data samples used for machine learning was 400 for each movement that were randomly split into training and validation sets with a ratio of 8:2. Each movement contained 160 data in the training set and 40 data in the validation set. If the data sample exceeded 400, then the training set was randomly selected from the whole sample. For the 5 movements with a data sample of <400, we increased the probability of the movements selected by the model by further slicing the samples into frames, with the mean frame number ranging from 204 to 625.
Validation of the action recognition model for tracking and classification of the 31 movements in preterm and full-term infants combined altogether showed an overall accuracy of 0.91, a precision of 0.91, a recall of 0.91, and an F1 score of 0.91 (Fig. 3). Twenty-eight movements (90.3%) showed high precision, recall, and F1 score (all >0.80); 3 (9.7%) [“forearm support (1),” “forearm support (2),” and “supine lying (4)”] showed fair precision, recall, and F1 score (0.76–0.79). Further examination of the confusion matrix revealed that 2 sets of movements were easily confused: “forearm support (2)” versus “forearm support (1)” and “reciprocal creeping (1)” versus “reciprocal creeping (2)” (Fig. 4A).
Validation of the artificial intelligence model of movement classification for all infants, preterm infants, and full-term infants.
Confusion matrix of the action recognition model in the classification of 31 movements for (A) all infants, (B) preterm infants, and (C) full-term infants, with color indicating the number of infants in agreement, ranging from 1 to 80.
Validation results of the action recognition model for tracking and classification of the 31 movements in preterm infants and full-term infants are illustrated in Figure 4B and C, respectively. The validity values for preterm infants (accuracy = 0.91, precision = 0.91, recall = 0.91, and F1 score = 0.90) were close to those of full-term infants (accuracy = 0.92, precision = 0.91, recall = 0.91, and F1 score = 0.91) (Fig. 3).
Discussion
This was the first study to establish an AI algorithm for movement tracking and recognition in full-term and preterm infants at 4 to 18 months of age. The annotated movement video recordings served as the gold standards in the development and validation of the AI algorithm. After a short course of training in movement annotation, the 3 pediatric physical therapists with at least 2 years of clinical experience achieved high levels of intra- and interrater agreement in labeling full-term and preterm infants’ movements. The guideline of video recording regarding setup, dressing, and camera view together with the annotation procedure may be used to ensure reliable gold standards for future establishment of a machine learning model.
The AI algorithm successfully classified 31 of the 58 movements in full-term and preterm infants aged 4 to 18 months with an overall accuracy of 0.89. The 31 movements selected in this study were frequently observed and developmentally significant. Reich et al28 utilized a 25-point pose estimation model and a shallow multilayer neural network to accurately recognize fidgety movements of 51 full-term infants in the supine position at age 4 to 16 weeks. In contrast, our use of fewer body points for pose estimation incorporating PoseConv3D was accurate in classifying more complex infant movements in various positions across a wider time period. The results suggest that the AI algorithm is an accurate approach for classifying 31 movements in full-term and preterm infants in a standardized clinical setup.
Confusion matrix results of the AI algorithm showed that 2 sets of movements were easily confused: “forearm support (2)” was easily confused with “forearm support (1)”; “reciprocal creeping (2)” was easily confused with “reciprocal creeping (1).” The confusion might relate to the similarity in their movement features and insufficient amount of data samples. “Forearm support (1)” and “forearm support (2)” are similar in weight-bearing on forearms and antigravity movement with head raising but vary in posture, with the elbows in line with the shoulders in the former and the elbows in front of the shoulders in the latter. “Reciprocal creeping (1)” and “reciprocal creeping (2)” are similar in weight-bearing on hands and knees and antigravity movements with reciprocal limb movements but vary in posture, with lumbar lordosis in the former and flattening in the latter. However, the infant’s arms, shoulders, and trunk might occlude with toys or the movement being recorded in the frontal view that these key body parts were not obviously viewed for annotation and machine learning. The size of video samples is another influencing factor for pattern recognition by AI.42 Increasing the data sample size and augmenting the data for these easily confused movement categories are plausible ways to enhance the statistical power in fine tuning the algorithm.
The AI algorithm was shown to be an innovative approach for movement tracking and classification in full-term and preterm infants. To adapt for changing recording settings from the standardized laboratory to the home environment and ordinary clinic setup, further investigation of this type of assessment in the followings is necessary. First, enlarging the data sample particularly those prone and supine movements at younger age that were omitted from analysis to help capture and classify a wider spectrum of infant movements through machine learning. Second, a cellphone mobile application is currently under development to be incorporated into the AI model to offer a prospective solution to provide caregivers access to digital infant motor screening. Third, the algorithm established on laboratory videos in this study requires validation on infant movements recorded and uploaded via the app by caregivers at home or by health care providers at ordinary clinical setting. Finally, computer automation of video trimming and selection in preprocessing of movement recognition is essential to minimize the manual cropping efforts. The ultimate goal of this AI technology of infant motor assessment is to diminish health care disparities for some high-risk pediatric populations and low-resourced communities.
Limitations
The study has 3 limitations of note. First, the infants were mainly from families of middle to high socioeconomic status, and their movement characteristics may be homogeneous in establishing the AI algorithm. Second, infants were scheduled for assessment at 4 months or older because of parental concern of possible infection when going outdoor without vaccination during the pandemic period. This may decrease the likelihood of those movements of young age being included for machine learning. Third, infant movements collected at the laboratory may differ from those performed in situ. Movement records collected from diverse populations and contexts will increase the generalizability of the results.
Conclusions
This study demonstrated that the AI model was reliable and accurate in classifying 31 movements in full-term and preterm infants aged 4 to 18 months in a standardized laboratory setting. The results have implicated the potential of machine learning approach to augment the merits of infant movement assessment and boost its application. The AI model is currently under refinement and validation for its action recognition of infant movements that are video recorded at home or ordinary clinical setting. This study facilitates further development of the AI model as an innovative and remote assessment to assist physical therapists and pediatricians in early detection of infants with developmental risk or neuromotor disorder.
Author Contributions
Concept/idea/research design: J.Y.-J. Hsu, S.-F. Jeng, P.N. Tsao, W.-C. Liao, W.-J. Chen
Writing: S.-F. Jeng, J.Y.-J. Hsu, S.-C. Lin, E. Chandra
Data collection and environmental setup: S.-C. Lin, E. Chandra, P.N. Tsao, T.-A. Yen, J.Y.-J. Hsu, S.-F. Jeng
Data analysis: S.-C. Lin, E. Chandra, J.Y.-J. Hsu
Project management: S.-F. Jeng
Providing facilities/equipment: J.Y.-J. Hsu, S.-F. Jeng
Consultation (including review of manuscript before submitting): S.-C. Lin, E. Chandra, P.N. Tsao, W.-C. Liao, W.-J. Chen, T.-A. Yen, J.Y.-J. Hsu, S.-F. Jeng
All authors have approved the final manuscript as submitted and have agreed to be accountable for all aspects of the study.
Acknowledgments
The authors thank the infants and families for their participation in this study and the medical, nursing, and physical therapist staff at the National Taiwan University Children’s Hospital in Taiwan for their assistance in recruitment and data collection. We also thank Li-An Ting, Nien-Tse Lin, Emily Yi-Ning Chen, Jian-X Chen, Chin-Yi Liao, Chung-Che Chang, Zhao-Kai Lu, and all participating members of iAgents Lab of NTU CSIE for the contributions and collaboration, and Yu-Ching Hsiao, Yohanes Purwanto, and Chun-Wun Hsieh of Infant Motor Development Laboratory of School of Physical Therapy at NTU for the developmental assessment and follow-up assistance.
Ethics Approval
This study was approved by the Human Rights Review Committee at the study hospital (202010031RINB and 202012089RINB).
Funding
This study was supported by the National Science and Technology Council (MOST 110-2314-B-002-055) in Taiwan.
Clinical Trial Registration
This study was registered with the National Taiwan University Hospital, NCT04684173 and NCT05456126 at https://clinicaltrials.gov/.
Data Availability
Data are available on reasonable request. Data are in the form of video recordings and are stored in a password-protected research drive accessible only to the researchers of this study. Video recordings contain identifiable data and will not be made available on request to maintain participant anonymity.
Disclosures
The authors completed the ICMJE Form for Disclosure of Potential Conflicts of Interest and reported no conflicts of interest.
References
Author notes
Shiang-Chin Lin and Erick Chandra are both first authors.
Yung-Jen Hsu and Suh-Fang Jeng have equal contribution to the study.
Comments