Evaluation of the Military Functional Assessment Program for Return-to-Duty Decision Making: A Longitudinal Study

Abstract

Following concussion, return-to-duty decisions are made by medical providers and leaders using informal assessment, review of symptom resolution, and clinical assessments. In an effort to provide military-specific tasks for use in the decision making regarding return-to-duty, a battery developed from the set of basic soldier skills, the Military Functional Assessment Program (MFAP), was evaluated for construct validity using clinical assessment outcomes as well as the relationships between MFAP performance and indicators of long-term performance and satisfaction. A total of 48 participants were enrolled in the study providing baseline and post-treatment clinical assessment, and MFAP performance data. Twenty participants provided follow-up data at 6-month post-MFAP. Correlational analyses suggest significant relationships between MFAP performance ratings and clinical measures of vestibular and cognitive functions and psychological well-being. These findings are consistent with those from previous research on construct validity of the MFAP tasks. Performance on one MFAP task related to perceptions of performance and overall MFAP ratings related to satisfaction reported at 6 months. These findings provide preliminary, however limited, support for these tasks being indicative of the motivation and mental state of the program participant.

INTRODUCTION

Military operations during Operation Enduring Freedom and Operation Iraqi Freedom have resulted in significantly higher rates of traumatic brain injury (TBI) among Soldiers as compared with previous wars, mainly due to the prevalence of improvised explosive attacks.¹ Severe brain injuries are less common than the milder forms of TBI, given adequate protective gear, yet the milder forms are more difficult to diagnose and therefore treat. Severe cases of TBI are more straightforward in their clinical presentation, whereas mild TBI (mTBI) symptoms are often invisible in their presentation, causing many cases to go undiagnosed until sequelae occur later in the form of persistent symptoms (e.g., headache, fatigue, dizziness, concentration problems, anxiety), or post-concussive syndrome (PCS), which can be ongoing.² Such symptoms of mTBI can result in long-lasting deficits that significantly impact a Soldier’s ability to complete mission-critical tasks, making it imperative that providers accurately and reliably determine if a Soldier is capable of and ready to return-to-duty (RTD), or if he or she requires further rehabilitation and treatment, or medical discharge.^3,4 Standard clinical assessments, upon which the U.S. Army has traditionally relied, have not been adequately investigated in terms of the mission-critical consequences of mTBI, specifically the complex cognitive and neurosensory requirements of effective military functioning in the field. Military RTD assessments must take into account the unique physical and mental stressors Soldiers face on the battlefield (e.g., combat environments, high workloads, sleep deprivation, fatigue) and the specificity of critical military abilities, such as marksmanship, military vehicle egress, first aid, tactical formations, and land navigation.⁴

To address the need for a military-specific assessment tool for RTD, the National Intrepid Center of Excellence Intrepid Satellite-III (NICoE ISIII) located at Fort Campbell, KY, developed the Military Functional Assessment Program (MFAP), which has been utilized since 2008 to assist military and clinical professionals in determining Soldier readiness to RTD following mTBI. The MFAP is 5-day assessment program consisting of 10 independent tasks (adapted from the Soldier Manual of Common Tasks: Warrior Skills Level 1)⁵ and 1 unrated confidence and team building exercise. The MFAP exercises have been developed to accommodate a variety of Soldier injuries while maintaining a high enough level of difficulty to reflect active-duty requirements that maximize operational success. By testing Soldiers in a more applied setting, NICoE ISIII staff can better determine a Soldier’s ability to function while under the pressure of realistic environmental stressors that approximate actual combat scenarios – elements that can significantly impair performance not otherwise captured by traditional clinical assessments.⁶

In order to determine whether this program has potential for widespread implementation, the validity and reliability of the assessment program need to be established. Construct validity of these tasks was initially explored in a correlational study of an archival dataset from NICoE ISIII.⁴ The results of this study supported initial convergent validity with a number of commonly used cognitive and vestibular tests that identify critical functioning deficits related to overall neurocognitive status, depression, anxiety, disorientation, alcohol use, and dynamic visual acuity. Most tasks tax multiple functional domains (e.g., cognitive, vestibular, vision, and psychological), whereas some of the simpler tasks are only cognitive in nature. Specific descriptions of each task, the demands identified by the professionals at NICoE ISIII, and summary results from the archival construct validity⁷ are provided in the Appendix.

A NICoE ISIII staff non-commissioned officer (NCO), who has been trained on the MFAP, rates each Soldier’s performance on 10 tasks and gives each task a subjective rating score using a guided rating scale that incorporates subjective (observed independence level [IL] of the patient) and objective (U.S. Army operational performance standards) criteria (Table I). The ratings are on a 5-point scale and translate to a binary Pass/Fail outcome (Fig. 1). The MFAP occurs following a 12- to 24-week treatment program at NICoE ISIII.

The NCO rates performance on all of the tasks and additional ratings are provided on a subset of the tasks by an occupational therapist (OT), physical therapist (PT), and mental health (MH) counselor. The OT rates global functioning; PT rates physical strength, agility, and balance; and MH counselor rates anxiety level and psychological level of independence. All raters collaborate at the end of the week to derive a single overall IL score based upon each of their respective operational assessment criteria. Task-specific rating scores based on the IL criteria provided in Table I are collected from all raters and reported on the MFAP Final Results Reporting Sheet (Fig. 2), which includes an overall go/no-go/stop determination for each task, the number and percentage of tasks passed, a general level of independence score (LOI), and recommendation for RTD, among other findings. All scores on the final sheet are based on a collaborative decision made by all four MFAP raters.

Further establishing the validity and reliability of the MFAP for RTD decision making is needed in order to determine whether and how the program can be implemented at other facilities. The archival dataset used in the previous study⁴ did not include any information about the MFAP participants following completion of the program thus prohibiting evaluation of the relationships between MFAP task performance and long-term success. Thus, the primary objective of this study was to explore the predictive validity of MFAP performance employing a longitudinal design. A secondary objective was to build upon the archival dataset study and further explore construct validity.

Method and Materials

This longitudinal study examined the relationships between MFAP performance, clinical assessments, and RTD success at 6 months following completion of the MFAP. Additionally, this study evaluated the convergent validity of the MFAP tasks relative to the clinical assessments administered at the beginning and end of the program. This study was reviewed and approved by the U.S. Army Medical Research and Materiel Command’s Institutional Review Board.

Participants

Study participants were 48 U.S. Army Soldiers at NICoE ISIII completing a treatment program following concussion/traumatic brain injury and subsequently taking part in the MFAP. The mean age was 28.90 years (SD = 7.52, min = 19, max = 54). Three participants were female. Rank and Military Occupational Specialty (MOS) data are provided in Table II. Prior to enrollment, a formal power analysis was not conducted to estimate sample size given the exploratory nature of the study. Also, the experienced attrition rate was not anticipated. Note that complete medical histories were not available for review and thus prior injuries are unknown.

Materials

Participants completed a set of assessments (Table III) including both clinical tests and self-report measures for clinical purposes by the NICoE ISIII personnel. At 6-month post-MFAP, the research team administered the Canadian Occupational Performance Measure (COPM), a self-report measure of one’s perceived level of ability and degree of satisfaction of one’s own job performance. Participants gave consent for their clinical scores and outcomes to be used for the purposes of this research study. The NICoE ISIII assessments were administered at intake and following completion of each individual’s treatment and rehabilitation plan, prior to the MFAP.

Procedure

Informed consent was obtained for all participants prior to study procedures and occurred during the week-long MFAP assessment. Following consent procedures, participants completed a brief demographics form requesting rank, age, MOS, and gender. Upon completion of the MFAP, NICoE ISIII provided the research team with the clinical assessment data (Table III), MFAP individual task ratings, MFAP overall ratings, and whether the individual was recommended for RTD. At 6-month post-MFAP completion, participants were contacted by the research team to conduct the first set of follow-up assessments and, for those who were still in the military, to request performance documents, if applicable and available, including NCO evaluation reports, performance points worksheets, standard weapon scorecards, performance/event-related counselings, weapon scorecards – known distance course, weapon scorecards – scaled target course, PT scorecards, and weapon scorecards – combat field fire. All the requested documents are readily accessible to Soldiers. The COPM was administered over the telephone by a member of the research team as it is a structured interview.

Quality Control and Statistical Analysis Approach

All data were inspected to identify any quality concerns (e.g., participant inattention, technical failure). Data distributions were examined for normality as well as skewness and kurtosis. To address the primary objective of evaluating predictive validity, performance data following RTD (including assessments and requested documents) and the individual MFAP task independence level ratings as well as overall MFAP independence level ratings were explored using correlation matrices and stepwise linear regression. To address the secondary objective of evaluating construct validity, MFAP ratings and clinical assessment scores were examined using correlational matrices.

RESULTS

This study experienced a very high attrition rate (58.3%) and none of the requested performance documents were provided by the participants. Multiple attempts were made by the research team to follow-up and despite these efforts, the documents were not provided nor was feedback as to why. Specifically, of the 48 participants, 20 completed the COPM at 6-month post-MFAP completion. No distinguishable pattern with respect to any of the data collected (e.g., MFAP performance, demographics) emerged with respect to those who chose to withdraw nor were reasons for withdrawal and/or non-response provided (withdrawn participants did not respond to repeated attempts to contact for follow-up).

Descriptive Statistics of MFAP Data

Of those who completed the MFAP (N = 47), 4 cases failed (8.51%) and were not recommended for RTD. The mean ratings provided by the NCO, physical therapy (PT), occupational therapy (OT), and mental health (MH) as well as the percentage of failures for each MFAP task are included in Table IV.

Descriptive Statistics of Clinical Assessment Data

Mean and standard deviations were calculated for the clinical assessment data (presented in Table V). Paired-samples t-tests were used to compare scores on the assessments, where available, from pre- to post-treatment (Table V). For the available pre- and post-treatment assessment data, participants performed better at post-treatment (supported by results from the Sensory Organization Test, Dynamic Visual Acuity, and Neuropsychological Assessment Battery assessments) and reported higher levels of self-report performance and satisfaction with their skill levels (supported by results from the COPM performance and satisfaction scores). These results suggest improvement in function over the course of the treatment program.

Objective 1: Predictive Validity

Given that the majority of performance outcomes and assessments were not made available or completed by the participants, the analyses to address the primary objective of evaluating predictive validity of the MFAP with respect to RTD success were limited to the two COPM scores, self-reported satisfaction with one’s performance, satisfaction, and self-reported level of performance, performance. Mean COPM performance and satisfaction scores are presented in Figure 3. Paired-samples t-tests were used to evaluate differences in scores from pre- and post-treatment, and at 6 months following the MFAP. Two significant comparisons emerged. Specifically, performance and satisfaction scores increased from pre- to post-treatment, t(29) = −4.831, p < 0.001, t(29) = −6.559, p < 0.001, respectively.

In order to evaluate the predictive validity of the overall MFAP level of independence ratings and individual MFAP task independence level ratings (provided by the NCO) for self-reported performance and satisfaction, a hierarchical regression model approach was implemented. The independence level ratings were limited to those provided by the NCO given that these ratings are based on performance to U.S. Army standards, the NCO rates all MFAP tasks, and the NCO ratings correlated very strongly with those from the PT, OT, and MH. First, correlational matrices were conducted for the individual MFAP task independence level ratings, overall MFAP independence level rating, and COPM performance and satisfaction scores collected at 6 months post-MFAP (Table VI). Ratings from the TCCC and overall independence level correlated with COPM performance and satisfaction scores at 6 months, respectively.

A stepwise linear regression model predicting COPM satisfaction scores at 6 months post-MFAP was not significant, F(5,18) = 0.893, p = 0.603. A stepwise linear regression model predicting COPM performance scores at 6 months post-MFAP was significant, F(2,18) = 5.353, p = 0.017, adjusted R²= 0.326. The predictors included in this model are TCCC ratings, β = −2.623, p = 0.01, and overall independence level ratings, β = −2.175, p = 0.045. Note that age and gender were also included in the analysis and were not related to the outcome variables.

Objective 2: Construct Validity

Correlation matrices were used to evaluate construct validity similar to that in the previous archival database study. The results are presented in Table VII. Findings were primarily consistent with previous findings.⁴

DISCUSSION

The primary objective of this study was to evaluate the predictive validity of the MFAP with respect to success following RTD or separation from the military. Unfortunately, the study faced a number of challenges that ultimately prohibit drawing any profound inferences regarding predictive validity. The findings with respect to the self-reported levels of satisfaction and performance suggest a potentially meaningful relationship between these outcomes and performance on the TCCC task as well as the overall MFAP independence-level rating.

To speculate as to why these relationships may exist with the TCCC task, two aspects of the task were considered: the demands of the task and the administration of the task. Unlike many of the other MFAP tasks, the TCCC primarily taxes a single domain (cognition). It is possible that the relative simplicity of the task reduces the variability associated with more complex factors and thus is more easily mapped to the outcomes. Additionally, the task includes an instructional portion (provided in Microsoft PowerPoint format) and follows a specific set of tasks with very little flexibility in the administration. Thus, the ratings for this task may be more consistent between raters and administrations also contributing to more easily mapped associations with outcomes. This is supported in part by the findings of an evaluation of the interrater reliability of the MFAP tasks.¹⁸

With respect to the MFAP independence level rating and self-reported satisfaction and performance scores, this relationship may possibly be attributed to the collaborative nature of the meetings conducted to determine these overall ratings. Given that the PT, MH, OT, and NCO gather to discuss overall performance, the ratings may be more reflective of overall function than any one specific rating or task.

The secondary objective of the study was to further evaluate the construct validity of the tasks given the findings of an archival data study.⁴ Overall, the findings are consistent with the previous findings further supporting the vestibular and cognitive demands identified for the individual tasks. However, there are three findings that are postulated by the demands identified by the task that the previous study did not support. Specifically, the current results suggest a cognitive component of the land navigation task and a psychological health component of the weapons qualification and shoot/no-shoot tasks. Two additional findings not supported by previous research nor by the demands identified for the tasks emerged. Both suggest a vestibular component of the land navigation preparation and tactical combat casualty care tasks. Both of the relationships are negative suggesting that worse performance on the tasks corresponds to better performance on the vestibular assessments, which at this time is uninterpretable. Given the lack of vestibular demands in these tasks, and the general concept that correlation does not imply causation, it is possible that there is a third confounding factor driving these associations. Given the available information, it is not reasonable to speculate as to what this third factor may be.

Limitations

The study findings are limited in their generalizability and interpretability given the significant challenges faced in the execution of this study as well as the limited variability in participant demographics and MFAP performance. Specifically, the study experienced a high attrition rate and a lack of compliance with respect to requested performance documentation post-MFAP. More direct interaction between the research team and participants may help eliminate these challenges in future studies. Also, the research team was provided with clinical data and as such did not have control over the clinical assessments administered during the course of treatment. Additional standardized procedures would be preferable in future studies. Given these limitations, inferences with respect to predictive validity of the MFAP for post-RTD success are not possible and further research is needed.

CONCLUSIONS

The findings of this study provide weak support for predictive validity of the MFAP as it was designed to evaluate and are limited in their interpretation. However, this lack of support can largely be attributed to the constraints associated with the study, the high attrition rate, and lack of compliance with respect to requested performance documentation post-MFAP. The findings should not be interpreted as evidence of a lack of predictive validity but a failure to detect it should it exist. Alternatively, the findings do further support the construct validity of the MFAP as previously reported, specifically, confirming results for eight of the tasks and suggesting additional support for the TCCC task.

Supplementary Material

Supplementary material is available at Military Medicine online.

Previous Presentation

Presented as an oral talk at the 2017 Military Health System Research Symposium, August 2017, Kissimmee, FL; abstract # MHSRS-17-0975.

Funding

U.S. Army Medical Research and Materiel Command Military Operational Medicine Research Program. This supplement was sponsored by the Office of the Secretary of Defense for Health Affairs.

Acknowledgments

The authors would like to express their sincere gratitude to the research team who assisted with data collection procedures at the U.S. Army Aeromedical Research Laboratory Aircrew Health and Performance Division as well as the personnel and staff at the National Intrepid Center of Excellence Intrepid Site III.

References

Author notes

The views, opinions, and/or findings contained in this report are those of the authors and should not be construed as an official Department of the Army position, policy, or decision, unless so designated by other official documentation. Citation of trade names in this report does not constitute an official Department of the Army endorsement or approval of the use of such commercial items.