Feasibility of Remote Administration of the Uniform Data Set-Version 3 for Assessment of Older Adults With Mild Cognitive Impairment and Alzheimer’s Disease Open Access

Abstract

Objective

Assess the feasibility and concurrent validity of a modified Uniform Data Set version 3 (UDSv3) for remote administration for individuals with normal cognition (NC), mild cognitive impairment (MCI), and early dementia.

Method

Participants (N = 93) (age: 72.8 [8.9] years; education: 15.6 [2.5] years; 72% female; 84% White) were enrolled from the Wake Forest ADRC. Portions of the UDSv3 cognitive battery, plus the Rey Auditory Verbal Learning Test, were completed by telephone or video within ~6 months of participant’s in-person visit. Adaptations for phone administration (e.g., Oral Trails for Trail Making Test [TMT] and Blind Montreal Cognitive Assessment [MoCA] for MoCA) were made. Participants reported on the pleasantness, difficulty, and preference for each modality. Staff provided validity ratings for assessments. Participants’ remote data were adjudicated by cognitive experts blinded to the in person-diagnosis (NC [N = 44], MCI [N = 35], Dementia [N = 11], or other [N = 3]).

Results

Remote assessments were rated as pleasant as in-person assessments by 74% of participants and equally difficult by 75%. Staff validity rating (video = 92%; phone = 87.5%) was good. Concordance between remote/in-person scores was generally moderate to good (r = .3 −.8; p < .05) except for TMT-A/OTMT-A (r = .3; p > .05). Agreement between remote/in-person adjudicated cognitive status was good (k = .61–.64).

Conclusions

We found preliminary evidence that older adults, including those with cognitive impairment, can be assessed remotely using a modified UDSv3 research battery. Adjudication of cognitive status that relies on remotely collected data is comparable to classifications using in-person assessments.

INTRODUCTION

Face-to-face administration is the “gold standard” for both research and clinical cognitive assessments, and as such, most neuropsychological measures are intended to be administered in person. However, conditions often arise when research participants and patients cannot meet face to face for assessments, for example, because they must travel great distances, may have limited mobility or poor eyesight, or do not have transportation. Further, there is ample evidence that underserved populations, including rural populations and racial/ethnic minorities, cite travel-related burdens due to study/clinic locations as a barrier to care and research participation (Buzza et al., 2011; Williams et al., 2010). Additionally, these populations have been disproportionately affected by the COVID-19 pandemic further widening gaps in access to care and clinical research participation (Henning-Smith, 2020; Ruprecht et al., 2021). Alternatives to face-to-face assessments may provide an opportunity to alleviate the testing burden caused by barriers such as the COVID-19 pandemic and longer-standing social inequities.

Prior studies have demonstrated that both telephone- and video-administered cognitive assessments can be used with older adults (Alegret et al., 2021; Castanho et al., 2017; Crooks et al., 2005; Grosch et al., 2011; Hunter et al., 2021; Miller & Barr, 2017; Wilson et al., 2010). Studies have documented that video and telephone cognitive assessments also have suitable levels of acceptability and are perceived positively by older adults, even those with cognitive impairment or dementia ( Castanho et al., 2016; Caze et al., 2020; Cullum et al., 2014; Lacritz et al., 2020; Loh et al., 2007). One study found 98% satisfaction ratings with a clinical tele-neuropsychology appointment for patients with mild cognitive impairment (MCI) or early-stage AD (Parikh et al., 2013). A recent meta-analysis found that across 12 studies with nearly 500 participants, there was high concordance between video-administered and face-to-face assessments (Brearly et al., 2017). A slight difference was noted for timed tests, with virtual scores falling a 1/10th of a standard deviation unit lower than face-to-face scores. Several measures, including the Mini-Mental Status Exam (MMSE), Hopkins Verbal Learning Test–Revised (HVLT-R), Digit Span test, Oral Trail Making Test (OTMT), and Letter and Category Fluency, have also been shown to have intraclass correlations (ICCs) between video and in-person administrations averaging ~0.75–0.85 (Wadsworth et al., 2018). Visually dependent tasks, like the Clock drawing task, tended to have lower ICCs than verbally mediated tasks, but differences in scores across modalities were not statistically significant (Brearly et al., 2017). Importantly, several studies have shown that assessments administered by video could discriminate cognitively impaired from unimpaired older adults and were well tolerated by both groups (Alegret et al., 2021; Barton et al., 2011; Loh et al., 2007; Wadsworth et al., 2018). A few studies have also used video-based assessment techniques successfully with minority and underserved populations including rural Native American and rural Latino adults tested in Spanish or Portuguese (Castanho et al., 2016; Vahia et al., 2015; Wadsworth et al., 2016).

The COVID-19 pandemic created an urgency to put in place alternatives to in-person assessment, including telehealth and tele-neuropsychology. However, clinicians and investigators alike require validated assessment tools to conduct comprehensive cognitive exams remotely. Recent studies document the use of remote assessments in patients with epilepsy-related cognitive concerns (Tailby et al., 2020), pediatric neuropsychology patients (Lichtenstein et al., 2022; Salinas et al., 2020; Sherwood & MacDonald, 2020), Parkinson’s disease patients (York et al., 2021), and older adults. A review of remote administration of neuropsychological tests with older adults found strong evidence of its validity compared to face-to-face assessments (Marra et al., 2020). They also reported no differences between face-to-face and remote testing in mild-to-moderate AD but did report differences in more severe disease. While encouraging, a recent survey of neuropsychologists identified a need for improved norms, additional domain coverage, improved access to technology, and further validation studies (Rochette et al., 2021).

The Uniform Data Set Neuropsychological Battery, Version 3 (UDSv3) was developed by the Neuropsychology Work Group of the NIH-NIA Clinical Task Force in 2015 for use by a network of Alzheimer’s disease research centers (ADRCs). These measures, described in detail in Weintraub and colleagues (2018), were chosen to assess a broad range of cognitive domains, including global cognitive status, attention/working memory, verbal episodic memory, and language. To our knowledge, remote administration of the UDSv3 cognitive battery has not been systematically evaluated and reported. Given its broad use within the ADRC network and potential use beyond, the current study describes the development of a UDSv3 cognitive battery modified for telephone and video administration and provides preliminary data on the concordance with in-person cognitive testing and adjudicated cognitive outcomes. We also report user ratings of remote assessment compared with in person.

METHODS

Participants

We assessed the feasibility of remote administration of an adapted UDSv3 battery in a sample of 93 existing, English-speaking participants in the Wake Forest ADRC Clinical Core cohort who completed or were due to complete their annual cognitive assessment within 6 months of enrollment into the present study. We enrolled participants who, at their most recent in-person adjudications, were adjudicated to have MCI (amnestic single and multi-domain, and non-amnestic single and multi-domain presentations included; N = 35), Dementia (N = 11), or normal cognition (NC; N = 44). Three participants were classified as having “other cognitive status.” The race, gender, and educational background of these participants mirrored that of the larger ADRC Clinical Core. All participants gave informed consent to participate in these ADRC-related study procedures.

Cognitive Testing and Adaptation of UDSv3

All participants completed cognitive testing from their own homes via an HIPAA-compliant Zoom meeting, the link for which was provided via email to the participant or a trusted contact. Participants were able to use their own device if they had at-home Wi-Fi and a device capable of video interfacing with both audio and visual capabilities, such as a laptop computer, desktop computer, or tablet-type device. Use of smartphones was not permitted due to their small screen size. Individuals who did not have an acceptable at-home device and Wi-Fi, those who were unsure about their Wi-Fi access, and those who preferred not to use their own devices were given the option of using a study-provided tablet (GrandPad®). GrandPad® screens are ~5 in. × 9 in. and were the smallest devices used for this study. GrandPads® are designed specifically for use by older adults with few buttons and icons and come with a built-in cellular network and Zoom functionality. They do not require a Wi-Fi connection. Accessing the link for the Zoom meeting on this device required only three steps—turning the device on and clicking two buttons—which were explained to participants via telephone as well as via written and pictorial instructions shipped to participants with the device.

Cognitive tests were organized into a Core Battery and an Expanded Battery (Table 1). The “Core Battery” included tests from the UDSv3 measuring global cognitive function, memory, language, executive function, and mood that did not include visual stimuli and thus could be administered by phone (i.e., Benson Complex Figure, portions of the MoCA, Trail Making Test (TMT), and Multilingual Naming Test). An adapted version of the MoCA (Blind/Telephone MoCA; Nasredine, 2005; Nasredine, 2019; Wittich et al., 2010) that excludes visual items was added. The other tests within the Core Battery did not require modifications for remote administration and comprise the minimum cognitive data required for completion of a remote UDSv3 visit within an ADRC. To allow for a more comprehensive assessment of cognitive functioning, the Expanded Battery included all of the Core measures plus several additional tests of executive function (OTMT; Ricker et al., 1996; Ricker & Axelrod, 1994) and naming (Verbal Naming Test; Wynn et al., 2020; Yochim et al., 2015). To strengthen the assessment of episodic verbal memory recall, the Rey Auditory Verbal Learning Test (RAVLT; Lezak, 1997) was also added as a verbal learning task (Table 1). The RAVLT is not a standard part of the UDSv3 but is administered to all Wake Forest ADRC participants as part of their annual cognitive evaluations.

Cognitive raters were staff psychometrists within the Wake Forest ADRC experienced in administration of the UDSv3. Additional training was provided to orient the raters to non-UDSv3 assessments (e.g., Verbal Naming Test and OTMT) as well as issues unique to remote administration of neuropsychological assessment.

Adjudication Procedures

After remote testing was completed, participants’ cognitive status (NC, MCI, or Dementia) was adjudicated using identical procedures that are used with face-to-face administrations/adjudications. Adjudications were performed by a panel of neurologists and neuropsychologists (BS, SC, BW, & JB) who were blinded to adjudication status from participants’ corresponding most recent testing in the other modality. Adjudicators met regularly as a group to review cases and reach a consensus decision for each participant.

Participant and Staff Ratings of Remote Assessment

Likert-scale questions were developed to measure participants’ ratings of acceptability and pleasantness of each remote modality compared to their face-to-face administration and their preference for remote versus in-person sessions. Participants answered three questions on 1–10-point Likert scales regarding the simplicity of the visit (how challenging was it to use the phone/video technology for this visit?), their likelihood of selecting a remote visit over an in-person visit (how likely are you to choose a phone/video visit over coming into the clinic again in the future?), and their own perception of how validly the session measured their functioning (how confident are you that today’s test results are a valid sample of your memory and thinking?). Additionally, they reported how the difficulty and pleasantness of remote testing compared to in-person testing. Participants also provided information about the auditory quality, fatigue, and any technical problems. We further queried participants about what they liked and disliked about the process with an open-ended question. Additionally, cognitive testers provided ratings of perceived validity of the assessment by indicating whether they thought the assessment was valid, questionably valid, or invalid. If testing sessions were rated as questionably valid or invalid, raters were asked to provide the reason for compromised validity (hearing issues, distractions, interruptions, etc.).

Statistical Analysis

Participants (Table 2) completed either the core battery by video or telephone or the expanded battery by video or telephone. Data analyses were done using SAS and R. Data were checked for normality using scatterplots and deemed acceptable for subsequent analyses. Frequency counts were reported to assess participant’s ratings of their subjective experience and perceived validity of testing. Statistical analyses included paired samples t-tests, as well as Pearson’s correlations, to assess the relationship between scores obtained in person and remotely. Kappa coefficients (k) and raw percent agreement were computed between adjudication diagnoses obtained in person, which served as the index diagnosis, and the remote adjudication, which was the comparator (Gisev et al., 2013).

RESULTS

Sample Characteristics

Ninety-three individuals (mean age: 72.8 years; education: 15.6 years; 72% female; 84% White) enrolled in our ADRC were included. Their most recent index-adjudicated cognitive status was NC (N = 44), MCI (N = 35), mild Dementia (N = 11), or other (N = 3).

Feasibility and Preference of Administration Modality

Seventy-four of 93 participants completed user perception questionnaires about their experience with remote assessment. Across both batteries (Core and Expanded) and both administration modes (telephone and video), ~75% of participants rated the level of difficulty completing the remote cognitive assessment the same as the in-person testing, 16% rated the remote assessment as less difficult, and only 8% rated the remote assessment as more difficult. All but one of the reports of greater difficulty came from a telephone assessment with the core battery. Regarding pleasantness, 74% of participants felt the remote assessment was as pleasant as the in-person assessment, 20% felt it was more pleasant, and only 6% felt it was less pleasant. Of the four individuals who rated the experience as less pleasant, three received the telephone battery. When asked to rate how likely they were to choose the remote assessment over an in-person visit, (10 = “extremely likely,” 1 = “not likely at all”), the median ratings were 9.5 and 8, for video and telephone, respectively. The perceived validity of testing results by the participants and examiners was high. Participants had median ratings of 9 and 8 for video and telephone, respectively. Examiners rated 92% of the remotely collected video data as valid and 87.5% of the telephone data as valid (Table 3). Eight percent and 12%, respectively, were judged “questionably valid.” Questionably valid ratings occurred for four MCI participants, two NC participants, and two early Dementia participants. Three were deemed questionably valid because of suspicion that the participant used aids during testing (e.g., paper and pencil). The other reasons were not specific to remote assessment (e.g., examiner administration error). Qualitative comments from participants showed that participants liked the convenience of remote assessment but disliked occasional hearing difficulties and distractions in their homes, and some missed human contact with staff.

Concordance Between Test Scores

Video administration showed a somewhat stronger and more consistent pattern of correlations with face-to-face administrations than did telephone administration. This was also true for the supplemental tests—the MoCA, VNT, and Oral TMT-Part B—which all correlated equally well or better with video than telephone modality. This pattern was particularly true for the RAVLT Trial 5, for which the video administrations (r = .85; p < .05) correlated noticeably better with in-person administrations than the telephone administration did (r = .39, p < .05). The Oral TMT-Part A correlated weakly with TMT-A when administered by either telephone (r = .28; p > .05) or video (r = .30; p > .05), likely due to the dissimilarity of TMT-A and OTMT-A. Overall, correlations between face-to-face and remote assessment were mostly moderate to high (r = .3 to .8) (Table 4). Paired-samples t-tests (Table 5; in person—phone; in person—video) revealed no significant differences, regardless of modality of administration (p > .05) on tests of phonemic fluency, semantic fluency, number span forwards and backwards, and Craft Immediate recall—paraphrase and verbatim. Participants tested by phone tended to recall one to two more words (of 15) on the short and long delay variables on the RAVLT (p < .05), though this pattern held true only for long delay recall on video testing. No significant differences were noted on Craft—delayed recall (verbatim or paraphrase) for phone testing; on video testing, participants tended to earn 1 or 2 more points (out of 25 or 44 possible points, respectively) compared to in-person testing (p < .05).

Concordance of Cognitive Status

Overall, there was substantial agreement between adjudications conducted with in-person and remote data (79% agreement, k = .64), with slightly better agreement between video and in-person (82%, k = .68) versus phone and in person (76%, k = .61). Agreement was better for CN participants (86%) than for participants with cognitive impairment (MCI: 77%; Dementia: 73%), though there were only slight differences between MCI and Dementia adjudication agreements. When adjudication was made based on video data versus data obtained in person (index adjudication), three Dementia cases were classified as MCI (27%); seven MCIs were classified as normal (20%) and one classified as Dementia (3%); five normals were classified as MCI (11%) and one as other (2%); and two others were classified as MCI (67%) and one as normal (33%) (Table 6).

DISCUSSION

This study assessed the utility of a modified UDSv3 research cognitive battery adapted for remote administration by telephone or video in older adults with and without cognitive impairment within an ADRC.

We found generally moderate-to-high correlations between the same tests administered across in-person and remote modalities, though there was some variability by test and modality. We also found that there was generally good concordance between adjudicated diagnostic classifications based on either in-person or remote assessments. Participants rated remote assessment as equally difficult, pleasant, and valid relative to in-person assessment. However, it is worth nothing that individuals with MCI and Dementia may have difficulty accurately remembering their in-person testing well enough to compare it with their remote testing experience.

There was strong correspondence between remotely administered tests and in-person tests, though these relationships were slightly stronger for video administrations than they were for telephone administrations, and some differences (one to two points) were observed on tests of memory recall variables, most notably for verbal list learning tests administered over the phone. The slightly better performance on remotely administered verbal list learning tests may be due to multiple reasons; for instance, the order of test modality was not randomized for this study, so it may reflect practice effects. Alternatively, because participants tested by phone are unobserved, they may not strictly adhere to standard test protocols and instructions that require testing be done without the use of paper and pencil. The fact that these score differences on phone administrations were observed on the list learning test, which is much more easily aided by paper and pencil, appears to provide some possible support for the latter hypothesis. The Trail Making Tests, particularly TMT-A and OTMT-A, also correlated less well, which is consistent with past literature (Kaemmerer & Riordan, 2016) and unsurprising given the notable discrepancy between the oral and written versions of this task. The OTMT-A task of counting aloud from 1 to 25 is a highly overlearned task and requires none of the visuomotor functioning required by the written task. Though TMT-B and OTMT-B appear similar in that they both require set shifting, the cognitive abilities required to successfully shift-set likely vary as a consequence of how those shifts are cognitively and/or motorically executed, with the latter requiring mental-shifting more heavily tasking working memory circuits. Lastly, adjudication agreement was high between in-person data and remote data, particularly for individuals without cognitive impairment.

These findings replicate existing work demonstrating that remotely administered research cognitive assessments are generally well tolerated by older adults and can discriminate between those with and without cognitive impairment (Alegret et al., 2021; Barton et al., 2011; Loh et al., 2007; Wadsworth et al., 2018). They extend these findings by demonstrating that the widely used UDSv3 cognitive test battery translates adequately into remote administration and results in non-discordant adjudication outcomes.

These results have several important implications. Comparability of remote assessment to in-person assessment can help centers expand recruitment to traditionally under-studied populations—this is especially relevant to those centers that focus on the oldest old, rural, and underserved populations and participants for whom transportation to research centers is difficult. Inconvenience and limited time for travel have been documented as barriers to clinical research participation, particularly for underserved groups and those in rural communities (Friedman et al., 2015; Williams et al., 2010), and remote assessment has the potential to partially alleviate these barriers. In fact, this potential has already been demonstrated for veterans participating in telehealth programs implemented by the VA care network. For example, veterans participating in a sleep telemedicine program have reported increased care access and improvements in sleep quality (Nicosia et al., 2021). Additionally, the majority of veterans surveyed about healthcare visits via VA-distributed tablets have reported video appointments to be preferable or the same as in-person visits (Slightam et al., 2020). There is a pressing need for more accessible and representative research in the Alzheimer’s disease field (Babulal et al., 2019) which could be partially addressed with remote assessments.

One concern regarding remote assessments with older adults is limited technological literacy. Though there are generational differences in technology use and competency, older adults are increasingly using technology (Anderson & Perrin, 2017). There is mounting evidence that they are quite capable and willing to use technological devices for research studies (Joddrell & Astell, 2016; Nicosia et al., 2022). Providing participants with user-friendly, pre-programmed tablet devices can reduce technical difficulties and extend access to those who do not have video-interfacing equipment or Wi-Fi available via their own devices. Devices such as the GrandPad® have been created specifically for use by older adults with adaptations to mitigate hearing and vision difficulty (e.g., easy-to-adjust volume and oversized text and buttons). These devices can be pre-loaded with cellular network access to allow for use by older adults without home internet. The use of video-devices allows for closer observation of participant behaviors during the testing session. A follow-up randomized study comparing video-administered to in-person assessment with the UDSv3 in a large sample of ADRC participants with no impairment, MCI, and mild Dementia is currently underway with the support of NIH funding.

Limitations, Challenges, and Future Directions

There are several limitations of the present study. As noted, some participant “satisfaction” ratings rely on their memory of in-person testing, which may be imprecise, especially in those with MCI or mild Dementia. The small sample size and its limited diversity constrain generalization. Though there was representation of Black participants, there was no representation comparable to census benchmarks, nor was there representation of additional racial/ethnic minorities which is in part a reflection of the demographic makeup of the larger Wake Forest ADRC Clinical Cohort. Sampling from a more diverse population would strengthen the generalizability of the results and include representation of populations for which this burden is often greatest (rural, underserved, and underrepresented groups of individuals; Henning-Smith, 2020; Ruprecht et al., 2021). Larger studies of the remote neuropsychological assessment with a more diverse cohort of older adults could allow for the development of both normative data and crosswalks to improve the interpretability at the group level and increase the utility of these assessments in the adjudication of cognitive status.

Though, in this study, many of the remotely administered assessments correlated well with their in-person counterparts, one clear exception was TMT-A. This is consistent with other research demonstrating weak cross-modal correlations and poor ability to discriminate between groups for OTMT-A suggesting that the oral and written versions of this task are fundamentally different from one another and likely measure different cognitive constructs (Bastug et al., 2013; Jaywant et al., 2018). If remote assessments that do not clearly correspond to well-validated in-person assessments are to continue to be administered, it will be important to assess what cognitive abilities they do capture as well as to develop norms or test “crosswalks” that allow for interpretation of this data moving forward.

An additional challenge with remote neuropsychological testing in general is the possibility of non-adherence to standardized test protocols, namely, not following standard test instructions (e.g., utilizing external aids such as computers, clocks, or watches; transcribing words on tests that are meant to be completed orally) that may affect tasks like list-learning paradigms, for example. Modifying testing scripts to emphasize the importance of not using aids may help. Video administration has an advantage over phone administration because it allows the rater to better monitor the participant’s behavior during testing. Video administrations also allow raters more insight into other household distractions and the general test environment of the participant.

A lack of standardization in the test–retest window and balance in the remote assignment groups may have influenced the results. On average, there were ~4 months between participants’ two assessment sessions, with an outer time of ~9 months. This long interval could allow for changes in cognitive testing performance due to factors other than the testing modality, including progression of disease, and this potentially influenced test scores and the adjudication process. Further, practice effects on the second exam may have also affected scores, as order of administration was not randomized. In the future, it would be ideal for studies comparing remote neuropsychological assessments to in-person assessments to standardize the test–retest window and randomize participants from each cognitive strata to receive remote testing either before or after in-person testing. It will also be important to examine longitudinal data to assess how comparable the methods are in detecting cognitive changes over time.

CONCLUSION

This study offers preliminary evidence that remote cognitive assessments are feasible for older adults, including individuals with normal cognition and those with milder forms of cognitive impairment, though there are important nuances to consider such as selection of tests and modality. Though the COVID-19 pandemic was the initial impetus for increased interest in remote assessment, these findings have additional implications for reaching underrepresented and underserved communities for which in-person testing burden is often highest.

FUNDING

This work was supported by the National Institute of Health (P30 AG049638 and AG049638-05S1).

CONFLICT OF INTEREST

None declared.

REFERENCES