https://orcid.org/0000-0001-8619-9564
Abstract
The COVID-19 pandemic induced rapid adoption of telemedicine services for neuro-oncology patients at an increased risk of infection. Neuropsychological assessment is important to neuro-oncology care yet challenging to complete outside of a structured testing environment. Teleneuropsychology (TNP) has been explored in limited populations and proven feasible and reliable. Conducting TNP visits directly to patients’ home (DTH) had minimal prior study.
We used two voluntary surveys to examine acceptance (patients) and feasibility (providers) of DTH-TNP at two regionally diverse medical institutions providing neuropsychological services to neuro-oncology patients from April to September 2020.
A total of 119 patients were scheduled during the study period, 79 of whom completed neuropsychological testing via DTH-TNP. Neuropsychology providers completed surveys on 68 of these encounters (86%). In 98% of cases, neuropsychologists were able to achieve or partially achieve the individually defined goals of their assessment. Common problems reported included patient dysregulation (16%) and slow/unreliable internet (15%). Of the 52 patients who responded, 98% were satisfied with the DTH-TNP experience, and 92% would recommend the virtual visit to others. All respondents felt understood by the examiner (100%) and the majority denied technical difficulties (90%), communication challenges (94%), or privacy concerns (98%). Patients reported reduced risk of infection and saved travel time as favorable aspects of DTH-TNP.
These preliminary results suggest neuro-oncology patients find DTH-TNP acceptable and neuropsychologists find it a feasible practice, while also recognizing its limitations. Results suggest that further study of DTH-TNP (eg, reliability, validity) for neuro-oncology patients is warranted.
First recorded in December 2019,1 the World Health Organization declared the novel coronavirus (COVID-19) a global pandemic on March 11, 2020. The highly contagious virus severely impacted medical services worldwide. Healthcare systems focused efforts on understanding and treating the virus as physical distancing and stay-at-home orders were imposed to reduce transmission. Appointments for outpatient healthcare procedures were markedly reduced around the world to further these efforts.
Nonessential outpatient appointments deferred in mid-March reemerged at a slowed and monitored pace. Medical institutions have taken precautions when scheduling in-person visits (ie, minimizing the number of consumers in the waiting area, additional sanitization procedures between patients, screening for symptoms before scheduling, and use of personal protective equipment by staff and patients). Neuro-oncology patients, who are on treatments that may be immunosuppressant, must take extra precautions2 to mitigate the risk of infection. Guidelines for neuro-oncology providers incorporated stringent practices,3,4 including postponing chemotherapy if a patient was infected with COVID-19, yet continuing with radiation treatment in mild COVID-19 cases (see Mohile et al.3 for review), and using remote visits when possible. Similarly, neuropsychologists providing evaluations and treatment for neuro-oncology patients were forced to choose between ceasing care or finding alternative methods to deliver services.
Telemedicine, which originated in the early 1960s,5 had only recently been integrated into the US health care provision prior to the COVID-19 pandemic. A 2019 survey indicated 66% of Americans were open to trying telehealth, yet only 8% of those surveyed had used remote services.6 Since March 2020, 42% of adults have used telehealth services with 82% endorsing satisfaction with the experience.7 Video teleconference administration of neurocognitive tests, teleneuropsychology (TNP), has previously been used to provide service to patients who reside in rural regions and where access to hospitals or clinical care is cumbersome.8–10 This method of care has proven feasible in both large11,12 and smaller controlled studies.8,13–16 In fact, Parikh et al.17 found that 95% of older adults with a cognitive impairment, who completed evaluations remotely, reported positive experiences with TNP. However, such studies examined assisted-TNP, in which a neuropsychologist at a medical center interacts with a patient in a clinical space at a remote location, ensuring adequate internet connectivity and an environment that limits distraction. In contrast, during a direct-to-home-teleneuropsychology (DTH-TNP) visit, the neuropsychologist instructs the patient to adapt their environment for testing, such as securing a private and quiet space, having paper and pencil readily available, ensuring camera access, and assessing the internet connection before beginning any assessments (ie, during the clinical interview). Neuro-oncology patients require continuity of care despite the unique challenge of the COVID-19 pandemic; therefore, neuropsychologists involved in cancer care are propelled to investigate the acceptance and feasibility of DTH-TNP administration, which had not previously been studied with neuro-oncology patients.
To address the gaps in the literature, we collected data from two regionally diverse academic medical centers and report preliminary findings regarding patient acceptance and provider feasibility of DTH-TNP in a neuro-oncology patient population. Based on the promising findings in the assisted-TNP literature, we hypothesized that patients would endorse high acceptance rates and that neuropsychologists practicing in a neuro-oncology clinic would attest to the feasibility of DTH-TNP.
Materials and Methods
Institutions
The Stephen and Catherine Pappas Center for Neuro-Oncology at Massachusetts General Hospital (MGH) in Boston, Massachusetts and the Massey Cancer Center Neuro-Oncology Clinic at Virginia Commonwealth University (VCU) in Richmond, Virginia participated in this Quality Improvement Initiative. Both neuropsychology clinics provide evaluations for neuro-oncology patients including those with primary or secondary brain tumors or those experiencing neurocognitive symptoms related to non-central nervous system cancer or cancer therapy. Both neuropsychology services were closed at the start of the pandemic (MGH closed on March 13, 2020; VCU closed on March 16, 2020), and patients were offered DTH-TNP appointments upon re-opening of the clinics virtually. This project followed the policies laid out by each site’s Institutional Review Board for Quality Improvement Initiatives.
Triage Process
Patients whose neuropsychological evaluations were canceled due to COVID-19 closure or were referred shortly thereafter underwent a triage process. At MGH, referrals were triaged by the neuropsychologist (M.W.P.) and program coordinator (G.V.Z.) to determine if a virtual visit was possible (eg, whether the patient had access to videoconferencing technology and had a quiet and private space to perform the visit). Referral questions were evaluated by the neuropsychologist for urgency and initial video visits were scheduled beginning on March 31, 2020. Patients in the VCU practice were contacted to determine whether they had access to videoconferencing technology and an adequate space to complete testing. If so, they were offered the option of DTH-TNP, with the first virtual visit on June 15, 2020. Otherwise, they were deferred to in-person testing, beginning in August 2020. All patients underwent an informed consent process reviewing the novel nature of this clinical service and were seen for an initial interview, concurrent testing, or scheduled in-person at a later date (MGH). Patients were instructed to be in a quiet room with minimal distractions (eg, turn off phones, remove pets, etc).
Surveys
Two surveys were conducted for this study, one of which was directed at providers, inquiring about feasibility of the assessment, and the other of which was directed at patients to inquire about acceptability of the DTH-TNP experience. A preexisting patient survey17 was modified for DTH-TNP and implemented using the Research Electronic Data Capture (REDCap) system (see Table 1).18 After completion of the DTH-TNP appointment, participants were invited to complete the voluntary satisfaction survey via email or message through the videoconferencing platform. Steps were taken to reduce any sense of coercion: no patient identifying information was included and surveys were sent by psychometrists or graduate students instead of the neuropsychological provider. Following the clinical interview and testing session, the neuropsychological provider had the option to complete a short survey to evaluate the feasibility of the DTH-TNP appointment (see Table 1). Patient satisfaction was examined using multiple questions rated on a five-point Likert scale (see Table 2) and included space for free-text responses.
Questions Administered as Provider Survey
| Provider Survey |
|---|
| Age of patient ___ Sex of patient (M, F, Nonbinary) Duration of total evaluation (minutes) |
| 1. This evaluation used: |
| ___ Telephone only |
| ___ Video/audio via computer |
| ___ Combination of telephone and video/audio session |
| 2. This evaluation included (choose all that apply): |
| ___ Interview with patient |
| ___ Interview with family |
| ___ Screening test (eg, a mental status screen) |
| ___ Formal neuropsychological testing |
| 3. Did you use testing help in this session? |
| ___ Yes |
| ___ No |
| 4. Did you break up the evaluation into more than one session? |
| ___ Yes |
| ___ No |
| 5. What strategies were used to optimize the testing session (choose all that apply)? |
| ___ Brief instructions |
| ___ Frequent breaks |
| ___ Having another person in the room with patient |
| ___ None or N/A |
| 6. Other strategies used to optimize the testing session: ______________________________________ |
| 7. What were some challenges you had during the session (choose all that apply)? |
| ___ Slow or unreliable internet or phone connection |
| ___ Audio or visual problems with the technology |
| ___ Hearing or vision problems of the patient |
| ___ Interruptions |
| ___ Difficulty understanding instructions |
| ___ Patient self-regulation difficulties (eg, attention or behavior) |
| ___ None or N/A |
| 8. Other challenges you had during the session:________________________________ |
| 9. How did you handle these challenges: _____________________________________ |
| 10. Did you feel that you were effectively able to achieve the goal of this evaluation (eg, interview, screening, formal assessment)? |
| ___ Yes___ No___ Partially |
| 11. Comments regarding your ability to achieve the goal of this evaluation: ____________ |
| 12. What would you do differently during the next session? ______________________________________ |
| Provider Survey |
|---|
| Age of patient ___ Sex of patient (M, F, Nonbinary) Duration of total evaluation (minutes) |
| 1. This evaluation used: |
| ___ Telephone only |
| ___ Video/audio via computer |
| ___ Combination of telephone and video/audio session |
| 2. This evaluation included (choose all that apply): |
| ___ Interview with patient |
| ___ Interview with family |
| ___ Screening test (eg, a mental status screen) |
| ___ Formal neuropsychological testing |
| 3. Did you use testing help in this session? |
| ___ Yes |
| ___ No |
| 4. Did you break up the evaluation into more than one session? |
| ___ Yes |
| ___ No |
| 5. What strategies were used to optimize the testing session (choose all that apply)? |
| ___ Brief instructions |
| ___ Frequent breaks |
| ___ Having another person in the room with patient |
| ___ None or N/A |
| 6. Other strategies used to optimize the testing session: ______________________________________ |
| 7. What were some challenges you had during the session (choose all that apply)? |
| ___ Slow or unreliable internet or phone connection |
| ___ Audio or visual problems with the technology |
| ___ Hearing or vision problems of the patient |
| ___ Interruptions |
| ___ Difficulty understanding instructions |
| ___ Patient self-regulation difficulties (eg, attention or behavior) |
| ___ None or N/A |
| 8. Other challenges you had during the session:________________________________ |
| 9. How did you handle these challenges: _____________________________________ |
| 10. Did you feel that you were effectively able to achieve the goal of this evaluation (eg, interview, screening, formal assessment)? |
| ___ Yes___ No___ Partially |
| 11. Comments regarding your ability to achieve the goal of this evaluation: ____________ |
| 12. What would you do differently during the next session? ______________________________________ |
Questions Administered as Provider Survey
| Provider Survey |
|---|
| Age of patient ___ Sex of patient (M, F, Nonbinary) Duration of total evaluation (minutes) |
| 1. This evaluation used: |
| ___ Telephone only |
| ___ Video/audio via computer |
| ___ Combination of telephone and video/audio session |
| 2. This evaluation included (choose all that apply): |
| ___ Interview with patient |
| ___ Interview with family |
| ___ Screening test (eg, a mental status screen) |
| ___ Formal neuropsychological testing |
| 3. Did you use testing help in this session? |
| ___ Yes |
| ___ No |
| 4. Did you break up the evaluation into more than one session? |
| ___ Yes |
| ___ No |
| 5. What strategies were used to optimize the testing session (choose all that apply)? |
| ___ Brief instructions |
| ___ Frequent breaks |
| ___ Having another person in the room with patient |
| ___ None or N/A |
| 6. Other strategies used to optimize the testing session: ______________________________________ |
| 7. What were some challenges you had during the session (choose all that apply)? |
| ___ Slow or unreliable internet or phone connection |
| ___ Audio or visual problems with the technology |
| ___ Hearing or vision problems of the patient |
| ___ Interruptions |
| ___ Difficulty understanding instructions |
| ___ Patient self-regulation difficulties (eg, attention or behavior) |
| ___ None or N/A |
| 8. Other challenges you had during the session:________________________________ |
| 9. How did you handle these challenges: _____________________________________ |
| 10. Did you feel that you were effectively able to achieve the goal of this evaluation (eg, interview, screening, formal assessment)? |
| ___ Yes___ No___ Partially |
| 11. Comments regarding your ability to achieve the goal of this evaluation: ____________ |
| 12. What would you do differently during the next session? ______________________________________ |
| Provider Survey |
|---|
| Age of patient ___ Sex of patient (M, F, Nonbinary) Duration of total evaluation (minutes) |
| 1. This evaluation used: |
| ___ Telephone only |
| ___ Video/audio via computer |
| ___ Combination of telephone and video/audio session |
| 2. This evaluation included (choose all that apply): |
| ___ Interview with patient |
| ___ Interview with family |
| ___ Screening test (eg, a mental status screen) |
| ___ Formal neuropsychological testing |
| 3. Did you use testing help in this session? |
| ___ Yes |
| ___ No |
| 4. Did you break up the evaluation into more than one session? |
| ___ Yes |
| ___ No |
| 5. What strategies were used to optimize the testing session (choose all that apply)? |
| ___ Brief instructions |
| ___ Frequent breaks |
| ___ Having another person in the room with patient |
| ___ None or N/A |
| 6. Other strategies used to optimize the testing session: ______________________________________ |
| 7. What were some challenges you had during the session (choose all that apply)? |
| ___ Slow or unreliable internet or phone connection |
| ___ Audio or visual problems with the technology |
| ___ Hearing or vision problems of the patient |
| ___ Interruptions |
| ___ Difficulty understanding instructions |
| ___ Patient self-regulation difficulties (eg, attention or behavior) |
| ___ None or N/A |
| 8. Other challenges you had during the session:________________________________ |
| 9. How did you handle these challenges: _____________________________________ |
| 10. Did you feel that you were effectively able to achieve the goal of this evaluation (eg, interview, screening, formal assessment)? |
| ___ Yes___ No___ Partially |
| 11. Comments regarding your ability to achieve the goal of this evaluation: ____________ |
| 12. What would you do differently during the next session? ______________________________________ |
Questions Administered as Patient Survey
| Patient Survey | ||||
|---|---|---|---|---|
| This survey was completed by: | ||||
| ___ The person who completed the assessment (examinee) | ||||
| ___ A parent/guardian of the examinee | ||||
| ___ Another family member/caregiver of the examinee | ||||
| Age of the examinee | ||||
| ___ 0-5 years ___ 6-11 ___ 12-17 ___ 18-30 ___ 31-50 ___ 51-70 ___ 70+ | ||||
| Clinician’s name (optional): _________ | ||||
| Approximate duration of appointment (in minutes): _________ | ||||
| My appointment included (check all that apply): | ||||
| ___ An interview with the neuropsychologist | ||||
| ___ Testing (eg, of memory, attention, language, or other abilities) | ||||
| ___ Feedback on test results | ||||
| For the following six questions, please select the answer that best represents how you felt about today’s evaluation: | ||||
| 1 | 2 | 3 | 4 | 5 |
| Strongly disagree | Strongly agree | |||
| 1. Overall, I was satisfied with the virtual neuropsychology assessment. | ||||
| 2. There were no technical problems with conducting this virtual session. | ||||
| 3. There were no problems with communication during the virtual testing session. | ||||
| 4. I was not concerned about my privacy during the virtual session. | ||||
| 5. I felt that the examiner understood me and responded promptly to my questions during the virtual session. | ||||
| 6. I would recommend virtual neuropsychology appointments to others. | ||||
| 7. Do you think an office assessment, as opposed to this virtual visit, would (please check all that apply): | ||||
| ___ Make it easier to express my concerns and symptoms to the clinician | ||||
| ___ Allow better communication with the examiner | ||||
| ___ Promote a better personal connection with the examiner | ||||
| ___ Allow a more extensive evaluation | ||||
| 8. What did you like about virtual assessment: (please check all that apply)? | ||||
| ___ Less anxious than I might be with an examiner in the room | ||||
| ___ Reduced risk of infection due to one less visit to the hospital | ||||
| ___ Easier to concentrate without examiner in the room | ||||
| ___ Saved time traveling to a hospital for this type of visit | ||||
| 9. If you needed to see a doctor for this type of testing, how far would you be willing to drive/ride before choosing virtual assessment (please mark only one answer): | ||||
| ___ Less than 1 hour | ||||
| ___ 1-3 hours | ||||
| ___ 3-6 hours | ||||
| ___ I would drive/ride as far as it takes and spend the night, if needed | ||||
| ___ I would prefer the virtual appointment | ||||
| 10. What could improve the virtual neuropsychology assessment experience? Please add any other comments or feedback here. _______________________________________ |
| Patient Survey | ||||
|---|---|---|---|---|
| This survey was completed by: | ||||
| ___ The person who completed the assessment (examinee) | ||||
| ___ A parent/guardian of the examinee | ||||
| ___ Another family member/caregiver of the examinee | ||||
| Age of the examinee | ||||
| ___ 0-5 years ___ 6-11 ___ 12-17 ___ 18-30 ___ 31-50 ___ 51-70 ___ 70+ | ||||
| Clinician’s name (optional): _________ | ||||
| Approximate duration of appointment (in minutes): _________ | ||||
| My appointment included (check all that apply): | ||||
| ___ An interview with the neuropsychologist | ||||
| ___ Testing (eg, of memory, attention, language, or other abilities) | ||||
| ___ Feedback on test results | ||||
| For the following six questions, please select the answer that best represents how you felt about today’s evaluation: | ||||
| 1 | 2 | 3 | 4 | 5 |
| Strongly disagree | Strongly agree | |||
| 1. Overall, I was satisfied with the virtual neuropsychology assessment. | ||||
| 2. There were no technical problems with conducting this virtual session. | ||||
| 3. There were no problems with communication during the virtual testing session. | ||||
| 4. I was not concerned about my privacy during the virtual session. | ||||
| 5. I felt that the examiner understood me and responded promptly to my questions during the virtual session. | ||||
| 6. I would recommend virtual neuropsychology appointments to others. | ||||
| 7. Do you think an office assessment, as opposed to this virtual visit, would (please check all that apply): | ||||
| ___ Make it easier to express my concerns and symptoms to the clinician | ||||
| ___ Allow better communication with the examiner | ||||
| ___ Promote a better personal connection with the examiner | ||||
| ___ Allow a more extensive evaluation | ||||
| 8. What did you like about virtual assessment: (please check all that apply)? | ||||
| ___ Less anxious than I might be with an examiner in the room | ||||
| ___ Reduced risk of infection due to one less visit to the hospital | ||||
| ___ Easier to concentrate without examiner in the room | ||||
| ___ Saved time traveling to a hospital for this type of visit | ||||
| 9. If you needed to see a doctor for this type of testing, how far would you be willing to drive/ride before choosing virtual assessment (please mark only one answer): | ||||
| ___ Less than 1 hour | ||||
| ___ 1-3 hours | ||||
| ___ 3-6 hours | ||||
| ___ I would drive/ride as far as it takes and spend the night, if needed | ||||
| ___ I would prefer the virtual appointment | ||||
| 10. What could improve the virtual neuropsychology assessment experience? Please add any other comments or feedback here. _______________________________________ |
Questions Administered as Patient Survey
| Patient Survey | ||||
|---|---|---|---|---|
| This survey was completed by: | ||||
| ___ The person who completed the assessment (examinee) | ||||
| ___ A parent/guardian of the examinee | ||||
| ___ Another family member/caregiver of the examinee | ||||
| Age of the examinee | ||||
| ___ 0-5 years ___ 6-11 ___ 12-17 ___ 18-30 ___ 31-50 ___ 51-70 ___ 70+ | ||||
| Clinician’s name (optional): _________ | ||||
| Approximate duration of appointment (in minutes): _________ | ||||
| My appointment included (check all that apply): | ||||
| ___ An interview with the neuropsychologist | ||||
| ___ Testing (eg, of memory, attention, language, or other abilities) | ||||
| ___ Feedback on test results | ||||
| For the following six questions, please select the answer that best represents how you felt about today’s evaluation: | ||||
| 1 | 2 | 3 | 4 | 5 |
| Strongly disagree | Strongly agree | |||
| 1. Overall, I was satisfied with the virtual neuropsychology assessment. | ||||
| 2. There were no technical problems with conducting this virtual session. | ||||
| 3. There were no problems with communication during the virtual testing session. | ||||
| 4. I was not concerned about my privacy during the virtual session. | ||||
| 5. I felt that the examiner understood me and responded promptly to my questions during the virtual session. | ||||
| 6. I would recommend virtual neuropsychology appointments to others. | ||||
| 7. Do you think an office assessment, as opposed to this virtual visit, would (please check all that apply): | ||||
| ___ Make it easier to express my concerns and symptoms to the clinician | ||||
| ___ Allow better communication with the examiner | ||||
| ___ Promote a better personal connection with the examiner | ||||
| ___ Allow a more extensive evaluation | ||||
| 8. What did you like about virtual assessment: (please check all that apply)? | ||||
| ___ Less anxious than I might be with an examiner in the room | ||||
| ___ Reduced risk of infection due to one less visit to the hospital | ||||
| ___ Easier to concentrate without examiner in the room | ||||
| ___ Saved time traveling to a hospital for this type of visit | ||||
| 9. If you needed to see a doctor for this type of testing, how far would you be willing to drive/ride before choosing virtual assessment (please mark only one answer): | ||||
| ___ Less than 1 hour | ||||
| ___ 1-3 hours | ||||
| ___ 3-6 hours | ||||
| ___ I would drive/ride as far as it takes and spend the night, if needed | ||||
| ___ I would prefer the virtual appointment | ||||
| 10. What could improve the virtual neuropsychology assessment experience? Please add any other comments or feedback here. _______________________________________ |
| Patient Survey | ||||
|---|---|---|---|---|
| This survey was completed by: | ||||
| ___ The person who completed the assessment (examinee) | ||||
| ___ A parent/guardian of the examinee | ||||
| ___ Another family member/caregiver of the examinee | ||||
| Age of the examinee | ||||
| ___ 0-5 years ___ 6-11 ___ 12-17 ___ 18-30 ___ 31-50 ___ 51-70 ___ 70+ | ||||
| Clinician’s name (optional): _________ | ||||
| Approximate duration of appointment (in minutes): _________ | ||||
| My appointment included (check all that apply): | ||||
| ___ An interview with the neuropsychologist | ||||
| ___ Testing (eg, of memory, attention, language, or other abilities) | ||||
| ___ Feedback on test results | ||||
| For the following six questions, please select the answer that best represents how you felt about today’s evaluation: | ||||
| 1 | 2 | 3 | 4 | 5 |
| Strongly disagree | Strongly agree | |||
| 1. Overall, I was satisfied with the virtual neuropsychology assessment. | ||||
| 2. There were no technical problems with conducting this virtual session. | ||||
| 3. There were no problems with communication during the virtual testing session. | ||||
| 4. I was not concerned about my privacy during the virtual session. | ||||
| 5. I felt that the examiner understood me and responded promptly to my questions during the virtual session. | ||||
| 6. I would recommend virtual neuropsychology appointments to others. | ||||
| 7. Do you think an office assessment, as opposed to this virtual visit, would (please check all that apply): | ||||
| ___ Make it easier to express my concerns and symptoms to the clinician | ||||
| ___ Allow better communication with the examiner | ||||
| ___ Promote a better personal connection with the examiner | ||||
| ___ Allow a more extensive evaluation | ||||
| 8. What did you like about virtual assessment: (please check all that apply)? | ||||
| ___ Less anxious than I might be with an examiner in the room | ||||
| ___ Reduced risk of infection due to one less visit to the hospital | ||||
| ___ Easier to concentrate without examiner in the room | ||||
| ___ Saved time traveling to a hospital for this type of visit | ||||
| 9. If you needed to see a doctor for this type of testing, how far would you be willing to drive/ride before choosing virtual assessment (please mark only one answer): | ||||
| ___ Less than 1 hour | ||||
| ___ 1-3 hours | ||||
| ___ 3-6 hours | ||||
| ___ I would drive/ride as far as it takes and spend the night, if needed | ||||
| ___ I would prefer the virtual appointment | ||||
| 10. What could improve the virtual neuropsychology assessment experience? Please add any other comments or feedback here. _______________________________________ |
Assessment
Prior to the COVID-19 pandemic, neuropsychological test procedures had not been validated using DTH-TNP methodology. Guidelines for test procedures were developed and published by a collaboration of neuropsychology professional groups, the Inter-Organizational Practice Committee (IOPC).19 Neuropsychological tests were selected by the attending neuropsychologists based on the IOPC guidance and existing literature regarding TNP20,21 to address the referral question and optimize validity of remote assessment, acknowledging that the literature demonstrating validity of these tests used assisted-TNP methodology.8,19 Certain tests, which require manipulable objects, cannot be administered via DTH-TNP, such as tests of motor function that involve pegboards, or tests that require use of a specific computer program that cannot be operated remotely. Neuropsychologists used clinical judgment to substitute similar tests whenever possible and acknowledged the limitation of the virtual assessment batteries when relevant. To record all nonverbal measures (eg, Clock Drawing), the patient was asked to hold up the design for the neuropsychologist to view and take a screenshot of the image. To preserve test security, any written or drawn items that contained protected information (eg, visual memory test designs) were destroyed by patients at the end of the testing session. A complete list of tests used at the two sites is included in Table 3. All testing visits were conducted using computer or tablet. Encrypted videoconference technology19,22 was used. At MGH, this was initially a stand-alone platform, but later transitioned to software within the patient’s medical record portal; providers similarly accessed the link through their electronic medical record (EMR). Some assessments were administered from clinician’s home to patient’s home, some from clinician’s office to patient’s home.
Complete List of Neuropsychological Tests That Attending Neuropsychologist(s) Were Able to Access for DTH-TNP Based on Inter-Organizational Practice Committee19 Guidance
| Domain | Test(s) |
|---|---|
| Intellectual ability | WAIS-IVa; ranged from subtests to full scale |
| RBANSb; ranged from subtests to full scale | |
| Estimated premorbid intelligence | Test of Premorbid Functioning |
| Language | Boston Naming Test |
| Auditory Naming Test | |
| Attention/working memory | WAIS-IVa; Working Memory |
| Processing speed | Oral Trail Making Test, Part A |
| Symbol Digit Modalities Test—Oral | |
| DKEFSc Color Naming | |
| DKEFSc Color Reading | |
| Executive function | Oral Trail Making Test, Part B |
| Controlled Oral Word Association Test | |
| DKEFSc Color/Word Inhibition | |
| DKEFSc Color/Word Inhibition/Switching | |
| Verbal memory | HVLT-Rd Total Recall |
| HVLT-Rd Delayed Recall | |
| HVLT-Rd Retention | |
| Visual memory | BVMT-Re Total Recall |
| BVMT-Re Delayed Recall | |
| BVMT-Re Recognition | |
| Visuospatial and quantitative functions | Greek Cross, Cube, Clock, Loops |
| Validity | Test of Memory Malingering |
| Emotional, behavioral, and adaptive functioning | Beck Anxiety Inventory |
| Beck Depression Inventory | |
| Death and Dying Distress Scale | |
| Functional Assessment of Cancer Therapy—Brain | |
| Fear of Cancer Recurrence | |
| Generalized Anxiety Disorder—7-Item | |
| Patient Health Questionnaire—9-Item | |
| Posttraumatic Growth Inventory |
| Domain | Test(s) |
|---|---|
| Intellectual ability | WAIS-IVa; ranged from subtests to full scale |
| RBANSb; ranged from subtests to full scale | |
| Estimated premorbid intelligence | Test of Premorbid Functioning |
| Language | Boston Naming Test |
| Auditory Naming Test | |
| Attention/working memory | WAIS-IVa; Working Memory |
| Processing speed | Oral Trail Making Test, Part A |
| Symbol Digit Modalities Test—Oral | |
| DKEFSc Color Naming | |
| DKEFSc Color Reading | |
| Executive function | Oral Trail Making Test, Part B |
| Controlled Oral Word Association Test | |
| DKEFSc Color/Word Inhibition | |
| DKEFSc Color/Word Inhibition/Switching | |
| Verbal memory | HVLT-Rd Total Recall |
| HVLT-Rd Delayed Recall | |
| HVLT-Rd Retention | |
| Visual memory | BVMT-Re Total Recall |
| BVMT-Re Delayed Recall | |
| BVMT-Re Recognition | |
| Visuospatial and quantitative functions | Greek Cross, Cube, Clock, Loops |
| Validity | Test of Memory Malingering |
| Emotional, behavioral, and adaptive functioning | Beck Anxiety Inventory |
| Beck Depression Inventory | |
| Death and Dying Distress Scale | |
| Functional Assessment of Cancer Therapy—Brain | |
| Fear of Cancer Recurrence | |
| Generalized Anxiety Disorder—7-Item | |
| Patient Health Questionnaire—9-Item | |
| Posttraumatic Growth Inventory |
aWechsler Adult Intelligence Scale—4th edition.
bRepeatable Battery for the Assessment of Neuropsychological Status.
cDelis-Kaplan Executive Function System.
dHopkins Verbal Learning Test—Revised.
eBrief Visuospatial Memory Test—Revised.
Complete List of Neuropsychological Tests That Attending Neuropsychologist(s) Were Able to Access for DTH-TNP Based on Inter-Organizational Practice Committee19 Guidance
| Domain | Test(s) |
|---|---|
| Intellectual ability | WAIS-IVa; ranged from subtests to full scale |
| RBANSb; ranged from subtests to full scale | |
| Estimated premorbid intelligence | Test of Premorbid Functioning |
| Language | Boston Naming Test |
| Auditory Naming Test | |
| Attention/working memory | WAIS-IVa; Working Memory |
| Processing speed | Oral Trail Making Test, Part A |
| Symbol Digit Modalities Test—Oral | |
| DKEFSc Color Naming | |
| DKEFSc Color Reading | |
| Executive function | Oral Trail Making Test, Part B |
| Controlled Oral Word Association Test | |
| DKEFSc Color/Word Inhibition | |
| DKEFSc Color/Word Inhibition/Switching | |
| Verbal memory | HVLT-Rd Total Recall |
| HVLT-Rd Delayed Recall | |
| HVLT-Rd Retention | |
| Visual memory | BVMT-Re Total Recall |
| BVMT-Re Delayed Recall | |
| BVMT-Re Recognition | |
| Visuospatial and quantitative functions | Greek Cross, Cube, Clock, Loops |
| Validity | Test of Memory Malingering |
| Emotional, behavioral, and adaptive functioning | Beck Anxiety Inventory |
| Beck Depression Inventory | |
| Death and Dying Distress Scale | |
| Functional Assessment of Cancer Therapy—Brain | |
| Fear of Cancer Recurrence | |
| Generalized Anxiety Disorder—7-Item | |
| Patient Health Questionnaire—9-Item | |
| Posttraumatic Growth Inventory |
| Domain | Test(s) |
|---|---|
| Intellectual ability | WAIS-IVa; ranged from subtests to full scale |
| RBANSb; ranged from subtests to full scale | |
| Estimated premorbid intelligence | Test of Premorbid Functioning |
| Language | Boston Naming Test |
| Auditory Naming Test | |
| Attention/working memory | WAIS-IVa; Working Memory |
| Processing speed | Oral Trail Making Test, Part A |
| Symbol Digit Modalities Test—Oral | |
| DKEFSc Color Naming | |
| DKEFSc Color Reading | |
| Executive function | Oral Trail Making Test, Part B |
| Controlled Oral Word Association Test | |
| DKEFSc Color/Word Inhibition | |
| DKEFSc Color/Word Inhibition/Switching | |
| Verbal memory | HVLT-Rd Total Recall |
| HVLT-Rd Delayed Recall | |
| HVLT-Rd Retention | |
| Visual memory | BVMT-Re Total Recall |
| BVMT-Re Delayed Recall | |
| BVMT-Re Recognition | |
| Visuospatial and quantitative functions | Greek Cross, Cube, Clock, Loops |
| Validity | Test of Memory Malingering |
| Emotional, behavioral, and adaptive functioning | Beck Anxiety Inventory |
| Beck Depression Inventory | |
| Death and Dying Distress Scale | |
| Functional Assessment of Cancer Therapy—Brain | |
| Fear of Cancer Recurrence | |
| Generalized Anxiety Disorder—7-Item | |
| Patient Health Questionnaire—9-Item | |
| Posttraumatic Growth Inventory |
aWechsler Adult Intelligence Scale—4th edition.
bRepeatable Battery for the Assessment of Neuropsychological Status.
cDelis-Kaplan Executive Function System.
dHopkins Verbal Learning Test—Revised.
eBrief Visuospatial Memory Test—Revised.
Statistical Analysis
Survey responses were categorized on a five-point Likert scale with “strongly agree” and “agree” combined to represent overall satisfaction. The statements “strongly disagree” and “disagree” were combined to endorse a negative experience, and “neutral” responses were demarcated separately. Data analyses were conducted in SPSS version 2723 to examine differences in practice between the two clinics. Frequencies were used to examine overall patient satisfaction and provider feasibility. Chi-square tests determined if patient (eg, travel time, concern for rapport) or provider responses (eg, met goals of the evaluation, audio/video difficulties) were significantly different across the two clinics.
Results
A total of 119 patients were scheduled during the study period (MGH = 83; VCU = 36). Of those, 79 completed some or all of their neuropsychological testing via DTH-TNP (MGH = 60; VCU = 19) and are the subjects of this study. Patients who completed testing in-person during the study period (MGH, n = 23; VCU, n = 17) were compared to those who completed DTH-TNP. Of the 23 patients who were seen for in-person testing at MGH, 9 were deferred due to clinician opinion that DTH-TNP was not appropriate for them (4 to optimize comparison with prior in-person assessments, 2 because the patient lacked equipment, 1 because the patient lacked time for testing, 1 because the patient was too impaired to complete formal testing, and 1 because the clinician was not confident the referral question could be answered using tests that could be administered via DTH-TNP). The other 14 patients seen in-person were due to patient choice. At VCU, all patients were given the choice of DTH-TNP or in-person assessments. Five patients who initially stated a preference for in-person testing later opted for DTH-TNP due to continued COVID concerns. Chi-square tests of independence showed no significant differences between patients seen by DTH-TNP vs those seen in-person in terms of the reason for referral (χ 2 = 5.21, P = .27) or diagnosis (χ 2 = 7.24, P = .30). Patients who had been seen for previous neuropsychology assessments were marginally more likely to be seen in person than by DTH-TNP (χ 2 = 3.55, P = .056; see Table 4). Patients who participated in DTH-TNP ranged from young adults to geriatric, with an equal distribution of sex and a wide range of education level (see Table 3). Karnofsky Performance Status, assigned at the neuro-oncology visit most proximal to neuropsychological assessment, ranged from unaffected to moderate levels of functional impairment.
Characteristics of the Patients Who Were Referred for a Neuropsychological Evaluation and Characteristics of Those Who Participated in DTH-TNP
| Virtual, n (%) | In-person, n (%) | |
|---|---|---|
| Patient characteristics by visit type (N = 119) | ||
| Evaluations | 79 (66) | 40 (34) |
| Diagnosisa | ||
| High-grade primary brain tumor | 18 (23) | 6 (15) |
| Low-grade primary brain tumor | 18 (23) | 14 (35) |
| Brain metastases | 8 (10) | 2 (5) |
| Non-CNS Ca/NTX | 22 (28) | 12 (30) |
| CNS lymphoma | 6 (8) | 3 (8) |
| Brain mass of unknown etiology | 5 (6) | 0 |
| Other | 2 (3) | 3 (8) |
| Reason for referrala | ||
| New mass/lesion/preoperative evaluation | 7 (9) | 0 |
| Baseline prior to Tx (chemo, radiation) | 12 (15) | 4 (10) |
| Cognitive concerns | 60 (76) | 36 (90) |
| Prior neuropsychological evaluation?b | ||
| No | 59 (75) | 23 (58) |
| Yes | 20 (25) | 17 (43) |
| Characteristics of DTH-TNP patients (n = 79) | ||
| Age: median (range) | 59 (21-81) | |
| Sex: n (%) | ||
| Male | 38 (48) | |
| Female | 41 (52) | |
| Highest education achieved: median (range) | 16 (10-20) | |
| High school (0-12) | 14 (18) | |
| Associate’s degree (13-15 y) | 16 (20) | |
| Bachelor’s degree | 26 (33) | |
| Graduate school (17-20+ y) | 23 (29) | |
| KPS at time of evaluation: n (%) | ||
| 50 | 4 (5) | |
| 60 | 16 (20) | |
| 70 | 9 (11) | |
| 80 | 25 (32) | |
| 90 | 22 (28) | |
| 100 | 3 (4) | |
| Treatment typec: n (%) | ||
| Neurosurgery | 47 (59) | |
| Brain radiation | 35 (44) | |
| Chemotherapy | 56 (71) | |
| No treatment | 4 (5) | |
| Brain tumor patients (n = 58) | ||
| Hemisphere: n (%) | ||
| Left | 19 (33) | |
| Right | 19 (33) | |
| Bilateral | 20 (34) | |
| Tumor region: n (%) | ||
| Frontal | 16 (28) | |
| Temporal | 12 (21) | |
| Parietal | 2 (3) | |
| Subcortical | 5 (9) | |
| Multifocal | 20 (34) | |
| Cerebellar | 3 (5) |
| Virtual, n (%) | In-person, n (%) | |
|---|---|---|
| Patient characteristics by visit type (N = 119) | ||
| Evaluations | 79 (66) | 40 (34) |
| Diagnosisa | ||
| High-grade primary brain tumor | 18 (23) | 6 (15) |
| Low-grade primary brain tumor | 18 (23) | 14 (35) |
| Brain metastases | 8 (10) | 2 (5) |
| Non-CNS Ca/NTX | 22 (28) | 12 (30) |
| CNS lymphoma | 6 (8) | 3 (8) |
| Brain mass of unknown etiology | 5 (6) | 0 |
| Other | 2 (3) | 3 (8) |
| Reason for referrala | ||
| New mass/lesion/preoperative evaluation | 7 (9) | 0 |
| Baseline prior to Tx (chemo, radiation) | 12 (15) | 4 (10) |
| Cognitive concerns | 60 (76) | 36 (90) |
| Prior neuropsychological evaluation?b | ||
| No | 59 (75) | 23 (58) |
| Yes | 20 (25) | 17 (43) |
| Characteristics of DTH-TNP patients (n = 79) | ||
| Age: median (range) | 59 (21-81) | |
| Sex: n (%) | ||
| Male | 38 (48) | |
| Female | 41 (52) | |
| Highest education achieved: median (range) | 16 (10-20) | |
| High school (0-12) | 14 (18) | |
| Associate’s degree (13-15 y) | 16 (20) | |
| Bachelor’s degree | 26 (33) | |
| Graduate school (17-20+ y) | 23 (29) | |
| KPS at time of evaluation: n (%) | ||
| 50 | 4 (5) | |
| 60 | 16 (20) | |
| 70 | 9 (11) | |
| 80 | 25 (32) | |
| 90 | 22 (28) | |
| 100 | 3 (4) | |
| Treatment typec: n (%) | ||
| Neurosurgery | 47 (59) | |
| Brain radiation | 35 (44) | |
| Chemotherapy | 56 (71) | |
| No treatment | 4 (5) | |
| Brain tumor patients (n = 58) | ||
| Hemisphere: n (%) | ||
| Left | 19 (33) | |
| Right | 19 (33) | |
| Bilateral | 20 (34) | |
| Tumor region: n (%) | ||
| Frontal | 16 (28) | |
| Temporal | 12 (21) | |
| Parietal | 2 (3) | |
| Subcortical | 5 (9) | |
| Multifocal | 20 (34) | |
| Cerebellar | 3 (5) |
Abbreviations: CNS, central nervous system; DTH-TNP, direct-to-home-teleneuropsychology; KPS, Karnofsky Performance Status.
aNot significantly different.
bMarginally significant difference (P = .056).
cPatients may have had more than one treatment type.
Characteristics of the Patients Who Were Referred for a Neuropsychological Evaluation and Characteristics of Those Who Participated in DTH-TNP
| Virtual, n (%) | In-person, n (%) | |
|---|---|---|
| Patient characteristics by visit type (N = 119) | ||
| Evaluations | 79 (66) | 40 (34) |
| Diagnosisa | ||
| High-grade primary brain tumor | 18 (23) | 6 (15) |
| Low-grade primary brain tumor | 18 (23) | 14 (35) |
| Brain metastases | 8 (10) | 2 (5) |
| Non-CNS Ca/NTX | 22 (28) | 12 (30) |
| CNS lymphoma | 6 (8) | 3 (8) |
| Brain mass of unknown etiology | 5 (6) | 0 |
| Other | 2 (3) | 3 (8) |
| Reason for referrala | ||
| New mass/lesion/preoperative evaluation | 7 (9) | 0 |
| Baseline prior to Tx (chemo, radiation) | 12 (15) | 4 (10) |
| Cognitive concerns | 60 (76) | 36 (90) |
| Prior neuropsychological evaluation?b | ||
| No | 59 (75) | 23 (58) |
| Yes | 20 (25) | 17 (43) |
| Characteristics of DTH-TNP patients (n = 79) | ||
| Age: median (range) | 59 (21-81) | |
| Sex: n (%) | ||
| Male | 38 (48) | |
| Female | 41 (52) | |
| Highest education achieved: median (range) | 16 (10-20) | |
| High school (0-12) | 14 (18) | |
| Associate’s degree (13-15 y) | 16 (20) | |
| Bachelor’s degree | 26 (33) | |
| Graduate school (17-20+ y) | 23 (29) | |
| KPS at time of evaluation: n (%) | ||
| 50 | 4 (5) | |
| 60 | 16 (20) | |
| 70 | 9 (11) | |
| 80 | 25 (32) | |
| 90 | 22 (28) | |
| 100 | 3 (4) | |
| Treatment typec: n (%) | ||
| Neurosurgery | 47 (59) | |
| Brain radiation | 35 (44) | |
| Chemotherapy | 56 (71) | |
| No treatment | 4 (5) | |
| Brain tumor patients (n = 58) | ||
| Hemisphere: n (%) | ||
| Left | 19 (33) | |
| Right | 19 (33) | |
| Bilateral | 20 (34) | |
| Tumor region: n (%) | ||
| Frontal | 16 (28) | |
| Temporal | 12 (21) | |
| Parietal | 2 (3) | |
| Subcortical | 5 (9) | |
| Multifocal | 20 (34) | |
| Cerebellar | 3 (5) |
| Virtual, n (%) | In-person, n (%) | |
|---|---|---|
| Patient characteristics by visit type (N = 119) | ||
| Evaluations | 79 (66) | 40 (34) |
| Diagnosisa | ||
| High-grade primary brain tumor | 18 (23) | 6 (15) |
| Low-grade primary brain tumor | 18 (23) | 14 (35) |
| Brain metastases | 8 (10) | 2 (5) |
| Non-CNS Ca/NTX | 22 (28) | 12 (30) |
| CNS lymphoma | 6 (8) | 3 (8) |
| Brain mass of unknown etiology | 5 (6) | 0 |
| Other | 2 (3) | 3 (8) |
| Reason for referrala | ||
| New mass/lesion/preoperative evaluation | 7 (9) | 0 |
| Baseline prior to Tx (chemo, radiation) | 12 (15) | 4 (10) |
| Cognitive concerns | 60 (76) | 36 (90) |
| Prior neuropsychological evaluation?b | ||
| No | 59 (75) | 23 (58) |
| Yes | 20 (25) | 17 (43) |
| Characteristics of DTH-TNP patients (n = 79) | ||
| Age: median (range) | 59 (21-81) | |
| Sex: n (%) | ||
| Male | 38 (48) | |
| Female | 41 (52) | |
| Highest education achieved: median (range) | 16 (10-20) | |
| High school (0-12) | 14 (18) | |
| Associate’s degree (13-15 y) | 16 (20) | |
| Bachelor’s degree | 26 (33) | |
| Graduate school (17-20+ y) | 23 (29) | |
| KPS at time of evaluation: n (%) | ||
| 50 | 4 (5) | |
| 60 | 16 (20) | |
| 70 | 9 (11) | |
| 80 | 25 (32) | |
| 90 | 22 (28) | |
| 100 | 3 (4) | |
| Treatment typec: n (%) | ||
| Neurosurgery | 47 (59) | |
| Brain radiation | 35 (44) | |
| Chemotherapy | 56 (71) | |
| No treatment | 4 (5) | |
| Brain tumor patients (n = 58) | ||
| Hemisphere: n (%) | ||
| Left | 19 (33) | |
| Right | 19 (33) | |
| Bilateral | 20 (34) | |
| Tumor region: n (%) | ||
| Frontal | 16 (28) | |
| Temporal | 12 (21) | |
| Parietal | 2 (3) | |
| Subcortical | 5 (9) | |
| Multifocal | 20 (34) | |
| Cerebellar | 3 (5) |
Abbreviations: CNS, central nervous system; DTH-TNP, direct-to-home-teleneuropsychology; KPS, Karnofsky Performance Status.
aNot significantly different.
bMarginally significant difference (P = .056).
cPatients may have had more than one treatment type.
Provider Feedback
Sixty-eight surveys were completed by providers (MGH = 49; VCU = 19) regarding DTH-TNP evaluations that took place between April and September 2020 for a response rate of 86% (Figure 1). Provider respondents included two neuropsychologists (M.W.P. and A.R.L.) and one neuropsychology fellow (S.E.B.) under the supervision of a neuropsychologist (A.R.L.).
DTH-TNP provider ratings (N = 68). Agreement with specific items from the provider survey expressed as percentages of the total number of respondents (n = 68). Abbreviation: DTH-TNP, direct-to-home-teleneuropsychology.
Providers reported the median duration of DTH-TNP evaluations to be 124 minutes (MGH = 134 minutes, range 60-210; VCU = 98 minutes, range = 45-140). In 25% of cases, the assessment was divided into more than one session (MGH = 33%, VCU = 5%; χ 2 = 5.478, P = .019); the remainder were completed in a single session. Testing assistance (eg, a psychometrist or graduate student) was involved in 58% of the assessment cases (MGH = 57%, VCU = 63%).
There were no differences between institutional provider responses when discussing DTH-TNP challenges (P > .05). Providers reported they were able to fully achieve the intended goal of the assessment in 88% of clinical encounters and partially achieved their goals in an additional 10% of evaluations. In only one case, an evaluation at VCU, did the providers feel they were completely unable to achieve the goal of the evaluation. Despite this high rate of success, providers endorsed challenges during DTH-TNP visits, including patient dysregulation (16%), slow or unreliable internet (15%), audio or visual problems with technology (9%), interruptions during testing (10%), and/or difficulty understanding instructions (1%). When asked about strategies used to optimize the DTH-TNP or overcome specific challenges, providers indicated that no strategies were necessary in most evaluations (54%). When strategies were used, having another person or family member assist with technical matters to initiate appointment was the most frequently endorsed (21%), followed by taking frequent breaks (9%).
Patient Feedback
Fifty-two patient feedback surveys were received between April and September 2020 (MGH = 36, VCU = 16) with a response rate of 74%. Respondents were primarily the patient themselves (MGH = 86%, VCU = 100%), though some surveys were completed by a caregiver or family member (MGH = 14%). Appointments included multiple activities such as interview, testing, or feedback, with 92% of respondents indicating that their appointment included some formal testing (MGH = 89%, VCU = 100%). Subjects ranged in age from 18 to 70+ years old.
There were no differences between institutional responses in patient satisfaction ratings (P > .05; Figure 2). Most respondents (98%) indicated satisfaction with the DTH-TNP assessment and would recommend the virtual visit to others (92%). All respondents felt understood by the examiner (100%) and the majority denied technical difficulties (90%), communication challenges (94%), or privacy concerns (98%).
Patient satisfaction ratings (N = 52). Agreement with specific items from the patient survey expressed as percentages of the total number of respondents (n = 52). For the purposes of illustration, responses of “strongly agree” and “agree” were combined into the bar indicating agreement, and responses of “strongly disagree” and “disagree” were combined into the bar indicating disagreement.
There were also no differences between the two institutions on patient evaluation preference ratings (P > .05; Figure 3). Respondents were asked to select all the ways an in-person assessment could enhance the evaluation: 40% said it would promote improved personal connection with the examiner, 23% said it would allow for a more extensive assessment, 17% said it would improve communication with examiner, and 8% said it would make it easier to express concerns. Patients endorsed the following perceived benefits of DTH-TNP: saved travel time (88%), reduced risk of infection (79%), a reduction in anxiety (27%), and improved concentration without an examiner in the room (23%). Furthermore, 37% of respondents reported preferring the DTH-TNP to an in-person evaluation. Twenty-nine percent of patients reported a willingness to drive 1-3 hours for an in-person evaluation, while 31% reported they would drive up to 1 hour before selecting a virtual visit.
Patient preferences by evaluation type (N = 52). Agreement with specific items from the patient survey expressed as percentages of the total number of respondents (n = 52). For the purposes of illustration, responses of “strongly agree” and “agree” were combined into the bar indicating agreement, and responses of “strongly disagree” and “disagree” were combined into the bar indicating disagreement.
Discussion
This study assessed the acceptability and feasibility of DTH-TNP at two regionally diverse neuro-oncology clinics. Patients found DTH-TNP to be highly acceptable and providers found DTH-TNP to be a feasible method for conducting neuropsychological evaluations. Nearly all patients reported being satisfied with the experience and would recommend the virtual appointment to others, regardless of the acknowledged limitations (eg, technical issues, effects on communication/connection with examiner). Similarly, providers found DTH-TNP feasible in the majority of cases for accomplishing the objectives of their neuropsychological evaluations, despite the limitations present in a virtual testing environment (eg, test constraints, technological disruptions). These findings suggest that neuro-oncology patients and their neuropsychologists find the DTH-TNP model feasible and acceptable. To our knowledge, this is the first demonstration of acceptability and feasibility of DTH-TNP in a neuro-oncology clinical sample.
Providers were able to evaluate patients with a broad range of referral questions, diagnoses, and levels of functioning. The only difference in distribution of in-person vs DTH-TNP assessments was a preference to evaluate patients in-person if they had previously been evaluated to reduce the potential confounding factor of modality of assessment when comparing test performance. Illustrating this concern, a free-text comment from one clinician said, “The patient was being seen for a re-evaluation. In-person testing session is being added to make sure data are comparable.” Challenges reported by providers, such as issues with internet connection and patient behavior, occasionally required flexibility. For instance, one survey included the following free-text response from the provider: “There was one instance in which the patient’s verbal response was inaudible (digit span).” This required an adaptation by the clinician; “administered an extra item at the same level of difficulty.” An assessment rated as partially feasible was also affected by the patient’s cognitive impairment: “Had to add on an in-person testing session because patient’s executive function difficulties made it hard for him to manage the virtual interface.” Despite these challenges, the majority of clinician responses suggested that neuropsychological assessment questions could be answered using DTH-TNP, such as “This was a complex case involving unidentified brain mass and possible neurodegenerative disorder. We were able to gather cognitive data that pointed more toward one than the other etiology.” Providers were also able to have input into important outcomes for patients based on DTH-TNP evaluations: “Was able to identify potential cognitive issues re: return to work planning, provide feedback, and arrange follow-up that has resulted in patient returning to work successfully.”
Patient experience with DTH-TNP was largely favorable. Perceived advantages included saved travel time (eg, “The virtual visit was much easier than driving to VCU downtown and dealing with parking”) and reduced infection risk of remote assessment (eg, “My husband was able to join the appointment from quarantine”). Patients also noted the limitations of DTH-TNP, such as loss of some of the personal element of neuropsychological assessment (eg, “In-person promotes a better connection”) and potential vulnerability of test results to distractions in the home environment (eg, “The location where you are taking the test is an unknown variable and may cause results to be less reliable”). The majority of patient respondents denied technical difficulties, but 10% experienced problems such as “There was a small glitch in getting connected…” and “Video and audio skipping during the discussion.”
The inclusion of two institutions in diverse geographical parts of the country allowed for an additional regional exploration of DTH-TNP. While both sites are in urban settings, the two cities have distinct population densities and ethno-racial distributions. Boston is geographically smaller than Richmond (48.3 vs 59.9 square miles),24 though has a population three times that of Richmond (692,000 vs 230,436). Further, while the median age in Boston (32 years) and Richmond (33.5 years) are comparable, there are ethno-racial differences between the cities (Boston: Asian 9.6%; Black 25.3%; Hispanic 19.7%; White 44.5%; Richmond: Asian 2.1%; Black 47.8%; Hispanic 6.7%; White 40.4%) that are important considerations regarding access to healthcare. Further, outside of the city, MGH has a predominantly suburban catchment area. Conversely, VCU is bordered by both suburban and rural communities just outside the metro area. While respondents were not asked to report specific demographic variables to maximize anonymity and promote unbiased feedback, responses across the clinics were largely consistent suggesting that DTH-TNP is suitable in a wide range of settings.
There are several features of practice in this DTH-TNP study that differed from our typical in-person assessments. Limitations in the assessment battery acknowledged by providers included an inability to assess motor functioning (eg, Grooved Pegboard25) and loss of access to executive functioning tests with manipulatives (eg, Delis-Kaplan Executive Function Tower test,26 Wisconsin Card Sorting test27, though a recent computerized administration of this test has been made available for DTH-TNP recently) which are important measures of problem-solving and reasoning. Furthermore, the common practice of “testing the limits” of performance in neuropsychological assessments, which is dependent on expert clinician judgment and observation, may be more substantially limited by the TNP modality. As such, the conclusions one can draw about the patient’s abilities in these domains are limited when using a DTH-TNP model. Other logistical challenges are also a factor to consider in DTH-TNP, such as requiring multiple visits to complete evaluations (25%). As clinicians gained experience with DTH-TNP, we increasingly were able to complete evaluations in a single session. Similarly, technician assistance in this survey is lower (59%) than is typical in our practices, which also gradually increased with experience providing DTH-TNP.
This study is limited by the voluntary survey methodology. Not all patients chose to complete surveys, thus it is possible that individuals who did not participate had a higher rate of dissatisfaction. Further, in-depth consideration of differences in patient satisfaction and feasibility is difficult to evaluate in this study because we collected very minimal patient identifying information. We sought to reduce feelings of coercion or anxiety that might have influenced feedback by essentially making the survey anonymous, but the negative consequences of that limitation are that we do not have individual data (eg, cancer type, age, degree of cognitive impairment, etc.) that might be important in understanding factors that influence the experience of patients. In addition, there were some differences in the procedures and offerings of DTH-TNP between the two clinics, as well as differences in patient response rates (MGH = 63%, VCU = 84%) that may have impacted findings. Provider surveys were completed by a small group of clinicians who performed these TNP evaluations. While this in-depth knowledge of the clinical encounters likely provides the most detailed information about feasibility, the opinions inherently integrate the potential biases of the clinicians/authors. Future research should assess feasibility and acceptability in a larger cohort of providers to increase generalizability. Despite those limitations, the consensus between the two institutions speaks to degree of generalizability of DTH-TNP across clinics and clinic procedures.
Finally, it should be noted that the circumstances surrounding these evaluations were highly atypical. The novel COVID-19 virus likely caused elevated distress for immunosuppressant patients, increasing the odds of participating from the safety of their home, and potentially inflating overall patient satisfaction. However, managing risk for infection has been and will continue to be an ongoing concern in this immunocompromised population. DTH-TNP may be more suitable for continued care in this population given the risk associated with seasonal influenza and the future possibility of other novel viral pathogens.28 Moreover, the majority of patient and provider respondents were at least willing and able to engage in DTH-TNP, which may have led to a self-selection bias in the present sample. It is possible that individuals with less reliable internet access, sophisticated video conferencing equipment, or who are hesitant using technology would be less likely to opt for DTH-TNP or view it favorably. Therefore, the current findings are not generalizable to individuals without access or the technical know-how.
Future Directions
DTH-TNP raises questions related to standardization of test procedures using this novel method of assessment and the substantial work needed to test reliability and validity of this method of assessment. Modifications may need to be made to testing procedures in order to translate in-person instruments to a virtual platform, enhance behavior management, and ensure test security. Future research that evaluates the comparability of in-person neurocognitive testing and DTH-TNP is needed—without establishing the validity and reliability of DTH-TNP, it is difficult to assess the viability of integration of DTH-TNP into regular clinical practice. Despite these limitations, the current findings make a strong case for future investigation of DTH-TNP in clinical care. An additional issue that will need to be explored is the use of DTH-TNP approaches to administer neuropsychological tests as part of clinical trials. Even small differences in performance caused by different administration methods could have a large impact on clinical trials in which these tests are used as outcome metrics. At the same time, clinical trials that integrate neurocognitive outcomes have been ongoing during the pandemic, and the question of whether to administer tests virtually, stick to in-person administration (which can also be affected by COVID-19 precautions such as wearing a mask), or abandon cognitive test data entirely is a pressing concern. Future studies will need to carefully evaluate the reliability of DTH-TNP for research purposes.
It is important to note that special permissions have been granted given the current crisis, and this has in turn been communicated to the patient and treatment team. Specifically, publishing companies (ie, Pearson) permitted neuropsychologists to utilize their products while working remotely with use of a disclosure statement19,29 and best practices include a clear statement in neuropsychological reports distinguishing that normative data are based on in-person assessments and results are to be interpreted with caution.19 An important next step for neuropsychologists working in neuro-oncology will be identifying tests that are both suitable to the DTH-TNP environment and robust enough to assess the cognitive functioning of neuro-oncology patients.
While more work is needed to assess comparability to in-person testing, DTH-TNP has the potential to enhance continuity of care in neuro-oncology patients. Neuropsychologists providing care in neuro-oncology clinics continue to advocate for timely and preemptive baseline testing in conjunction with routine follow-up care. However, patients often live at great distances from hospitals or may not be able to easily access the resources provided by neuropsychologists. Beyond the current public health crisis, DTH-TNP may allow these neuro-oncology patients in such rural or remote locations access to neuropsychologists with a specialized focus in neuro-oncology that understand the unique needs of this patient population. By enhancing the ability to reach people more frequently and advancing long-term monitoring abilities, we have an opportunity to provide better care for the cognitive needs of our patients. Additional benefits of DTH-TNP demonstrated during the COVID-19 pandemic, such as the removal of transportation barriers, decreased need for space, and willingness to reimburse for these services may result in continued incorporation of TNP services moving forward.
Conclusions
These preliminary data demonstrate the acceptability of DTH-TNP for neuro-oncology patients and the feasibility of this modality for neuropsychological providers. Continued research to improve upon DTH-TNP for neuro-oncology patients is warranted, including improvements of virtually delivered neurocognitive testing that can identify impairments specific to brain tumors.
Funding
This work was not funded by any external sources of financial support.
Acknowledgments
We would like to thank the Massachusetts General Hospital Psychology Assessment Center faculty, staff, and trainees, led by Janet Sherman, PhD for their support in the development and implementation of these surveys.
Conflict of interest statement: The authors declare that they have no conflicts of interest regarding the current work.
