Validation of three health literacy screening questions compared with S-TOFHLA in a low-income diverse English- and Spanish-Speaking population

Abstract

Background

Clinicians need a tool to gauge patients’ ability to understand health conditions and treatment options. The Short-form Test of Functional Health Literacy in Adults (S-TOFHLA) is the gold standard for this, but its length is prohibitive for use in clinical settings. This study seeks to validate a novel three-item question set for predicting health literacy.

Methods

This cross-sectional study utilized an in-person questionnaire alongside the S-TOFHLA. The sample included 2027 English- and Spanish-speaking adults (≥18 years) recruited from primary care practices serving a low-income eastern Pennsylvania community. Most patients (57.7%) identified as Hispanic. Diagnostic accuracy of each question and aggregated scores were assessed against the validated survey by calculating the area under the receiver operating characteristic (AUROC) curve.

Results

Questions in the ‘Problems Learning’ and ‘Help Reading’ domains (AUROC 0.66 for each) performed better than the ‘Confident Forms’ question (AUROC 0.64). Summing all three scores resulted in an even higher AUROC curve (0.71). Cronbach’s alpha of the combined items was 0.696.

Conclusions

Study results suggest that any of the three questions are viable options for screening health literacy levels of diverse patients in primary care clinical settings. However, they perform better as a summed score than when used individually.

Introduction

Recognizing that the key to quality care and improved health outcomes lies in effective patient-centered communication, The Joint Commission, the American Medical Association (AMA) and the Institute of Medicine have been seeking ways to help health care organizations and providers improve patient health literacy.^1–3 Patient safety is key among the population health issues that are likely to be improved with better communication between patients and health providers. Numerous studies have demonstrated an association between low health literacy and chronic disease, vaccination rates, child health and parent behaviors.^4–9

The AMA also notes that patient-centered communication is ‘ethically important,’ pointing to significant health disparities that exist. The AMA’s Ethical Force Program™ recommends that organizations collect data on the demographics and communication needs of the populations they serve, including health literacy.² These data then could be used to develop strategies to better communicate health information. Accreditation standards of The Joint Commission state that patients have a fundamental right to receive health information in a way they can understand, further emphasizing the importance of health literacy.¹⁰

Health literacy measures more than one’s ability to read and write in a specified language; other educational, social and cultural factors also affect an individual’s health literacy level.^3,11–14 Researchers and health systems have begun to examine ways to screen patients for these social determinants^11,12,14,15 to identify those in need of assistance. A number of studies have examined tools to measure health literacy within specific populations^16–33 including male veterans,^{19,34,35,36–40} majority English-speaking Whites,⁴¹ non-Hispanic Blacks⁴² and Hispanic/Latino individuals.^30,43

The Short-form Test of Functional Health Literacy in Adults (S-TOFHLA)¹⁸ assesses the ability to read health-related passages using a modified Cloze procedure (reader determines which words^30,43 belong where spaces are intentionally left blank) along with numeracy skill exercises (respondent must calculate doses and timing of medications). The S-TOFHLA has been used in numerous studies examining the association between health literacy and health outcomes. It also has served as the reference instrument for examining other tools such as the REALM^44–47 and the Newest Vital Sign,⁴⁶ among others.^{16,17,19–33,47} However, the S-TOFHLA includes more than 40 items and takes at least 14 minutes to administer, which is prohibitive in busy clinical settings. More recent assessment tools are composed of one to three items that include the domains ‘Confident Forms’ relating to reading medical documents, needing ‘Help Reading’ and ‘Problems Learning’ through written information. Previous studies have found that when compared with the TOFHLA and REALM, these three items accurately identified patients with inadequate health literacy,¹⁰ and they have been suggested for use in clinical settings. Slight variations in question wording and response categories were noted between instruments (Supplemental Table S5) The ‘Confident Forms’ and ‘Help Reading’ questions differ in the level of specificity of form type and source of help sought in the item stem. We chose to use the more specific ‘medical forms’ item in our study to ensure participants were considering this type of form in their response and the item providing sources of help for the ‘Help Reading’ item to ensure participants were considering different sources of help they may use when answering questions. The item we chose for the ‘Problems Learning’ domain asks specifically about understanding the written information, which we felt is more relevant to health literacy than, for example, asking about one’s difficulty with reading in general. For each item, we chose the response category more commonly used in research.

For this study—which was part of a larger cross-sectional study examining the association between health literacy and social needs—researchers were seeking a question set that was feasible for use in the outpatient setting, where diagnostic time is limited. The instrument needed to not only accurately predict low health literacy within a diverse population but also be quick and easy to administer. Therefore, researchers assessed the psychometric properties of the three health literacy items used by Chew et al.²⁹ as compared with the S-TOFHLA.

Methods

Our study sample included 2027 patients age ≥ 18 years recruited from three primary care practices serving a low-income, ethnically diverse community in eastern Pennsylvania. One practice primarily serves patients with human immunodeficiency virus (HIV) or hepatitis. Patients from these three practices were recruited to participate because they represent populations at risk for low health literacy. Participants were asked to complete both the S-TOFHLA and a set of three closed-ended health literacy screening questions. Patients whose primary language was not either Spanish or English were excluded from the study.

Patients were asked to participate by a bilingual research assistant (RA) while waiting in the reception area or examination room of the practices. The RA explained the study to each patient and asked for informed consent. Once the consent form was signed, participants were given 7 minutes to complete each section of the S-TOFHLA (reading comprehension and numeracy) in accordance with the tool’s instructions. If the participant reported an inability to read, the S-TOFHLA was not administered, and the RA proceeded with the three-question screening tool. Each participant received a $10 gift card. The study received institutional review board approval.

Data collection tools

The S-TOFHLA, comprising eight numeracy questions and 36 reading comprehension items, was used in this study. Reading comprehension raw scores range from 0 to 36 and categorize health literacy as follows: ‘inadequate’ (0–16), ‘marginal’ (17–22) and ‘adequate’ (23–36). The numeracy component uses a series of prompts consisting of prescription vials, an appointment slip and a chart describing eligibility for financial aid.

The three-question health literacy screening tool was formatted as a written questionnaire, which was either read independently by the participant or read aloud by the RA, depending on the participant’s reading ability. The participant documented the mode of administration. The questions asked were (i) ‘How confident are you filling out medical forms by yourself?’ (‘Confident Forms’ domain); (ii) ‘How often do you have someone (like a family member, friend, hospital/clinic worker or caregiver) help you to read hospital or clinic materials?’ (‘Help Reading’ domain); and (iii) ‘How often do you have problems learning about your medical condition because of difficulty understanding written information?’ (‘Problems Learning’ domain). Participants selected from a list of responses ranging from ‘extremely’ to ‘not at all’ for the ‘Confident Forms’ item and ‘all the time’ to ‘none of the time’ for the other two items.

Data analysis

Descriptive statistics were used to characterize study participants’ demographics, preferred language and health literacy level. The mean and standard deviation were calculated for continuous variables, while frequencies and percentages were analyzed for categorical variables. The S-TOFHLA reading scores were calculated and recommended cut points were used to categorize participants as having ‘inadequate,’ ‘marginal’ or ‘adequate’ health literacy.

The responses to the ‘Help Reading’ and ‘Problems Learning’ questions were reverse-coded so that higher scores indicated lower health literacy and matched the coding of the ‘Confident Forms’ question. The total score was calculated by summing the scores of each individual question (0–4) for a total ranging from 0 to 12.

The diagnostic accuracy of the three health literacy questions, as well as the summed score, were assessed against the S-TOFHLA using an area under the receiver operating characteristic (AUROC) curve for scores indicating ‘inadequate’ health literacy alone and ‘inadequate’ combined with ‘marginal’ health literacy. An AUROC value of 1.0 indicates perfect diagnostic ability, and a value of 0.5 indicates no ability.

Sensitivity, specificity and likelihood ratios with 95% confidence intervals (CI) were calculated for all possible cut-off values for each of the health literacy items using the S-TOFHLA ‘inadequate’ health literacy reading scores as the gold standard. The Youden index was used to determine the cut point that optimized the differentiating ability of each of the three individual questions as well as a single combined score.⁴⁸ The Youden index is the optimal criterion value when disease prevalence is 50%; equal weight is given to sensitivity and specificity, and costs of various decisions regarding categorization are ignored.⁴⁸ Higher Youden index values are better than lower values. In concept, the Youden index is the vertical distance between the receiver operating characteristic (ROC) curve and a 45-degree line on the plot of the true-positive rate (sensitivity) versus the false-positive rate (1-specificity).⁴⁹ The ROC curves also were plotted.

Additionally, health literacy levels measured with the three individual questions were compared between participants who completed the S-TOFHLA and those who did not. This was done to see whether low health literacy correlated with non-completion, which might mean the S-TOFHLA was too difficult to understand for those with lower health literacy. Statistical analyses were performed using IBM SPSS Statistics for Windows, Version 24.0 (Armonk, NY: IBM Corp) and MedCalc for Windows, Version 17.4.4 (MedCalc Software, Ostend, Belgium).

Results

A total of 1750 individuals—55% recruited from a large internal medicine residency practice, 28% from an internal medicine practice serving predominantly Spanish-speaking patients and 17% from a practice serving patients with HIV/hepatitis—completed all three health literacy questions and the S-TOFHLA. Another 277 individuals completed only the three health literacy questions (Table 1).

Among those who completed both instruments, a majority (63.4%) were women, with an average age of 44.5 years. More than one-third did not report their race, either indicating they were ‘unsure’ or not providing an answer. A large proportion of participants identified as Hispanic (57.7%). Two other notable characteristics were the large percentages reporting they were unemployed (64.4%) or covered by public health insurance (i.e. Medicare, Medicaid) (71.9%).

Participants who did not complete the S-TOFHLA (n = 277) were more likely to be male, Hispanic, unemployed and Spanish-speaking. They also tended to be older (mean 57.5 years) than those who completed the S-TOFHLA. Demographic differences between the two groups were statistically significant.

Results of the S-TOFHLA are described in Table 2. Nearly three-quarters (73.8%) of participants completed the English language version of the survey. The majority (82.0%) had ‘adequate’ health literacy, while 8.4% were in the ‘inadequate’ range and nearly 10% had ‘marginal’ health literacy.

Only participants who completed all three health literacy questions and the S-TOFHLA were included in the AUROC analysis (n = 1750). Like the findings of Chew and colleagues,³⁴ combining the ‘inadequate’ and ‘marginal’ health literacy categories did not improve the AUROC curve for any of the individual questions or when questions were combined (Table 3). All further analyses compared the accuracy of the three screening questions for identifying inadequate health literacy with that of the S-TOFHLA. The AUROC curves for all three health literacy questions were significantly different than 0.5 (P < 0.0001). The differences between the ROC curves for the pairwise comparisons of the three health literacy questions were not significantly different (‘Confident Forms’ ~ ‘Help Reading’ P = 0.3965; ‘Confident Forms’ ~ ‘Problems Learning’ P = 0.5949; ‘Help Reading’ ~ ‘Problems Learning’ P = 0.7390).

Two questions—‘Problems Learning’ and ‘Help Reading’—similarly distinguished low health literacy and more accurately identified people with low health literacy than the ‘Confident Forms’ question (Table 3). Summing the scores from the three items resulted in a higher AUROC curve as compared with any of the single-item scores. Cronbach’s alpha for the internal consistency of the three items as a single combined score was 0.696, demonstrating a good reliability rate. Using the Youden index, the response cut point that maximized the correct classification rate for ‘inadequate’ health literacy for the ‘Confident Forms’ items was >2, which correlates to a response of ‘a little bit’ (Table 4). For both the ‘Help Reading’ and ‘Problems Learning’ items, the optimizing cut point was >1 (a response of ‘some of the time’).

Finally, we compared the scores from the three individual health literacy screening questions between those who completed the S-TOFHLA and those who did not. These results showed that those who did not complete the S-TOFHLA had lower health literacy levels than those who completed the S-TOFHLA (see Supplemental Fig. S1).

Discussion

Main finding of this study

The AUROC curve for all health literacy items in this study was lower than those reported by others, with a few exceptions (Table 3). The AUROC curve value of the ‘Help Reading’ item in our study was similar to that reported in all but two studies.^19,50 For the ‘Problems Learning’ domain, the AUROC curve value in our study was similar to that reported by others, with the exception of three studies.^19,30,50 Differences in AUROC curve values likely resulted from study population differences.

The higher AUROC curve resulting from using the summed score as compared with any of the single-item scores indicates that the combined score can better detect ‘inadequate’ health literacy. This differs from the findings of Chew et al.,³⁴ who did not find a significant increase in AUROC curve when combining all three items. In addition, our study found a higher cut point for the ‘Confident Forms’ item than that found by Chew et al.³⁵ (>2, correlating to the ‘a little bit’ response versus >1, the ‘somewhat’ response).

Cut points should make sense from a clinical perspective. In this case, optimizing sensitivity would result in identifying more patients with low health literacy. This would also result in lower specificity and, therefore, a greater number of false positives (i.e. patients being identified as having low health literacy when they do not). In terms of health literacy, inaccurate categorization would not be harmful to patients. In fact, receiving additional explanation about how to manage one’s health could be useful, especially for patients whose health literacy scores come in just above the cut point. Based on our findings, the recommendation is for a score > 0 (i.e. any response other than ‘extremely [confident]’ for the ‘Confident Forms’ item and ‘none of the time’ for the ‘Help Reading’ and ‘Problems Learning’ items) be used as the cut point for identifying patients who need greater assistance to understand their health and navigating the health system.

What is already known on this topic

Administration of a brief HL screening tool that demonstrates a high predictive value for identifying patients who may need greater assistance is an important approach to improving patient-centered care. Limited health literacy has been associated with poor health outcomes, higher health care costs and lower patient satisfaction.^{32,33,42,51,52} Although current measures focus only on the capacity of the individual and not the characteristics of the health care system that contribute to health literacy, these measures can be used to inform strategies to alleviate individual’s challenges in navigating the system and improving health. The gold-standard measure of health literacy, the S-TOFHLA, is too time- and labor-intensive to administer as a screening tool in the clinical setting. With the increased interest in health care on the social determinants of health and the pursuit of social needs screening^{11–15,53–57} inclusion of a health literacy screening tool is important for health systems and providers to consider.

What this study adds

To our knowledge, this is the largest study to examine the diagnostic accuracy of a health literacy screening tool, in whole or in part, within a primary care population that includes a large number of Hispanic participants. Although the study population was not randomly selected, its demographics reflect the profile of the patient populations of the clinics from which they were recruited. In addition to the large sample size, another study strength is use of the S-TOFHLA, which is considered the gold standard for measuring health literacy. Additionally, the prevalence of inadequate health literacy in our study is different from that reported in other studies, providing insight into how these individual health literacy questions perform in populations with varying degrees of inadequate health literacy prevalence.

Compared with other studies, our population is slightly younger^19,30,34,37 with a higher proportion of women^{19,30,  34–36}and a lower proportion of Blacks and Whites.^{19,30,34,35,  37,51} Similar to other study populations, most participants had adequate health literacy. Differences were noted in the proportion of participants with inadequate health literacy in this study as compared with others; a higher proportion of participants had inadequate health literacy in two studies,^30,36 and one study involving a Veterans Affairs (VA) population had a lower proportion of participants with inadequate health literacy.^19,34 Both studies with higher prevalence of low health literacy had greater representation of Blacks, which may explain the differences observed.^58–61 Differences in the proportion of study participants with low health literacy between our study and those of Chew et al. may be due to the higher proportion of older Whites in those studies.^19,34

The results of this study add to research seeking to identify a health literacy screening tool for use in a clinical environment. A single-item literacy screener is preferred because of the time needed to administer and the ability to incorporate it into other assessments used in health care, such as emerging social needs screeners. Studies suggest that some items perform better with different populations, and the results of this study’s AUROC analysis further illustrate this point. It would place an undue burden on health systems and physician practices if a different screening tool were needed for each population subgroup. Therefore, it is important that clinicians consider the diagnostic value of the results before implementing a health literacy screening tool.

Limitations of this study

A limitation to this research is that its study population was composed of patients from one urban health system in the eastern US and, therefore, does not reflect the demographic composition of the country, limiting generalizability of the findings. The Hispanic population in this region is primarily of Puerto Rican and Dominican Republic ethnicities, which may not correspond to other Hispanic populations, such as those with ethnic derivations from Mexico or Cuba. In addition, there were relatively few African-American participants in our study. We used a convenience sample, which is a non-probability sampling method. Further research that includes a more diverse population, both geographically and ethnically, would greatly contribute to identifying a health literacy screening tool that can confidently be used by health systems and physician practices throughout the country. Some factors that may influence, or be influenced by, health literacy were not examined in this analysis, including health care utilization, chronic disease diagnoses and risk perception. A forthcoming report on the expanded study examines some of these variables.

Acknowledgements

The authors would like to thank Susan E. Hansen for her editorial contributions and manuscript preparation as well as the Health Care Trust of Anne Constance and Carol Robert Anderson [grant number 9189191] for its generous support of this project.

Authors’ contributions

HK and CAC conceived of and designed the study. TJF consulted on clinical measures selection. CAC oversaw data integrity, and RH managed data collection. All authors participated in data interpretation following statistical analysis by HK and RH. HK, CAC and RH drafted the initial version of the manuscript, and all authors participated in the critical review and revisions. All authors approved this final version to be submitted for publication.

Funding

This work was supported by the Health Care Trust of Anne Constance and Carl Robert Anderson [grant number 9189191].

Ethics approval and consent to participate

The study was approved by the Lehigh Valley Health Network institutional review board.

Consent for publication

All authors reviewed and approved this version of the manuscript.

Data availability

Data will be shared upon reasonable request to the corresponding author.

Hope Kincaid, Senior Biostatistician

Cathy A. Coyne, Associate Professor of Practice in Public Health

Roya Hamadani, Senior Quality Improvement Specialist

Timothy Friel, Chair, Department of Medicine, and Physician in Infectious Diseases

References