Quality Assessment of Online Patient Information for Cosmetic Botulinum Toxin Free

Abstract

Background

In an era of widespread Internet access, patients increasingly look online for health information. Given the frequency with which cosmetic botulinum toxin injection is performed, there is a need to provide patients with high-quality information about this procedure.

Objectives

The aim of this study was to examine the quality of printed online education materials (POEMs) about cosmetic botulinum toxin.

Methods

An Internet search was performed to identify 32 websites of various authorship types. Materials were evaluated for accuracy and inclusion of key content points. Readability was measured by Flesch Reading Ease and Flesch-Kincaid Grade Level. Understandability and actionability were assessed with the Patient Education Materials Assessment Tool for Printed Materials. The effect of authorship was measured by undertaking analysis of variance between groups.

Results

The mean [standard deviation] accuracy score among all POEMs was 4.2 [0.7], which represents an accuracy of 76% to 99%. Mean comprehensiveness was 47.0% [16.4%]. Mean Flesch-Kincaid Grade Level and Flesch Reading Ease scores were 10.7 [2.1] and 47.9 [10.0], respectively. Mean understandability and actionability were 62.8% [18.8%] and 36.2% [26.5%], respectively. There were no significant differences between accuracy (P > 0.2), comprehensiveness (P > 0.5), readability (P > 0.1), understandability (P > 0.3), or actionability (P > 0.2) by authorship.

Conclusions

There is wide variability in the quality of cosmetic botulinum toxin POEMs regardless of authorship type. The majority of materials are written above the recommended reading level and fail to include important content points. It is critical that providers take an active role in the evaluation and endorsement of online patient education materials.

Widespread access to the Internet has increasingly allowed patients to self-educate about health conditions and treatment options by reading printed online education materials (POEMs). Prior studies have demonstrated that high-quality POEMs improve patient satisfaction, while reducing both healthcare costs and clinical time.^1-3 Although POEMs are easily accessible on the World Wide Web, their accuracy and content are not regulated. In addition to providing accurate information, high-quality POEMs must be easy to read and understand.^4-6 Nevertheless, a growing body of evidence across specialties suggests that the majority of POEMs are well above recommended reading levels.^7-14

Injection of botulinum neurotoxin type A (BoNTA) is the most commonly performed cosmetic procedure in the world.¹⁵ Given its elective nature, there is ample time to educate and prepare patients prior to treatment. The most recent consensus statement on cosmetic BoNTA by the Global Facial Aesthetics Consensus Group encourages supplementation of verbal education with written handouts, audiovisual aids, and website content.¹⁶ This builds upon prior recommendations to uncover misinformation from outside sources by assessing patient expectations prior to treatment.¹⁷ Addressing these recommendations is critical to ensure that patients have accurate and understandable information to participate in the shared decision-making process of treatment.¹⁸ Furthermore, improving health literacy has been shown to affect clinical outcomes.¹⁹

A comprehensive evaluation of online patient education materials about cosmetic BoNTA injection does not currently exist, and therefore the quality of these materials is not known. The objective of this study was to examine the accuracy, comprehensiveness, readability, understandability, and actionability of cosmetic BoNTA POEMs.

METHODS

This study was exempt from institutional board review due to the publically accessible nature of the analyzed materials. To identify POEMs about cosmetic BoNTA treatment, a web search was performed (E.B.) in February 2020 with Google (Google LLC, Mountain View, CA). Search phrases included combinations of the following: Botox, cosmetic, wrinkles, face, injection, patient instructions, doctor. Search terms were chosen to reflect common language in order to obtain materials likely to be encountered by patients. No advanced search settings were used and location services were turned off. Prior to searching, search history and cookies were cleared. Websites dedicated to providing clinical patient information about cosmetic BoNTA treatment written in the English language were eligible for inclusion. Exclusion criteria included websites that were inaccessible, sponsored advertisements, duplicates, not written in the English language, news/tabloid articles, blogs, personal stories/experiences, published scientific journal articles, or primarily image- or video-based materials. Thirty-two POEMs were selected and categorized by authorship into 5 categories: academic institutions, medical organizations, unaffiliated health information websites, government-sponsored websites, and botulinum toxin manufacturer websites.

Assessment of Accuracy and Comprehensiveness

The accuracy and comprehensiveness of all 32 websites were evaluated by 3 board-certified facial plastic and reconstructive surgeons (D.S., S.O., and L.N.L.). Accuracy was assessed on a 5-point scale representing the percentage of accurate information in the POEM (<25% accurate, 26%-50% accurate, 51%-75% accurate, 76%-99% accurate, 100% accurate). Next, the same authors assessed the overall comprehensiveness of each website based on a list of 22 key content points regarding cosmetic BoNTA injection (Table 1).

Assessment of Readability

Readability refers to the ease with which a text can be read and understood. The most widely utilized measures of readability for printed materials are the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE). The FKGL and FRE are calculated from formulas that take into account the number of sentences, words, and syllables in a given text. The FKGL reflects the approximate grade level and age range for which a text is suitable. A lower FKGL score indicates the text is easier to read. The FRE is a direct measure of reading ease based on a scale of 0 to 100. A low FRE score indicates a text is complicated, whereas a high FRE score indicates a text is easy to read. In order to calculate FKGL and FRE, the text from each POEM was copied into separate blank Microsoft Word documents (Microsoft, Redmond, WA). A publicly available online word counter (https://wordcounter.net) was then used to calculate the number of sentences, words, and syllables in each POEM. These values were then inserted into the formulas for FKGL and FRE.

Assessment of Understandability and Actionability

The understandability and actionability of all 32 websites were evaluated with the Patient Education Material Assessment Tool for Printed Materials (PEMAT-P) by 2 board-certified facial plastic and reconstructive surgeons (D.S. and L.N.L.). The PEMAT-P is a validated tool from the Agency for Healthcare Research and Quality created to improve patient education materials.²⁰ The PEMAT-P consists of 26 questions; 19 to assess understandability and 7 to assess actionability. Each question is assigned a value of “0” (disagree), “1” (agree), or “N/A” (not applicable) in order to calculate a total score based on the PEMAT-P scoring guide. The total score is then converted to a percentage value, where higher percentage scores indicate better understandability and actionability.

Statistical Analysis

Statistical analysis was performed with XLSTAT version 2019.3.2 (Addinsoft SAS, Paris, France). Due to the small sample sizes of government-sponsored and botulinum toxin manufacturer websites, these POEMs were excluded from the comparative analysis. Mean FKGL, FRE, understandability, actionability, accuracy, and comprehensiveness by authorship were compared by analysis of variance. Linear regression analyses were performed to determine the relationship between FRE, actionability, quality, and comprehensiveness. Analysis of Cronbach’s α demonstrated a good interrater reliability for accuracy (α = 0.83) and actionability/understandability (α = 0.88). Statistical significance for all analyses was set at P < 0.05.

RESULTS

A total of 32 websites with POEMs pertaining to cosmetic BoNTA treatment were evaluated in our study. Of these POEMs, 14 were affiliated with medical organizations (ORG), 9 were unaffiliated health information websites (UHI), 5 were affiliated with academic institutions (EDU), 2 were government-sponsored websites (GOV), and 2 were botulinum toxin manufacturer websites (BTM). Of the organizational websites, 6 were facial plastic surgery organizations, 3 were dermatology organizations, 2 were ophthalmology organizations, 2 were multispecialty organizations, and 1 was a family medicine organization. A list of included websites is available online as Appendix A.

Accuracy and Comprehensiveness

The mean [standard deviation] accuracy score for all 32 POEMs was 4.2 [0.7], which represents a score in which the content is between 76% and 99% accurate. Only 5 POEMs (15.6%) were judged to be 100% accurate. The mean accuracy score for ORG POEMs was 4.2 [0.8], for UHI POEMs was 4.0 [0.5], for EDU POEMs was 4.6 [0.2], for GOV POEMs was 3.5 [0.7], and for BTM POEMs was 4.7 [0.5]. There was no significant difference between accuracy by authorship (P > 0.2; Figure 1).

Comprehensiveness was judged by the percentage inclusion of key content points. The mean comprehensiveness for all 32 POEMs was 47.0% [16.4%] (range, 22.7%-81.8%). The mean POEMs for ORG POEMs was 47.1% [16.6], for UHI POEMs was 42.4% [13.6], for EDU POEMs was 53.6% [22.8], for GOV POEMs was 45.5% [25.7], and for BTM POEMs was 52.3% [9.6]. There was no significant difference between comprehensiveness by authorship (P > 0.5; Figure 1). Comprehensiveness by author affiliation is listed in Table 1.

Among all POEMs, background content points (65.1%) were covered more often than pretreatment/procedural (40.1%) or posttreatment content points (41.8%; P > 0.1). When discussing background information about botulinum toxin, all POEMs mentioned the goal of treatment (100%). More than half of the POEMs included information about botulinum toxin’s mechanism of action (68.8%), common injection sites (75%), duration of treatment (71.9%), and contraindications to treatment (62.5%). However, fewer than half of the POEMs explained that botulinum toxin is more effective for dynamic vs static rhytids (34.4%), or the need for repeat treatments (43.8%). The most infrequently mentioned content points were patient positioning for the procedure (6.3%), postprocedure medication changes (6.3%), and when to seek medical advice after treatment (6.3%). Regarding side effects, the majority of POEMs mentioned pain and bruising at the injection sites (93.8%) and temporary eyebrow or eyelid ptosis (71.9%); however, only a quarter mentioned difficulty with swallowing, speaking, or breathing (25%; P < 0.0001).

Readability

The mean FKGL score for all 32 websites was 10.7 [2.1] (range, 7-15.3). The mean FKGL score for ORG POEMs was 10.5 [1.9], for UHI POEMs was 11.2 [3.3], for EDU POEMs was 12.1 [2.6], for GOV POEMs 11 [2.5], and for BTM POEMs was 12.2 [1.9]. No significant difference was found when comparing the average FKGL scores by authorship (P > 0.8). The mean FRE score for all 32 websites was 47.9 [10.0] (range, of 27.7-66.9) (Figure 1). The mean FRE score for ORG POEMs was 44.1 [10.5], for UHI POEMs was 51.5 [10.5], for EDU POEMs was 50.6 [4.3], for GOV POEMs 50.5 [18.9], and for BTM POEMs was 49.2 [4.2]. There was no significant difference between average FRE score by authorship (P > 0.1; Figure 1).

Understandability and Actionability

The overall mean understandability score was 62.8% [18.8] (range, 18.8%-90.6%), whereas the overall mean actionability score was 36.2% [26.5%] (range, 0%-80%). The highest scoring understandability items were “Item 6: Numbers appearing in the material are clear and easy to understand” (100%) and “Item 7: “The material does not expect the user to perform calculations” (100%), whereas the highest scoring actionability item was “Item 19: The material addresses the user directly when describing actions” (65%). None of the POEMs used tables or calculations and thus Items 17, 27, and 28 were excluded as per PEMAT-P scoring guidelines. The lowest scoring understandability item was “Item 11: The material provides a summary” (2%), whereas the lowest scoring PEMAT-P actionability criteria was “Item 23: The material provides a tangible tool (eg, menu planners, checklists) whenever it could help the user take action” (0%).

For ORG POEMs, the mean understandability was 59.8% [21%] and the mean actionability score was 32.9% [22.3%]. For EDU POEMs, the mean understandability was 73.9% [16.6%] and the mean actionability score was 54% [31.3%]. For UHI POEMs, the mean understandability was 63% [16.5%] and the mean actionability score was 28.6% [27.9%]. For GOV POEMs the mean understandability was 68% [11%] and the mean actionability score was 29% [22%]. For BTM POEMs the mean understandability was 64.1% [15.5%] and the mean actionability score was 17% [18%]. Figure 1 provides a summary of mean understandability and actionability scores. There was no significant difference between average understandability (P > 0.3) or actionability (P > 0.2) score by authorship (Figure 1).

DISCUSSION

Internet use continues to grow each year with 292 million American users in 2019.²¹ Given that 80% of Internet users have searched for health information, the availability of high-quality POEMs is of critical importance.²² Previous surveys have shown that a majority of patients seek guidance in choosing appropriate health information websites.²³ Without standardization or peer review of online health content prior to dissemination, a comprehensive appraisal of POEMs is warranted. Despite nearly 2 million cosmetic BoNTA injections per year, there is a paucity of literature assessing the quality of online patient education materials.¹⁵ The current study demonstrates that cosmetic BoNTA POEMs are provided by a variety of sources, most of which are medical organizations or unaffiliated health information websites. Regardless of source, there is wide variation in the quality of materials.

Although the overall quality of online education materials is multifactorial, providing patients with accurate information is of utmost importance. In the current study, the average accuracy among all POEMs was between 76% and 99%. Surprisingly, only 16% of POEMs were deemed 100% accurate. In other words, 84% of POEMs contained at least 1 false piece of information. Prior studies have shown that most patients who look on the Internet for health information feel empowered to make healthcare decisions based on their search results.²⁴ In the presence of inaccurate information, patient self-education may be a detriment to outcomes in the absence of thorough pretreatment counseling by providers. Furthermore, inaccurate information online may give patients a false sense of security to seek cosmetic BoNTA treatment at heavily advertised locations such as day spas and other nonmedical facilities.

The mean comprehensiveness of BoNTA POEMs as judged by the percentage of key content points included was 47%. Materials written by academic institutions (53%) and BoNTA manufacturers (52%) were most comprehensive; however, no significant difference was found between authorship types. Among all POEMs, background content points (65.1%) were covered more often than pretreatment/procedural (40.1%) or posttreatment content points (41.8%). The lack of posttreatment content may be explained by the fact that providers supply this information in person. Nevertheless, the lack of formal guidelines or a consensus BoNTA POEM risks excluding important treatment information.

Common injection sites were discussed by 75% of POEMs; however, few materials listed the US Food and Drug Administration–approved treatment areas. Although the impact of off-label BoNTA use on daily practice patterns may be insignificant, it is recommended that patients be informed and give consent about such use.²⁵ Regarding side effects, the majority of POEMs discussed pain and bruising at the injection sites (93.8%) and temporary eyebrow or eyelid ptosis (71.9%), but only a quarter mentioned difficulty with swallowing, speaking, or breathing (25%). Paired with the findings that only 6.3% of POEMs addressed when to seek medical advice after treatment, deficits in comprehensiveness have important safety and aesthetic implications. A list of websites referencing the official pharmacologic documentation for the most widely used cosmetic botulinum toxin products is available online as Appendix B.

The current study demonstrates that the average readability of cosmetic BoNTA POEMs is too complex. The mean FKGL score of 10.7 fails to comply with the American Medical Association’s recommended sixth-grade level, while also exceeding the literary capacity of the average US adult.²⁶ Likewise, the mean FRE score of 47.9 was lower than the recommended score of 65. None of the analyzed POEMs were written at a sixth-grade reading level, and only 2 POEMs had an FRE score above 65. These findings are consistent with other studies in the otolaryngology and cosmetic surgery literature.^{11,13,14,27,28} As a result of the mismatch between patient health literacy and available education materials, patients may avoid treatment or turn to nonmedical sources such as popularized blogs, social media, or news articles.

Understandability and actionability are important components of POEM quality that may be overshadowed by accuracy and readability. The PEMAT-P has been widely used throughout the medical literature to gauge understandability and actionability.^20,29-31 In the current study, the mean understandability of BoNTA POEMs was 62.8%, which falls short of the 70% threshold to qualify as adequately understandable.²⁰ Although no significant difference in understandability was found between authorship types, only academic POEMs reached the adequate threshold (73.9%). The lowest scoring understandability item was “Item 11: The material provides a summary,” which was achieved by only 1 POEM. Furthermore, only 23% of POEMs provided visual aids, which is surprising given the aesthetic nature of cosmetic BoNTA treatment. Going forward, the addition of visual aids and a summary are relatively easy ways to increase understandability.

Regarding actionability, the mean score among POEMs was 36.2%. Similar to other studies, these results suggest that BoNTA POEMS fare poorly in guiding patients how to utilize the presented information.^29,31 Just over half of the POEMs (55%) identified at least 1 action the user could take, which was most often related to scheduling a consultation. Based on these findings, we recommend that future POEMs focus on material that is not only understandable, but also actionable.

Despite the widespread use and popularity of cosmetic BoNTA injection, the current study demonstrates the need for higher-quality patient education materials. Specifically, existing materials are written above the recommended reading level and fail to include many important topics related to treatment and periprocedural care. Recognition of these findings should encourage greater awareness by the cosmetic surgery and facial rejuvenation community during the creation, evaluation, and endorsement of online patient education materials for cosmetic BoNTA injection.

LIMITATIONS

The current study is not without limitations. First, the assessment of accuracy, actionability, and understandability is subjective and introduces the opportunity for bias. In order to decrease the risk of bias, POEMs were assessed by 3 board-certified facial plastic and reconstructive surgeons and the results presented as an average score. Second, results obtained from search engine queries may not be consistent between users due to the influence of factors such as location, browser history, and search settings. For the purposes of this study, no advanced search settings were used, and location services were turned off. Additionally, search history and cookies were cleared prior to searching. Third, although websites with primarily audiovisual information were excluded, the authors did not specifically assess audiovisual supplements that appeared in the selected websites. As such, it is possible that content points that were not discussed in text were included in audiovisual form. Similarly, these audiovisual materials may contain inaccuracies that are not within the scope of assessment by the current study. Last, the assessment of readability based on FRE and FKGL is subject to variability due to text formatting. To reduce this risk, headers, titles, and individual bullet items were each considered separate sentences. Although this improved consistency, such methodology tends to make materials appear easier to read.

CONCLUSIONS

This study has demonstrated that there is wide variability in the quality of online patient education materials for cosmetic BoNTA injection irrespective of authorship. Specifically, these materials are written above the recommended reading level and fail to include many of the important topics related to cosmetic BoNTA injection. The cosmetic surgery and facial rejuvenation community must in future assume an active role in the critical evaluation and endorsement of online patient education materials for cosmetic BoNTA injection.

Disclosures

The authors declared no potential conflicts of interest with respect to the research, authorship, and publication of this article.

Funding

The authors received no financial support for the research, authorship, and publication of this article.

REFERENCES