Readability of Selected Domestic Public Health Communications during the COVID-19 Pandemic

Anjali Om, BA1, Amna Nawaz, BS2, and Namita Mathew, BS1
1 Emory University School of Medicine, Atlanta, GA 30322, USA
2 Georgetown University School of Medicine, Washington, DC 20007, USA

Correspondence concerning this article and requests for reprints should be addressed to Anjali Om (anjali.om@emory.edu)

ABSTRACT

Background: Since the outbreak of the novel coronavirus pandemic, public health Internet platforms have been inundated with constantly updating information. Despite American Medical Association and National Institutes of Health recommendations that patient education be no greater than a sixth grade reading level, health documents often exceed this. To our knowledge, the readability of public health education material related to COVID-19 has not been reported.

Aims: To analyze the readability of public health communications regarding COVID-19.

Methods: We searched CDC.gov and five state Department of Public Health sites (New York, New Jersey, Illinois, California, and Massachusetts) for COVID-19 educational material and obtained six readability indices by uploading article text into an online tool.

Results: Across 623 articles, mean Flesch Kincaid Reading Ease was 41.7, corresponding to a “difficult” reading level. Mean scores indicating grade levels appropriate for understanding were: Flesch Kincaid Grade Level 12.8, Gunning Fog Index 14.6, SMOG Index 11.4, Coleman Liau Index 13.8, and Automated Readability Index 12.7. Only 0.006% (n=4) of articles had readability scores within recommended levels.

Conclusion: Our study shows that public health communications regarding COVID-19 are well above recommended grade levels.

INTRODUCTION

Since the beginning of the novel coronavirus outbreak in December 2019, public health organizations have inundated Internet platforms by constantly updating information. Studies have repeatedly demonstrated that Americans are increasingly using the Internet for health care information; it is estimated that between 63-80% of Internet users have looked for online health information [1,2]. While the sheer quantity of information available to the public may seem conducive to broad-reaching education, this information often requires a high level of reading comprehension to be adequately received. 

Health literacy is defined by the National Academy of Medicine as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” [3]. Health literacy may play an important role in health outcomes, as patients with lower health literacy reportedly have worse health status and higher rates of hospitalization [4-6]. The only national data published on American literacy estimated that in 2003, 14% of Americans had below basic health literacy, defined as no more than the most simple and concrete literary skills [7-9].   For example, those with below basic health literacy might be able to locate the time and date of a medical appointment on a form but would be unable to read a pamphlet on hypertension and identify two features of the disease [8]. Advocates for improving health literacy argue that it encompasses a balance between the skills and abilities of the target audience and the demands and complexity of the materials delivered [10]. In other words, health literacy is dependent on both the skill of the user and the level of difficulty at which the information is presented. Health literacy, is well correlated to education and overall literacy [11,12]. Studies have shown that the average reading level of American adults is around eighth grade, while the average reading level of Medicaid enrollees is about fifth grade [13]. In addition, average reading level is three to five grades below the highest level of education completed [14].

Because health literacy seems to be an important indicator of health outcomes, the American Medical Association (AMA) and National Institutes of Health (NIH) have recommended that all patient education material be no greater than a sixth grade reading level [15].  Despite these guidelines, previous studies have shown that many medical subspecialties publish material well above the recommended reading level [14-17]. After the Ebola epidemic, a similar study showed difficult readability of public health material published on the Ebola virus disease [18]. However, the readability of public health education material related to COVID-19 has not been reported. The purpose of this study is to analyze the readability of communication resources distributed by the Centers for Disease Control and Prevention (CDC) and state Departments of Public Health (DPH) regarding COVID-19 using six readability indices.

METHODS

Study Design

On May 29, 2020, a list of educational materials published by the CDC regarding COVID-19 was populated by searching CDC.gov for “Communication Resources” filed under “Coronavirus Disease 2019 (COVID-19)” [19]. All articles published on or before the date of search were collected.

A list of states with the highest number of confirmed COVID-19 cases as of May 29, 2020 was also populated [20].  For the five states with the highest case numbers (New York, New Jersey, Illinois, California, and Massachusetts) DPH sites were searched [21-25]. The first 100 unique results when sorted by relevance for two independent search criteria, “COVID” and “coronavirus,” were collected.

Inclusion and Exclusion Criteria

Figure 1 illustrates the selection process used to identify qualifying articles. Content of each search result was evaluated and filtered to include only articles relevant to COVID-19. Results not classified as delivering public health information regarding COVID-19, presumed to be populated due to imprecise website algorithms, were excluded. Similarly, the target audience of each article was evaluated and classified as either “healthcare providers,” encompassing those responsible for diagnosing, processing collection specimens, or treating COVID-19, or “non-providers,” any member of a non-healthcare related field receiving COVID-19 related information. Articles targeted towards “healthcare providers” were not considered public health education material and were therefore excluded. Pages not in English, pages with fewer than 100 words of text, and pages exclusively composed of resource links, graphics, figures, or search tools were also excluded.  Broken URL links were also excluded. 

Readability Assessment

Readability assessment was then performed by uploading text from each qualifying article into a free online tool (https://www.webfx.com/tools/read-able/check.php) [26]. Website identifiers, internal webpage navigation links, advertisements, graphs, pictures, videos, tables, figures, figure legends, captions, additional resource URLs, references, legal disclaimers, content sources, and copyright notices were removed from the assessment to prevent altered results. From the online tool, six different numerical readability indicators were obtained: Flesch Kincaid Reading Ease (FRE), Flesch Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG) Index, Coleman Liau Index (CLI), and Automated Readability Index (ARI). Table 1 lists the formulas by which each readability index is calculated.

RESULTS

445 unique communication articles published by the CDC, and 782 unique articles by five state DPH sites (133 from NY, 130 from NJ, 187 from IL, 181 from CA, and 151 from MA) were collected through search criteria. Of these, a total of 50.8% (n=623) were included based on inclusion and exclusion criteria (271 from CDC, 53 from NY DPH, 47 from NJ DPH, 111 from IL DPH, 51 from CA DPH, and 90 from MA DPH).

Mean readability statistics are listed in Table 2. FRE scores indicate readability on a scale from 0-100, with higher scores corresponding to greater reading ease; scores 0-30 indicate very difficult reading at or above a college graduate level, while scores 90-100 indicate very easy reading at an estimated fifth grade level [27]. Overall mean FRE for all 623 included articles was 41.7 (SD 14.8; 95% CI 40.5-42.8), corresponding to “difficult to read.” Mean word count was 883 (SD 891; 95% CI 813-953) and mean percentage of complex words, defined as words containing three or more syllables, was 18.8 (SD 5.3; 95% CI 18.3-19.2). Mean readability statistics for each public health organization are listed in Table 2.

The remaining indices (FKGL, GFI, SMOG, CLI, and ARI) produced scores corresponding to a grade level deemed appropriate for understanding [27]. For all included articles, the overall mean FKGL was 12.8 (SD 3.8; 95% CI 12.4-13.0), mean GFI 14.6 (SD 3.2; 95% CI 14.3-14.8), mean SMOG index 11.4 (SD 4.3; 95% CI 11.1-11.7), mean CLI 13.8 (SD 3.1; 95% CI 13.5-14.0), and mean ARI 12.7 (SD 4.2, 95% CI 12.3-13.0). Mean readability index scores for each individual public health organization analyzed are depicted in Figure 2. For included articles, average grade level of readability was calculated by averaging the five indices that produced recommended grade levels for understanding. Of the 623 total included articles, 0.006% (n=4), 1 from IL DPH and 3 from NJ DPH, had average readability scores within the AMA/NIH guidelines for patient education materials (at or below a sixth grade reading level).

DISCUSSION

In the era of the COVID-19 pandemic, online education is becoming increasingly essential to stay updated on changing guidelines [28]. For a communicable disease, especially one for which containment depends on individual preparedness, it is especially important to be able to understand and follow public health policies that limit physical contact [29]. At the time of data collection, New York state had more than 368,000 confirmed cases, New Jersey had 157,000, Illinois had 115,000, California had 101,000 and Massachusetts nearly 95,000 [17]. Four of the five states also had at least partial lockdowns still in place, as Massachusetts began a plan to reopen on May 18 while New Jersey still had a statewide stay-at-home order set to expire June 5 [30].  For these states especially, online public health communication may represent a major source of information.

Our analysis is consistent with previously published material showing that medical information is often distributed at reading levels well above both the average and recommended levels.  It is important to note that states have individual variation in literacy. For example, 10% of Massachusetts’ population has below basic literacy skills, while for New York this number is 22% [8]. Regardless, the AMA/NIH recommendation for readability remains the same nationwide. In addition to readability, article length may also influence patient understanding.  Our study found an average word length between 622 and 1139 words. Based on a calculated average reading speed of 200-330 words per minute and not accounting for readability scores beyond level of average understanding, these articles would require a mean reading time of 1.9-5.7 minutes [31].  It is worth noting that a sixth grade reading level corresponds to a reading speed of about 170 words per minute, increasing the mean reading time for articles in this study to 3.7-6.7 minutes [32].

The mean FRE score of the included articles in this study ranges from 36-53. Texts with FRE scores between 30-50 are considered “difficult to read,” examples of which include academic or scholarly magazines [33]. Texts with FRE scores between 50-60 are considered “fairly difficult to read,” likened to magazines like The Atlantic [34,35]. For reference, the AMA/NIH guidelines roughly correspond to a FRE between 80-90, equivalent to a popular fiction novel [32]. The FKGL assessment inputs the same variables as the FRE put provides a grade level of education required to easily understand the material; for example, a FKGL score of 12.8 corresponds to a reading level between the 12th and 13th grades [36]. Like the FKGL, the GFI, SMOG, CLI, and ARI all provide a grade level at which the text can be easily understood and use higher scores to indicate more difficult texts. The GFI and SMOG tests use sentence length, defined as number of words per sentences, and the number of complex words, defined as words containing three or more syllables, to calculate readability [14,37]. The CLI takes into consideration word length, defined as number of letters per 100 words, and sentence length, defined as number of sentences per 100 words, rather than syllables [14,38].

This study has certain notable limitations. First, it included only public health information distributed by the CDC and five state DPHs.  Because our web tool was limited to analyzing text entries, many organizations that publish predominately graphical information, including but not limited to WHO, UN, and UNICEF, were not analyzed.  While the CDC and DPH are major and reputable sources of public health communication, they are not representative of all outlets, particularly international sources, that may distribute information regarding COVID-19 [39].  Second, this study relied on search criteria to populate lists of public health communications on each site studied. Despite secondarily verifying that articles collected were related to COVID-19, this methodology could only exclude superfluous articles, not include content beyond that which each website algorithm generated.  Third, despite using six different numerical indices, this study did not include all available metrics that may assess readability [40]. Fourth, despite being widely used, readability indices may present a biased and incomplete characterization of understanding.  For example, indices that correlate syllables with complexity (FRE, FKGL, GFI, and SMOG) may naively assume that words like educational are more difficult to understand than tropism [41]. Finally, readability may be influenced by factors not considered by readability indices, such as website organization, inclusion of substantiating images and graphs, and graphic design.

CONCLUSION

Our study shows that public health communications distributed by the CDC and state DPH regarding COVID-19 are at grade levels well above the AMA/NIH recommendations and substantiates a similar trend demonstrated by other analyses of published health information.  While the Internet exposes users to a vast quantity of information, the quality of information that is outside of a comprehensible level remains uncertain, even at trusted public sources for information. Along with work to ensure health literate documents, further studies that explore methods of increasing readability may help decrease health disparities and improve health outcomes.

REFERENCE

  1. B. W. Hesse, D. E. Nelson, G. L. Kreps et al., “Trust and sources of health information: the impact of the internet and its implications for health care providers: findings from the first health information national trends survey,” Archives of Internal Medicine, vol. 165, no. 22, pp. 2618–2624, 2005.

  2. H. S. Wald, C. E. Dube, and D. C. Anthony, “Untangling the web-the impact of internet use on health care and the physician-patient relationship,” Patient Education and Counseling, vol. 68, no. 3, pp. 218–224, 2007.

  3. Health Literacy. National Network of Libraries of Medicine. https://nnlm.gov/initiatives/topics/health-literacy. Accessed May 24, 2020.

  4. Health literacy: report of the Council on Scientific Affairs. Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. JAMA. 1999;281(6):552‐557.

  5. Office of the Surgeon General (US); Office of Disease Prevention and Health Promotion (US). Proceedings of the Surgeon General's Workshop on Improving Health Literacy: September 7, 2006, National Institutes of Health, Bethesda, MD. Rockville (MD): Office of the Surgeon General (US); 2006. Panel 1: Health Literacy, Literacy, and Health Outcomes. Available from: https://www.ncbi.nlm.nih.gov/books/NBK44260

  6. Ventura HO, Piña IL. Health Literacy: An Important Clinical Tool in Heart Failure. Mayo Clinic Proceedings. 2018;93(1):1-3. doi:10.1016/j.mayocp.2017.11.009.

  7. Understanding Literacy & Numeracy. Centers for Disease Control and Prevention.
    https://www.cdc.gov/healthliteracy/learn/UnderstandingLiteracy.html. Published November 13, 2019. Accessed May 24, 2020.

  8. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: Results from the 2003 National Assessment of Adult Literacy. National Center for Educational Statistics; 2006. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006483. Accessed May 24, 2020.

  9. Cutilli CC, Bennett IM. Understanding the health literacy of America: results of the National Assessment of Adult Literacy. Orthop Nurs. 2009;28(1):27‐34. doi:10.1097/01.NOR.0000345852.22122.d6

  10. Institute of Medicine (US) Roundtable on Health Literacy. Measures of Health Literacy: Workshop Summary. Washington (DC): National Academies Press (US); 2009. 6, Measuring Health Literacy: What? So What? Now What? Available from: https://www.ncbi.nlm.nih.gov/books/NBK45386/

  11. van der Heide I, Wang J, Droomers M, Spreeuwenberg P, Rademakers J, Uiters E. The relationship between health, education, and health literacy: results from the Dutch Adult Literacy and Life Skills Survey. J Health Commun. 2013;18 Suppl 1(Suppl 1):172‐184. doi:10.1080/10810730.2013.825668

  12. Rudd R. E. Health literacy skills of U.S. adults. American Journal of Health Behavior. 2007;31(Suppl. 1):S8–S18. doi: 10.5555/ajhb.2007.31.supp.S8

  13. Weis  B.  Health Literacy: A Manual for Clinicians. Chicago, IL: American Medical Association, American Medical Foundation; 2003.

  14. Huang G, Fang CH, Agarwal N, Bhagat N, Eloy JA, Langer PD. Assessment of Online Patient Education Materials From Major Ophthalmologic Associations. JAMA Ophthalmol. 2015;133(4):449–454. doi:10.1001/jamaophthalmol.2014.6104

  15. Eltorai AE, Han A, Truntzer J, Daniels AH. Readability of patient education materials on the American Orthopaedic Society for Sports Medicine website. Phys Sportsmed. 2014;42(4):125‐130. doi:10.3810/psm.2014.11.2099

  16. Kher A, Johnson S, Griffith R. Readability Assessment of Online Patient Education Material on Congestive Heart Failure. Advances in Preventive Medicine. 2017;2017:1-8. doi:10.1155/2017/9780317.

  17. Smith K, Buchanan P, Mcdonald P. How easy is it for a lay audience to read medical journals? A survey of the readability scores of a sample of research papers on diabetes. The Lancet. 2017;390. doi:10.1016/s0140-6736(17)33017-9.

  18. Castro-Sánchez E, Spanoudakis E, Holmes AH. Readability of Ebola Information on Websites of Public Health Agencies, United States, United Kingdom, Canada, Australia, and Europe. Emerg Infect Dis. 2015;21(7):1217‐1219. doi:10.3201/eid2107.141829

  19. What’s New. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/whats-new-all.html. Accessed May 29, 2020.

  20. Cases in the U.S. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/cases-in-us.html. Accessed May 29, 2020.

  21. Department of Health. New York State. https://www.health.ny.gov/. Accessed May 29, 2020.

  22. Department of Health. The State of New Jersey. https://www.nj.gov/health/. Accessed May 29, 2020.

  23. Department of Health. Illinois Department of Public Health. https://www.dph.illinois.gov/. Accessed May 29, 2020.

  24. California Department of Public Health. CA.gov. https://www.cdph.ca.gov/. Accessed May 29, 2020.

  25. Department of Public Health. Mass.gov. https://www.mass.gov/orgs/department-of-public-health. Accessed May 29, 2020.

  26. Readability Test Tool. Readable. https://www.webfx.com/tools/read-able/. Accessed May 24, 2020.

  27. Paz, S.H., Liu, H., Fongwa, M.N. et al. Readability estimates for commonly used health-related quality of life surveys. Qual Life Res 18, 889–900 (2009). https://doi.org/10.1007/s11136-009-9506-y

  28. Sahi PK, Mishra D, Singh T. Medical Education Amid the COVID-19 Pandemic [published online ahead of print, 2020 May 14]. Indian Pediatr. 2020;S097475591600181.

  29. Paakkari L, Okan O. COVID-19: health literacy is an underestimated problem. The Lancet Public Health. 2020;5(5). doi:10.1016/s2468-2667(20)30086-4.

  30. See How All 50 States Are Reopening. The New York Times. https://www.nytimes.com/interactive/2020/us/states-reopen-map-coronavirus.html. Accessed May 29, 2020.

  31. Rayner, K., Slattery, T.J. & Bélanger, N.N. Eye movements, the perceptual span, and reading speed. Psychon Bull Rev 17, 834–839 (2010). https://doi.org/10.3758/PBR.17.6.834

  32. Carver RP. Silent Reading Rates in Grade Equivalents. Journal of Reading Behavior. 1989;21(2):155-166. doi:10.1080/10862968909547667.

  33. Flesch  R.  A new readability yardstick.  J Appl Psychol. 1948;32(3):221-233.

  34. Gervasi O. Computational Science and Its Applications--ICCSA 2008: International Conference, Perugia, Italy, June 30 - July 3, 2008 ; Proceedings. Berlin: Springer; 2008.

  35. Cole RJ. Structure and Drafting of Safeguards Regulatory Documents. Washington: Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Division of Safeguards, Fuel Cycle, and Environmental Research; 1977.

  36. Williams AM, Muir KW, Rosdahl JA. Readability of patient education materials in ophthalmology: a single-institution study and systematic review. BMC Ophthalmol. 2016;16:133. Published 2016 Aug 3. doi:10.1186/s12886-016-0315-0

  37. McLaughlin  GH.  SMOG grading: a new readability formula.  J Read. 1969;12(8):638-646.

  38. Coleman, M., & Liau, T. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2), 283–284. https://doi.org/10.1037/h0076540

  39. Jernigan DB; CDC COVID-19 Response Team. Update: Public Health Response to the Coronavirus Disease 2019 Outbreak - United States, February 24, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(8):216‐219. Published 2020 Feb 28. doi:10.15585/mmwr.mm6908e1

  40. Kakazu R, Schumaier A, Minoughan C, Grawe B. Poor Readability of AOSSM Patient Education Resources and Opportunities for Improvement. Orthop J Sports Med. 2018;6(11):2325967118805386. Published 2018 Nov 2. doi:10.1177/2325967118805386

  41. Kandula S, Zeng-Treitler Q. Creating a gold standard for the readability measurement of health texts. AMIA Annu Symp Proc. 2008;2008:353‐357. Published 2008 Nov 6.