Rebecca Feltman-Frank [1], Ester Masati [1], James Myall [1], Lucas Tozzi [1], Amy Blair [2], Lucia Garcia [3]
[1] Loyola University Chicago Stritch School of Medicine; Chicago, Illinois, United States
[2] Department of Family Medicine-Center for Community and Global Health Department, Loyola University Chicago Stritch School of Medicine; Chicago, Illinois, United States
[3] Parkinson School of Health Sciences and Public Health, Center for Community and Global Health Department, Loyola University Chicago Stritch School of Medicine; Chicago, Illinois, United States
Correspondance: RFeltmanFrank@LUC.edu
ABSTRACT
Introduction: LatinX individuals comprise 18.3% of the United States population, of which 40% have limited English proficiency. Medical Spanish programs are emerging to bridge barriers with these patients, but data are still needed to determine the most effective teaching practices. In this paper, we evaluate the efficacy of a decade old Peer Led Medical Spanish Program (PLMSP) that reaches over 50% of first year medical students at Stritch School of Medicine. Methods: Students were placed into levels based on a pre-test that assessed comfort with Spanish, cultural competency, and reading/audio comprehension. After the completion of twenty classes taught by fluent peers that aligned with components of the medical history students were learning in English at that time, students were re-evaluated using the same exam. Intermediate and above students also completed an Objective Standardized Clinical Examination (OSCE) in which their performance in medical history taking was evaluated by standardized patients. Results: There was significant improvement in Spanish comfort for novice, beginner, and advanced students. Cultural competency growth was noteworthy amongst the novice and intermediate students. Nearly all levels showed statistically significant improvements in Spanish comprehension. For all levels participating in the OSCE, >90% of the history was discussed with standardized patients either agreeing or strongly agreeing that students had appropriate pronunciation, medical vocabulary, conversational fluidity, and cultural awareness. Conclusions: PLMSP offers promising results with regards to medical Spanish level of comfort, comprehension, and clinical performance. Further development of the program should focus on incorporating culture more effectively into the curriculum.
INTRODUCTION
LatinX Americans comprise the second largest ethnic group in the United States, consisting of nearly 60 million individuals or 18.3% of the population according to the U.S. Census Bureau (2018). Nearly 40% of LatinX individuals in the United States have limited English proficiency (1). It has been reported that language discordance in a healthcare setting is associated with increased health disparities, and consequently, more negative health outcomes (2) These include lower patient satisfaction, less access to preventative health care, increased risk of medical errors/adverse events, longer hospital stays, and increased cost of care (3, 4). Since language concordant care is associated with enhanced patient care, there is a growing necessity for effective Medical Spanish education efforts.
In 2012, The National Latino Medical Student Association (LMSA) assessed Medical Spanish Curricula in 132 US Medical Schools in a nationwide survey (1). Eighty-three percent of the schools completed the survey, of which sixty-six percent reported offering a Medical Spanish curriculum. Furthermore, 32% of schools reported an intention to initiate a Medical Spanish curriculum in the near future. These data show that medical schools are aware of the importance of Medical Spanish education in the training of future physicians and are acting to provide it. The increased interest in establishing Medical Spanish curricula in medical schools raises the question of best practices when it comes to curricula learner standards, efficacy, and evaluations.
To date, there are no guidelines on how to structure a Medical Spanish curriculum or how to evaluate programs. One of the reasons for this is that Medical Spanish education efforts are not consistently linked to learner assessments, and when they are, there is much variability in design without reliable outcome measures (5). For example, one longitudinal Medical Spanish program at a southeastern United States medical school evaluated its program utilizing a speaking proficiency phone interview test (6), yet other schools utilize standardized patient structured clinical examinations or oral proficiency interviews (2, 7). Lack of uniformity when it comes to evaluating Medical Spanish programs makes it challenging to compare program outcomes and determine best practices for curriculum establishment. In 2018, the University of Illinois College of Medicine and National Hispanic Health Foundation hosted a multidisciplinary expert panel to establish curricular guidelines for medical school Medical Spanish courses. This panel established goals to standardize Medical Spanish learner competencies and move to assessments utilizing evidence-based methods (8). Despite this important step forward, more research is needed on effective teaching practices in Medical Spanish curricula, which prompted our own evaluation of the efficacy of the Loyola University Chicago Stritch School of Medicine (SSOM) Peer Led Medical Spanish Program (PLMSP), a renowned program that is unique in its fully peer-taught and led structure, its expansiveness, and the manner in which the history oriented curriculum parallels the Stritch Patient Centered Medicine course throughout the academic year.
SSOM’s PLMSP began in 2009 and provides elective educational credit to medical students during their first two years of medical school. SSOM is one of only six medical schools to maintain the peer led method of teaching out of 62 total medical schools participating in the national LMSA study (1). Not only do medical students teach the course to their peers, but they also develop and update the curriculum, gather data on effectiveness, find and train student teachers, advertise and place students into classes, and oversee student growth over the course of the program. We have found that this model promotes acquisition of knowledge and skills across multiple competencies for student leaders, including the domains of professionalism and practice-based learning and improvement. For student participants, the model promotes flexibility and responsiveness to students’ curricular needs and pedagogy. Student participants and leaders alike are fully immersed in their roles as students, teachers, or program leaders. The program is also unique in that the curriculum is entirely focused on gathering medical history and is taught concurrently with the English medical interview for first year medical students. Beyond the distinctive structure of Stritch’s PLMSP, this program is wide-reaching, with greater than 50% of Stritch’s first year medical students successfully completing all coursework for credit.
In response to the increased need for research on effective medical Spanish teaching practices, we evaluated the efficacy of SSOM’s expansive, sustaining, and distinguished PLMSP by assessing student comfort, cultural competency, and comprehension skills before and after the elective and speaking skills following the elective. We hypothesized that the PLMSP improves student performance in the outcomes mentioned above, preparing intermediate, advanced, and proficient students to effectively obtain and comprehend medical histories upon completion of the course.
METHODS
Students interested in taking the Medical Spanish elective at SSOM during the 2020-2021 academic year took a placement exam administered electronically to demonstrate comfort with Spanish, cultural competency, and written and auditory Spanish comprehension (Appendix A). This placement exam served as the pre-test and was used to place students into one of the following course levels: novice, beginner, intermediate, advanced, or proficient. Rather than having hard cut-off values for student placement, students were grouped with others who scored similarly to them on the pretest while simultaneously trying to optimize student:teacher ratios to <12:1. Valuing smaller teacher to student ratios rather than making sure students had strict level cutoff scores reflected the course's efforts to provide students with ample speaking opportunities with access to direct feedback/learning. Furthermore, teachers were encouraged to pull material from higher or lower class levels as needed to assure they were addressing their students’ individualized needs. After students were initially placed into levels, they were able to request to be moved up or down a level during their first three classes if they felt that a different level would better support their personal growth. The data collected was based on the level that the student ultimately chose by the end of the third class and which they remained at for the remainder of the course.
Teachers for the course were selected after an interview process based on language capability, cultural awareness, and teaching experience. There were 19 teachers total, including 14 first year medical students, three nursing students, and two graduate students. There were two teachers assigned per class for any class size over 12 students. Teachers received dedicated training time in which they learned about teaching theories and strategies from the Chair of world languages at a local college. They also had access to standardized materials (Appendix B, available upon request) and received continuous support and guidance from past peer mentors throughout the elective.
Medical Spanish classes were adjusted by teachers such that they could be administered online over Zoom. Students attended 20 classes from September to May. During each class, teachers delivered presentations with standardized daily objectives aligned to components of the medical history. These presentations incorporated interactive learning experiences including listening activities, reading activities, patient-doctor role-play, and games. Much time was spent in breakout rooms to give students the opportunity to practice speaking. Towards the end of the year, more time was dedicated to practicing full patient encounters to prepare for the upcoming Objective Structured Clinical Assessment (OSCE). Beyond the classroom requirements, Medical Spanish students were required to attend four cultural competency events, one practical experience in which students had to actively use Spanish or engage with the LatinX community, and an encounter with a standardized patient (intermediate, advanced, and proficient students only). The cultural competency events included, but were not limited to, monthly online seminars hosted by the National Hispanic Medical Association covering a wide range of topics such as health disparities, film screenings portraying immigrant experiences, and speaker panels of Deferred Action for Childhood Arrivals (DACA) recipients. Practical experiences included registering LatinX patients to vote, attending the LatinX health symposium, or participating in a language exchange buddy program.
During the last class of the elective, students completed the post-test, which was identical to their placement exam (pre-test) and measured comfort, cultural competency, and comprehension changes throughout the curriculum. The pre- and post-tests included the following tools:
Student comfort with speaking and comprehension was self-measured using the Interagency Language Roundtable (ILR) scale (9). This was developed by the U.S. State Department’s Foreign Service Institute (FSI) and has been adopted as the standard measure for language proficiency in U.S. government agencies. The ILR is a scale from 0 to 5 with the following designations:
0 - No proficiency;
1 - Elementary proficiency;
2 - Limited working proficiency;
3 - General professional proficiency;
4 - Advanced professional proficiency;
5 - Functionally native proficiency.
Cultural competency was determined using a five-point Likert scale associated with the statements “I am aware of the manner in which culture influences health care needs and outcomes in the LatinX community” and “I am prepared to engage with LatinX patients in a culturally competent manner.” These statements were written to align with the program’s objectives. While this measure is subjective in nature, this was utilized rather than asking specific cultural questions due to the fact that the cultural components of the class fluctuate per the teachers and course leaders each year in order to adapt to topics most relevant to the current political and cultural environment.
To measure comprehension, students answered ten multiple choice questions based on two audio selections of mock doctor-patient interactions. The other ten questions were based on written medical interactions between a doctor and patient. Both portions of the pre-test were created by the Medical Spanish leaders to align with objectives covered in the elective.
After completion of the final class, intermediate, advanced, and proficient students participated in an OSCE modeled off the SSOM clinical skills course. The OSCE included standardized patients who utilized one of two scripts correlating with responses to a complete history checklist (Appendix C). This checklist consisted of 43 items (Appendix D) and was utilized to ensure students elicited a complete history. The standardized patient was a Spanish speaking individual not involved with the research. Following this exercise, students completed a ten-question online quiz in English to gauge student comprehension of the clinical encounter (Appendix E). Finally, the standardized patient assessed students on pronunciation, vocabulary, conversational ability, and cultural knowledge using a nine-point Likert scale (Appendix F).
Excel was predominately used for data analysis purposes. Mean scores were established for the measures above (comfort, cultural competency, audio/written comprehension, OSCE history completion, OSCE quizzes, OSCE pronunciation/vocabulary, conversational ability/cultural knowledge) and paired t tests were completed to analyze data and establish statistical significance with a predetermined cut off value of .05. Standard deviations and interquartile ranges were also established from each data set to better comprehend the range of values included in each data set. Finally, effect size was calculated from the mean values used for the paired t tests using Cohen’s d.
RESULTS
The 2020-2021 PLMSP at SSOM graduated a total of 100 medical students with varying levels of Spanish proficiencies, including 26 novice, 27 beginner, 28 intermediate, 16 advanced, and three proficient students. These 100 students who were included in our data analysis were enrolled at SSOM as first or second year medical students, completed all course requirements, and completed both the pretest placement exam and the post-test at the end of the course. Figure 1 demonstrates the total starting number of participants specifying reasoning for those who were excluded from our data.
Figure 1: Study Participant Inclusion and Exclusion Data. In order to be included in our study, participants had to be a medical student at Stritch School of Medicine and had to complete all course requirements including pre-test (placement exam), four cultural competency events, one practical experience, a post-test, and an OSCE for intermediate, advanced, and proficient students. Those who were not medical students (our program is open to nursing students and preliminary medical students as well) and who did not complete requirements necessary to receive credit for the course were excluded.
All class levels except for proficient students, increased their comfort with Spanish by the end of the elective, with statistically significant improvement (p<.05) noted for novice, beginner, and advanced students (Table 1).
ILR scale is a device developed by the U.S. State Department’s Foreign Service Institute (FSI) that has been adopted as the standard measure for language proficiency in U.S. government agencies. The scale ranges from 0 to 5 with the following designations: 0 - No proficiency; 1 - Elementary Proficiency; 2 - Limited Working Proficiency; 3 - General Professional Proficiency; 4 - Advanced Professional Proficiency; 5 - Functionally Native Proficiency. Standard deviation (SD) and interquartile range (IQR) values are included for each mean in the table above. IQR is not available for the proficient level due to limited participant number (n=3).
When it comes to cultural competency, while novice, intermediate, and proficient students felt they had improved in this measure, this difference was only significant (p<.05) for the novice and intermediate students with beginner students actually decreasing in their mean cultural competency scores (Table 2).
To measure cultural competency, students used a 5-point Likert scale to demonstrate agreement with the statements “I am aware of the manner in which culture influences health care needs and outcomes in the LatinX community” and “I am prepared to engage with LatinX patients in a culturally competent manner”. Standard deviation (SD) and interquartile range (IQR) values are included for each mean in the table above. IQR is not available for the proficient level due to limited participant number (n=3).
The Medical Spanish comprehension assessment demonstrated significant improvement amongst all levels except for proficient students (Table 3). Upon dividing up the Medical Spanish Comprehension exam into the listening and reading components, novice (p<.001), beginner (p <.001), and advanced students (p=.02) demonstrated statistically significant improvement in terms of listening beginner, intermediate (p<.001), and advanced (p = .001) students demonstrated statistically significant improvement in terms of reading comprehension.
To measure comprehension, students answered 20 multiple choice questions, 10 of which were based on an audio selection of a mock doctor-patient interaction. The other 10 questions were based on a written medical interaction between a doctor and patient. Standard deviation (SD) and interquartile range (IQR) values are included for each mean in the table above. IQR is not available for the proficient level due to limited participant number (n=3). Furthermore, the total mean scores are subsequently subdivided into audio and reading comprehension in each section of the table such that they can be analyzed separately.
In the OSCE, students across all levels assessed (intermediate, advanced, and proficient) successfully asked the majority of the 43 questions associated with the history taught in the clinical skills course at SSOM: intermediate students covered 91%, and advanced and proficient students covered 94% of the topics in the history. In the 10-point comprehension quiz in English to gauge understanding of the encounter, intermediates scored an average of 91%, advanced 94%, and proficient 97%. The standardized patients assessed students on pronunciation, vocabulary, conversational fluidity, and cultural competency with averages of 7.79, 8.60, and 9.00 for pronunciation for intermediate, advanced, and proficient students, respectively. In terms of vocabulary 8.18, 8.73, and 9.00 were the assessments for intermediate, advanced, and proficient students, respectively. When it came to conversational fluidity, intermediates averaged 7.89, advanced 8.53, and proficient 9.00. In terms of cultural competency, intermediate, advanced, and proficient students averaged at 8.61, 8.87, and 9.00, respectively. These numbers indicate that the standardized patients either agreed (8) or strongly agreed (9) that the students had good pronunciation, appropriate use of medical vocabulary, conversed fluidly with full sentences, and demonstrated sufficient cultural awareness (Table 4).
In the OSCE, standardized patients had one of two different scripts correlating with responses to a complete history checklist as the student progressed with the interview. Student points were designated based on their ability to elicit up to 43 different points of the patient history (Appendix D). Quiz scores were based on a 10-point English quiz designed to gauge student comprehension of the history completed with the standardized patient. Pronunciation, vocabulary, conversational fluidity, and cultural competency assessments were assigned scores based on standardized patients’ agreement with statements regarding student ideal capacity with each of these regards (0 = strongly disagree; 9 = strongly agree) as demonstrated in Appendix F. Standard deviation (SD) and interquartile range (IQR) values are included for each mean in the table above. IQR is not available for the proficient level due to limited participant number (n=3).
DISCUSSION
The results indicate a promising effect of the Peer Led Medical Spanish Program across a wide range of competencies including student comfort, reading and listening comprehension, and clinical performance. Particular areas of success include the notable improvement in comprehension exam scores across all levels of proficiency (with the exception of proficient students), which are statistically significant (p<0.05) and performance on the OSCE, with >90% history completion and quiz scores for all levels of proficiency assessed (intermediate, advanced, proficient). Of note, results from the self-reported cultural competency surveys did not display similar improvement. This represents an area of study that can be analyzed and revised in subsequent Medical Spanish curricula to further enhance the cultural experience for future student cohorts. Furthermore, proficient students did not demonstrate statistically significant improvement across any measures taken. While this is likely due to their competent performance in the pre-test, further study of the proficient student cohort is needed.
This study sets an important precedent of measuring outcomes of Medical Spanish programs to determine efficacy, and ultimately, guide best teaching practices. The inclusion of measurements for cultural competency, objective evaluation methods in the form of the audio/reading comprehension exam, and an OSCE with its associated comprehension quiz allowed a more comprehensive understanding of strengths and weaknesses of the program that can be used to guide curricular improvement at SSOM and to provide direction for other medical schools working to create a Medical Spanish program. Furthermore, the peer-led model, at both a teaching and administrative level, provides increased student leadership development and academic skills (e.g., Curricular development, assessment creation, setting of learner goals and objectives), readily allows for course adaptation based on current student interests and community needs, and broadens opportunities for learning Medical Spanish in schools that may not have sufficient faculty or financial resources to meet student demand.
While this study provides insight into the efficacy of the PLMSP, there were a lot of students who did drop out of the course and who were excluded from our analysis. Gathering data on reasons for students to drop out of the course would be beneficial to curricular improvement if it were collected in the future. It also must be acknowledged that many of the metrics used are non-validated, and results may be open to biases. The Medical Spanish comprehension test was created by the Medical Spanish leaders and is based on important aspects of the curriculum as determined and agreed upon by them. The cultural competency and degree of comfort tests are subjective in nature, and while this encourages students to assess these characteristics of themselves, the interpretation of these results must take the subjectivity into account when evaluating improvement. The OSCE performances were judged by the standardized patients themselves, and while they had a checklist to assess percent completion of the history, the evaluation on vocabulary, fluidity, cultural competency, and pronunciation were evaluated on a 9-point Likert scale and were subjective to biases of the standardized patients. This allows for variability in interpretation of performance in these areas. Furthermore, the OSCE was conducted after the completion of the course only, so there was no way to judge if clinical performance, itself, was impacted by the PLMSP.
Beyond limitations with the metrics, some bias may be involved in terms of the curriculum. While the curriculum of the PLMSP is based on standardized interactive presentations, individual student experiences are certainly dependent on the content delivery, which is likely to differ based on each teacher’s style of instruction. Of note, instruction for the 2020-2021 Medical Spanish Program was conducted fully online via Zoom due to the COVID-19 Pandemic. This mode of educational delivery, while convenient and necessary, creates significant difficulties in fostering the cultural competency component of the curriculum, as previously students were able to engage in-person with activities that satisfy this requirement, including educational cultural lectures and shadowing Spanish-speaking physicians.
CONCLUSIONS
Overall, the SSOM PLMSP improved Spanish language competency in medical students. Further program development should focus on incorporating culture more effectively into the curriculum and developing measurement tools for more advanced students. This student-led program serves as a model that can accommodate students of various levels, be far-reaching in terms of student enrollment, reinforce the medical history taught in English, and be sustained over time. It is a helpful example to other schools aiming to establish a Medical Spanish curriculum that promotes student leadership and academic skills, while contributing to the development of more standardized guidelines in the effective teaching of Medical Spanish.
SUPPLEMENTARY INFORMATION
Appendix A
Appendix B: Available upon request.
Appendix C
Appendix D
Appendix E
Appendix F
ACKNOWLEDGEMENTS
We would like to acknowledge the peer teachers who developed content throughout the program’s history, including founders Erin Stratta, MD and the late Jason Howell, MD, Katherine O’Rourke, MD, and the medical students Nicholas Sasso and Jaquelin Solis Solis. We would also like to thank the staff of the Center for Simulation Education, especially Donna Quinones, and all volunteers for the OSCE.
DISCLOSURES
Funding: Not applicable.
Conflicts of interest: None.
Availability of data and materials: Upon request.
Code availability: Not applicable.
Authors’ contributions: Authors listed in the manuscript have contributed per submission guidelines and standards for authorship.
Ethics approval: The questionnaire and methodology for this study was approved by the Human Research Ethics committee of Loyola University Chicago (212132).
Consent to participate: Informed consent and consent to publish this data was obtained from all individual participants included in the study.
REFERENCES
1. Morales R, Rodriguez L, Singh A, et al. National Survey of Medical Spanish Curriculum in U.S. Medical Schools. J Gen Intern Med. 2015;30(10):1434‐1439. doi:10.1007/s11606-015-3309-3
2. Ortega P, Pérez N, Robles B, Turmelle Y, Acosta D. Teaching Medical Spanish to Improve Population Health: Evidence for Incorporating Language Education and Assessment in U.S. Medical Schools. Health Equity. 2019;3(1):557‐566. Published 2019 Nov 1. doi:10.1089/heq.2019.0028
3. Al Shamsi H, Almutairi AG, Al Mashrafi S, Al Kalbani T. Implications of Language Barriers for Healthcare: A Systematic Review. Oman Med J. 2020 Apr 30;35(2):e122. doi: 10.5001/omj.2020.40.
4. Divi C, Koss RG, Schmaltz SP, Loeb JM. Language proficiency and adverse events in US hospitals: a pilot study. Int J Qual Health Care. 2007 Apr;19(2):60-7. doi: 10.1093/intqhc/mzl069. Epub 2007 Feb 2.
5. Hardin KJ, Hardin DM. Medical Spanish programs in the United States: a critical review of published studies and a proposal of best practices. Teach Learn Med. 2013;25(4):306‐311. doi:10.1080/10401334.2013.827974
6. Reuland DS, Frasier PY, Slatt LM, Alemán MA. A longitudinal medical Spanish program at one US medical school. J Gen Intern Med. 2008;23(7):1033-1037. doi:10.1007/s11606-008-0598-9
7. Ortega P, Park YS, Girotti JA. Evaluation of a Medical Spanish Elective for Senior Medical Students: Improving Outcomes through OSCE Assessments. Med Sci Educ. 2017 Jun;27(2):329-337. doi: 10.1007/s40670-017-0405-5. Epub 2017 Mar 27.
8. Ortega P, Diamond L, Alemán MA, et al. Medical Spanish Standardization in U.S. Medical Schools: Consensus Statement From a Multidisciplinary Expert Panel. Acad Med. 2020;95(1):22‐31. doi:10.1097/ACM.0000000000002917
9. Interagency Language Roundtable Language Skill Level Descriptions - Speaking. http://www.govtilr.org/skills/ilrscale2.ht