By Yonaira M. Rivera & Vivek Singh
As the public sector continues to adopt chatbots to assist individuals in acquiring information about important health and government services, multilingual access is key to fostering equitable digital health adoption. IMPACT-NJ aims to address this issue by developing policy guidelines to enhance language equity in chatbot responses, with an emphasis on New Jersey’s two most spoken languages, English and Spanish.
Conducting chatbot audits: Strengths, weaknesses, and future directions
The first step in our work includes conducting an audit of chatbots on different state-level websites that provide information on public services. We focused our audit on state websites that house information about Supplemental Nutrition Assistance Program (SNAP), which offers food assistance to low-income families. We selected SNAP as a starting point due to its availability to eligible participants across all 50 states and territories. Particular attention was paid to the chatbot features of each site.
Of the 56 websites identified through Google and the USDA SNAP state directory, only 11 states (19.6%) offered a chatbot; New Jersey did not feature a chatbot on its SNAP website.[1] While all 11 chatbots were available in English only 4 were available in Spanish. Montana’s chatbot was the only chatbot where users could conduct a primary screening of their SNAP benefits eligibility.
Strengths of the audited chatbots include visibility, support for privacy concerns, and user-feedback options. Most websites contained a large “chat-bubble” icon to draw the user’s attention to the chatbot feature. Some chatbots even opened immediately upon entering the SNAP website, enhancing visibility. Chatbot visibility was also more prominent on mobile devices. Upon starting a conversation with the chatbots, some delivered automated messages about privacy. For example, Missouri’s chatbot started by stating, “I value your privacy, so please do not share any of your medical history or health with me.” Finally, a few chatbots provide a feedback option, wherein users can “like” or “dislike” a chatbot response. After delivering this feedback, the chatbot confirms that the user’s feedback was received.
Chatbot areas of improvement include inaccessibility, difficulty interpreting nuance, and unequal performance across different languages. While some chatbots were prominently featured, others were either represented with a small icon in the bottom corner of the screen or were difficult to find (e.g., hidden in obscure links to other pages). Most chatbots were also unable to comprehend nuance in questions, frequently deferring to a generalized message about SNAP benefits hyperlinked in English. Lastly, while some chatbots offered equitable responses in English and Spanish, others could not discern subtle question differences (e.g., “when” versus “how”) or interpret simple SNAP-related questions in Spanish, often deferring to the keyword “SNAP” and hyperlink pages containing SNAP-related information in English. One Spanish chatbot could not interpret “SNAP” at all, indicating it did not understand or by providing no response.
Findings suggest newly developed chatbots should be easily visible (e.g., large widget and “pop-up” upon entering the website), provide a privacy warning, prime users to its response capabilities, elicit real-time feedback, and recognize and respond to nuanced English and Spanish prompts equally. Results will inform our development of a prototype bilingual chatbot for populations in New Jersey.
Yonaira Rivera is an assistant professor of communication at the Rutgers School of Communication and Information and Vivek Singh is an associate professor of library and information science at the Rutgers School of Communication and Information. We also acknowledge the work of doctoral student, M.J. Salas in preparing this brief.
References:
[1] States with chatbots available in English included Arkansas, Florida, Kansas, Kentucky, Massachusetts, Missouri, Montana, New Mexico, Ohio, South Carolina, and Tennessee. States with chatbots available in Spanish included Florida, Kansas, Kentucky, and Massachusetts.