Chatbot Research and Design

Local Dutch elections suffer from a voter turnout decrease with 23% between 1986 and 2022. This raises general quality concerns about the local democracy. A Conversational Agent Voting Advice Application (CAVAA) chatbot was developed to aid in solving this problem. The objective of the study was to analyze the effect of the CAVAA chatbot on the perceived political knowledge and intention to vote in municipal elections.

An experimental within-subject study (N = 95) in the run-up to the Dutch municipal elections of March 16, 2022 among a typical Dutch municipality of 27,000 inhabitants was conducted. The CAVAA chatbot asked the participant’s opinion on 20 local issues, provided further background information if desired in a dialogue, and concluded the interaction with a voting advice.

Participants reported a greater understanding of local political issues after CAVAA use. However, no effect was found on the intention to vote.

The study therefore suggests that a voting assistant chatbot may contribute to a higher voter turnout, but the results should be interpreted with caution and further research is desirable.

This study aimed to investigate (1) what errors and conversational repair strategies appear during conversations with a real-life customer service chatbot and (2) how people perceive these errors and repair strategies in terms of user satisfaction, brand attitude, and trust. This study involved a corpus study of real-life conversations (N=100) with a customer service chatbot to investigate which errors and repairs occurred to inform a follow-up online experiment (N=150) on the perception of these errors and repairs. The experiment employed a 3 (error; excess of information, unsolvable question, lack of information) by 3 (repair strategy; repeat, options, defer) mixed subject design with the type of error as between-subjects factor and repair strategy as within-subjects factor. The results revealed that the repair strategy defer most positively impacted perceptions of trust and brand attitude, followed by the strategy options, and lastly repeat. In contrast, no significant main effects of error type nor interaction effects were found on user satisfaction, trust, and brand attitude. However, the open-ended questions revealed that there might be a connection between the nature of the customer request and the repair strategy.

Chatbots are becoming more popular on websites. To ensure their widespread adoption and effectiveness, it is crucial that the development of these assistant technologies prioritizes user experience, integrating advanced computational methods without losing the human-centric perspective. This paper provides a comprehensive analysis of the insights obtained from the Aragón Intelligent Assistant Project, highlighting the main key lessons from deploying a chatbot dedicated to facilitating accessibility to open data for the regional government of Aragón. This article presents the difficulties and obstacles faced to meet the needs of real users while modern natural language processing technologies are being incorporated. The discussion underscores that, notwithstanding the sophistication of artificial intelligence, the user experience should be prioritized through ongoing evaluation and improvement. Chatbots must be continually tunned to align with human interaction paradigms if they are used to be as valuable tools for citizens.

Large language models (LLMs) have generated excitement in many areas and may also make human-like conversations with social robots possible. Drawing from human-robot interaction literature and interviews, we developed Saleshat based on the commercial social robot Furhat and the large language model GPT-4. Saleshat emphasizes refined natural language processing and dynamic control of the robot’s physical appearance through the LLM. Responses from the LLM are processed sequentially, enabling the robot to react quickly. The results of our first formative evaluation with six users engaging in a sales conversation about Bluetooth speakers show that Saleshat can provide accurate and detailed responses, maintain a good conversation flow, and show dynamically controlled non-verbal cues. With our findings, we contribute to research on social robots and LLMs by providing design knowledge for LLM-based social robots and by uncovering the benefits and challenges of integrating LLMs into a social robot.

In this paper, we explore the use of large language models, in this case the ChatGPT API, as simulated users to evaluate designed, rule-based conversations. This type of evaluation can be introduced as a low-cost method to identify common usability issues prior to testing conversational agents with actual users. Preliminary findings show that ChatGPT is good at playing the part of a user, providing realistic testing scenarios for designed conversations even if these involve certain background knowledge or context. GPT-4 shows vast improvements over ChatGPT (3.5). In future work, it is important to evaluate the performance of simulated users in a more structured, generalizable manner, for example by comparing their behavior to that of actual users. In addition, ways to fine-tune the LLM could be explored to improve its performance, and the output of simulated conversations could be analyzed to automatically derive usability metrics such as the number of turns needed to reach the goal. Finally, the use of simulated users with open-ended conversational agents could be explored, where the LLM may also be able to reflect on the user experience of the conversation.

This study employs a quantitative approach to investigate lexical alignment in human-agent interactions involving GPT-3.5 and GPT-4 language models. The research examines alignment performances across different conversational contexts and compares the performance of the two models. The findings highlight the significant improvements in GPT-4’s ability to foster lexical alignment, and the influence of conversation topics on alignment patterns.

Chatbots are increasing their relevance in the global market. Nonetheless, companies are still struggling to develop chatbots that provide their clients with an optimal experience and, so far, few insights have been obtained to improve their related User Experience (UX). This study investigates whether chatbots that consider users’ gender identity result in an improved UX, and whether sensitivity towards this social matter moderates this relationship. Therefore, a one-factor within-subjects experiment was conducted, involving participants interacting with two versions of a buying-assistant chatbot. In one condition, the chatbot provided a more personalised interaction by presuming the user’s gender identity by their sex assigned at birth and conversing with them using a gender-specific language (e.g., ‘women’s clothing’, ‘men’s clothing’). The second condition, instead, used a gender-neutral approach, using gender-neutral language (e.g., ‘clothing’). We hypothesised that the chatbot presuming a cisgender identity of the user would result in a higher UX than the gender-neutral chatbot, and that this effect would be more substantial for users who score low on gender sensitivity. UX was measured by evaluating the chatbot’s Usability, Empathy and Supportiveness. Results indicate that the chatbot presuming a cisgender identity was considered significantly more usable and supportive, but less empathetic. A moderation effect of gender sensitivity on the evaluation of the chatbots was not found.

Agent abuse is emerging as a significant concern in the realm of human-chatbot interactions. Despite the relevance of this phenomenon, from a social and psychological perspective, there have been relatively few published studies on the topic over the years. This calls for special attention and a need for a comprehensive understanding of the challenges posed by abusive behaviors towards conversational agents. Following the PRISMA protocol, this review intends to systematize the knowledge currently available in the scientific domain of chatbot abuse, identifying and evaluating research published between January 1989 and July 2023 across two databases. The review sheds light on the diverse range of studies that have contributed to defining and operationalizing chatbot abuse while exploring avenues for developing evidence-based interventions to discourage verbal mistreatment of conversational agents. By building on empirical, theoretical, and conceptual works, this research promotes awareness and consciousness-raising against chatbot mistreatment, advancing the scientific community's understanding of the complexities surrounding chatbot abuse and its possible implications. In doing so, the study fosters a more ethical, respectful, and empathetic approach toward conversational agents in the digital landscape, and considering the cross-cutting and cross-cultural nature of the issue, the author prompts the need for further empirical research on the topic.

Computational thinking, and by extension, computer programming, is notoriously challenging to learn. Conversational agents and generative artificial intelligence (genAI) have the potential to facilitate this learning process by offering personalized guidance, interactive learning experiences, and code generation. However, current genAI-based chatbots focus on professional developers and may not adequately consider educational needs. Involving educators in conceiving educational tools is critical for ensuring usefulness and usability. We enlisted nine instructors to engage in design fiction sessions in which we elicited abilities such a conversational agent supported by genAI should display. Participants envisioned a conversational agent that guides students stepwise through exercises, tuning its method of guidance with an awareness of the educational background, skills and deficits, and learning preferences. The insights obtained in this paper can guide future implementations of tutoring conversational agents oriented toward teaching computational thinking and computer programming.