Mohit Jain
Shwetak N. Patel
ABSTRACT
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
- 2002. A.L.I.C.E. Foundation website. (2002). Retrieved January 4, 2017 from http://alicebot.orgGoogle Scholar
- 2013. Mitsuku. (2013). Retrieved January 4, 2017 from http://www.mitsuku.comGoogle Scholar
- 2013. Rose. (2013). Retrieved January 4, 2017 from http://brilligunderstanding.com/rosedemo.htmlGoogle Scholar
- 2016. Facebook Messenger bots. (2016). Retrieved Dec 1, 2016 from https://chatbottle.co/bots/messengerGoogle Scholar
- 2017. Facebook Messenger Alterra. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/alterra.ccGoogle Scholar
- 2017. Facebook Messenger Call of Duty. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/CallofDutyGoogle Scholar
- 2017. Facebook Messenger chatShopper. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatShopperGoogle Scholar
- 2017. Facebook Messenger CNN. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/cnnGoogle Scholar
- 2017. Facebook Messenger Hi Poncho. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/hiponchoGoogle Scholar
- 2017. Facebook Messenger Pandorabots. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/chatbots.ioGoogle Scholar
- 2017. Facebook Messenger Swelly. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/swell.botGoogle Scholar
- 2017. Facebook Messenger Trivia Blast. (2017). Retrieved January 30, 2017 from https://www.messenger.com/t/triviablast1Google Scholar
- Timothy W. Bickmore and Justine Cassell. 2005. Social dialongue with embodied conversational agents. In Advances in natural multimodal dialogue systems. Springer, 23--54.Google Scholar
- Timothy W. Bickmore and Rosalind W. Picard. 2005. Establishing and Maintaining Long-term Human-computer Relationships. ACM Trans. Comput.-Hum. Interact. 12, 2 (June 2005), 293--327. Google ScholarDigital Library
- Dan Bohus and Alexander I. Rudnicky. 2003. Ravenclaw: dialog management using hierarchical task decomposition and an expectation agenda.. In INTERSPEECH. ISCA.Google Scholar
- Susan Brennan. 1990. Conversation as Direct Manipulation: An Iconoclastic View. The Art of Human-Computer Interface Design (1990).Google Scholar
- Justine Cassell. 2000. Embodied conversational agents. MIT press.Google Scholar
- Kathleen Chaykowski. 2016. More Than 11,000 Bots Are Now On Facebook Messenger. (2016). Retrieved Dec 28, 2016 from http: //www.forbes.com/sites/kathleenchaykowski/2016/07/01/ more-than-11000-bots-are-now-on-facebook-messenger/Google Scholar
- O' Brien Chris. 2016. Facebook Messenger chief says platform's 34,000 chatbots are finally improving user experience. (2016). Retrieved February 7, 2017 from http://venturebeat.com/2016/11/11/ facebook-messenger-chief-says-platforms-34000/ -chatbots-are-finally-improving-user-experience/Google Scholar
- Mary Czerwinski, Eric Horvitz, and Susan Wilhite. 2004. A Diary Study of Task Switching and Interruptions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 175--182. Google ScholarDigital Library
- Craig Elimeliah. 2016. Why chatbots are replacing apps. (2016). Retrieved January 20, 2017 from http://venturebeat.com/2016/08/02/ why-chatbots-are-replacing-apps/Google Scholar
- Facebook. 2017. Discover. (2017). Retrieved May 31, 2017 from https://developers.facebook.com/docs/ messenger-platform/discoverGoogle Scholar
- Matt Grech. 2017. The Current State of Chatbots in 2017. (2017). Retrieved Jan 5, 2018 from https://getvoip.com/blog/2017/04/21/ the-current-state-of-chatbots-in-2017/Google Scholar
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183.Google Scholar
- Orange Hive. 2017. First time bot users deserve good bots. (2017). Retrieved Jan 5, 2018 from https://unfiltered.orangehive.de/ first-time-bot-users-deserve-good-bots/Google Scholar
- Jason L Hutchens. 1996. How to pass the Turing test by cheating. School of Electrical, Electronic and Computer Engineering research report TR97-05. Perth: University of Western Australia (1996). Google ScholarDigital Library
- Mohit Jain, Ramachandra Kota, Pratyush Kumar, and Shwetak Patel. 2018. Convey: Exploring the Use of a Context View for Chatbots. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, 6. Google ScholarDigital Library
- Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic Online Evaluation of Intelligent Assistants. In Proceedings of the 24th International Conference on World Wide Web (WWW '15). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 506--516. Google ScholarDigital Library
- Stefan Kopp, Lars Gesellensetter, Nicole C. Krämer, and Ipke Wachsmuth. 2005. Lecture Notes in Computer Science. Springer-Verlag, London, UK, UK, Chapter A Conversational Agent As Museum Guide: Design and Evaluation of a Real-world Application, 329--343. Google ScholarDigital Library
- Q. Vera Liao, Matthew Davis, Werner Geyer, Michael Muller, and N. Sadat Shami. 2016. What Can You Do?: Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16). ACM, New York, NY, USA, 264--275. Google ScholarDigital Library
- Vera Q. Liao, Muhammed Masud Hussain, Praveen Chandar, Matthew Davis, Marco Crasso, Dakuo Wang, Michael Muller, Sadat N. Shami, and Werner Geyer. 2018. All Work and no Play? Conversations with a Question-and-Answer Chatbot in the Wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, 13. Google ScholarDigital Library
- J. C. R. Licklider. 1960. IRE Transactions on Human Factors in Electronics HFE-1 (March 1960), 4--11.Google Scholar
- Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf Between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5286--5297. Google ScholarDigital Library
- Donald A. Norman. 2002. The Design of Everyday Things. Basic Books, Inc., New York, NY, USA. Google ScholarDigital Library
- Amy Ogan, Samantha Finkelstein, Elijah Mayfield, Claudia D'Adamo, Noboru Matsuda, and Justine Cassell. 2012. "Oh Dear Stacy!": Social Interaction, Elaboration, and Learning with Teachable Agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 39--48. Google ScholarDigital Library
- Susan Robinson, Antonio Roque, and David R. Traum. 2010. Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue. In 7th International Conference on Language Resources and Evaluation (LREC). Valletta, Malta. http://people.ict. usc.edu/~traum/Papers/Robinson-LREC2010.pdfGoogle Scholar
- Susan Robinson, David R. Traum, Midhun Ittycheriah, and Joe Henderer. 2008. What would you ask a conversational agent? Observations of Human-Agent dialogues in a museum setting. In Language Resources and Evaluation Conference (LREC). Marrakech (Morocco). http://people.ict.usc.edu/~traum/Papers/ Blackwell-LREC08.pdfGoogle Scholar
- Ronald Rosenfeld, Dan Olsen, and Alex Rudnicky. 2001. Universal Speech Interfaces. interactions 8, 6 (Oct. 2001), 34--44. Google ScholarDigital Library
- Bayan Abu Shawar and Eric Atwell. 2002. A comparison between ALICE and Elizabeth chatbot systems. (2002).Google Scholar
- Statista. 2017. Most popular global mobile messenger apps as of January 2017. (2017). Retrieved February 7, 2017 from https://www.statista.com/statistics/258749/ most-popular-global-mobile-messenger-apps/Google Scholar
- N. Suzuki, K. Ishii, and M. Okada. 1998. Talking Eye: autonomous creature as accomplice for human. In Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110). 409--414. Google ScholarDigital Library
- Indrani M Thies, Nandita Menon, Sneha Magapu, Manisha Subramony, and Jacki O'Neill. 2017. How do you want your chatbot? An exploratory Wizard-of-Oz study with young, urban Indians. In Proceedings of the International Conference on Human-Computer Interaction (HCI) (INTERACT '17). IFIP, 20.Google Scholar
- Marilyn A. Walker, John S. Aberdeen, Julie E. Boland, Elizabeth Owen Bratt, John S. Garofolo, Lynette Hirschman, Audrey N. Le, Sungbok Lee, Shrikanth S. Narayanan, Kishore Papineni, Bryan L. Pellom, Joseph Polifroni, Alexandros Potamianos, P. Prabhu, Alexander I. Rudnicky, Gregory A. Sanders, Stephanie Seneff, David Stallard, and Steve Whittaker. 2001. DARPA communicator dialog travel planning systems: the june 2000 data collection. In INTERSPEECH.Google Scholar
- Joseph Weizenbaum. 1966. ELIZA - A computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36--45. Google ScholarDigital Library
- Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-hao Su, David Vandyke, and Steve J. Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. CoRR abs/1508.01745 (2015). http://arxiv.org/abs/1508.01745Google Scholar
- Yorick Wilks. 2010. Close Engagements with Artificial Companions: Key Social, Psychological, Ethical, and Design Issues. John Benjamins Publishing Company, Amsterdam.Google Scholar
- Steve Young. 1996. A review of large-vocabulary continuous-speech. IEEE Signal Processing Magazine 13, 5 (Sept 1996), 45--.Google Scholar
Index Terms
- Evaluating and Informing the Design of Chatbots
Recommendations
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and DesignAbstractIn this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...
Ubiquitous Chatbots: Workshop on Wearable and Embodied Conversational Agents
UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable ComputersHuman-computer interaction is progressively shifting towards natural language communication, determining the rise of conversational agents. In the context of ubiquitous computing, the opportunities for interacting with new services and systems in a ...
User Expectations of Conversational Chatbots Based on Online Reviews
DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems ConferenceOpen-domain chatbots that can engage in a conversation on any topic received significant attention in the last several years, which opened opportunities for studying user interaction with them. Drawing from reviews of chatbots posted on Google Play, we ...
Comments