Testing Uncertainty – and a chatbot named Ginger

Abstract:

Uncertainty has always been a key challenge for testers – be it with the ambiguous definition of requirements or with unstable test environments. But testing a chatbot adds a completely new level of uncertainty to a tester’s life.

There are a lot of platforms and tools available for chatbot development, but what we lack is a standardized chatbot testing strategy. The way testing is performed on chatbots differs a lot from the “traditional” testing (for example of an app or web portal) due to the apparent randomness of a conversation with a chatbot.

Testing numerous clients’ chatbots and our own chatbot Ginger, we experienced that it is impossible to anticipate and cover all the situations that can happen during a conversation with a chatbot.
As we introduced learning components to Ginger (AI / machine learning, intent training), the chatbot evolved and changed its behaviour compared to previous test runs. This increases the need for regression tests and complicates them at the same time.

There is no limitation on user input- any user can type anything to a chatbot, so functionality, security, performance and exception handling need to be robust. Key areas for testing Ginger were the conversational flow and the natural language processing model, as well as onboarding, personality, navigations, error management, speed and accuracy of the given answers.

Chatting with Ginger, we learned the importance of real-time feedback in order to collect data about unexpected behaviour and invalid data responses.

I will talk about the challenges faced during chatbot testing and how it can go wrong. We will address these challenges and suggest how they can be mitigated by different chatbot testing strategies. We will share our experience with commercial tools for chatbot testing, as well as with using our own advanced automation framework with open source tools.

Traditional testing strategies are not working well for chatbots – testers face new challenges in their daily work, while their experience and structured methods are becoming (to a certain degree) ineffective. Although testing new technology and applications always is exciting, seeing their own strategies and tools fail in chatbot testing can be frustrating even for well-seasoned testers. We aim to provide testers with a better understanding of chatbots like Ginger and help them to apply their critical thinking to deal with uncertainty in their test objects.

Takeaways:

  • Chatbot testing is different from conventional testing.
  • Testing uncertainty in chatbots needs new testing strategies.
  • Test automation can support chatbot testing with open source and commercial tools.
  • Testers have a new view on testing uncertainty and are ready to create a well-defined chatbot testing strategy for effective end-to-end testing of chatbots.

Rajni Singh
Manager – Quality Assurance, Nagarro

Rajni Singh Manager, Quality Assurance: Rajni has more than 12 years of experience in the field of software testing. She is a dynamic, self-driven and result oriented leader with excellent technical ability to deliver seamless solutions. She possesses excellent management and leadership skills with focus on continuous improvement and client-centricity. With understanding of various domains to deliver value to customers, her continuous focus is on digital testing, current industry testing trends like blockchain, AR/VR, IoT, chatbot, AI/ML and agile transformation.