Researching Chatbot Checkbox Behavior:
From testing one issue to uncovering another
This project is not about UI design but a case study on a usability test, showcasing how we set out to test one problem and uncovered another issue within the interface.
System Overview
PandoSelect, a subsidiary of PandoLogic, simplifies recruitment with two integrated systems: a chatbot application system that handles initial candidate interactions by asking predefined questions and a candidate management system that allows recruiters to manage open positions, view detailed candidate summaries, and send automated rejection emails with ease.
Previous Behavior
The concern
I speculated that this behavior might be confusing for users, leading them to potentially provide incorrect answers or abandon the application process altogether. Despite my concerns, there wasn't sufficient support from the development team to change the design and behavior.
To address this, I needed to demonstrate the importance and value of the proposed changes. This required concrete evidence to support my theory that the current design was problematic.
Usability Test
Objective
Confirm or disprove or confirm the hypothesis that disabling checkboxes based on selecting other checkboxes from the same list will harm the feature's usability.
Participants
I tested 15 participants: 5 on mobile, 5 on desktop, and 5 more on mobile in a second round to validate updates. I moderated all sessions.
Methodology
The usability test was conducted primarily in person, one-on-one, with some sessions held remotely via Zoom.
The Research Plan
We prepared two versions of persona in advance:
-
With Data in Checkboxes: This version had various options in the checkboxes.
-
Without Data in Checkboxes: This version required users to click on “None of these".
The Flow
The user receives their persona along with a link to the chat
The user is assigned a task: apply for a job on behalf of the persona
Once the user submits their response, we replace the persona
We observe and analyze the user's behavior as they handle the task
The results
of users managed to get out of the situation
of users understand how to change the disabled mode
of users are aware that all options are disabled when 'none of this' is selected
The mobile test revealed unexpected findings. One user didn’t reach a question because they didn’t notice the available scrolling to see more answers. This issue was identified thanks to the usability test.
Out of the five people tested, I could only refer to the results from four. This prompted us to redesign the scrolling feature to make it more obvious.
Usability Test
Objective
Confirm or disprove or confirm the hypothesis that disabling checkboxes based on selecting other checkboxes from the same list will harm the feature's usability.
Participants
I tested 15 participants: 5 on mobile, 5 on desktop, and 5 more on mobile in a second round to validate updates. I moderated all sessions.
Methodology
The usability test was conducted primarily in person, one-on-one, with some sessions held remotely via Zoom
The Research Plan
We prepared two versions of persona in advance:
-
With Data in Checkboxes: This version had various options in the checkboxes.
-
Without Data in Checkboxes: This version required users to click on “None of these"
The Flow
The user receives their persona along with a link to the chat
The user is assigned a task: apply for a job on behalf of the persona
Once the user submits their response, we replace the persona
We observe and analyze the user's behavior as they handle the task
.png)
Second Round
I tested five more mobile users after the update; scrolling issues were resolved, but the disabled mode remained unclear. We refined the interface to address this.
We adjusted the scrolling to continue usability testing
Before:
.png)
The question appears to the user when the first sentence is truncated
After:
.png)
the first answer is fully visible, but the last one is cut off. Due to the truncated sentence above, this small, minor change made a big difference."
After making small changes, we were able to continue with the test
We adjusted the scrolling to continue usability testing
Before:
.png)
The question appears to the user when the first sentence is truncated
.png)
After scrolling up to view the full answer, the screen appears as shown. One user missed the 'none of these' option because they forgot about the additional options below and were unaware of the scrolling
After:
.png)
The first answer is fully visible, but the last one is cut off. Due to the truncated sentence above, this small, minor change made a big difference.
After making small changes, we were able to continue with the test
The new design
.png)
.png)
What?
In the chatbot application system, there was a question where candidates had to mark their answers using checkboxes. The design included a list of checkboxes with various possible answers, with the last option being 'none of these'. The behavior was such that once a candidate selected one of the answers, the 'none of these' option became disabled, and vice versa—selecting 'none of these' disabled the other answers.
SO....
To validate or refute my concerns, I conducted a usability test. This involved observing how users interacted with the checkbox behavior and identifying any confusion or frustration they experienced. The goal was to gather empirical data to present a compelling case to the development team, highlighting the need for a design change to improve the user experience.
Check out More of my Projects
Check out More of my Projects
Overall Impact
The usability tests uncovered a critical issue: some chat users misunderstood the checkbox behavior, resulting in incorrect responses or prematurely abandoning the process. This validated our hypothesis and underscored the necessity for improvements. Acting on these insights, the product team redesigned the solution to enhance user experience and provide greater clarity.