A5: Behavioral Prototype

Alida Muongchan

Task

The prompt here was to learn behavioral prototyping, and in this case, the technique used was the Wizard of Oz Study. The idea of this technique tests the assumptions of HCI applications and the actual technology that may be too expensive or unavailable.

Scenerio & Requirements

There were a few different scenarios to choose from in this assignment, as well as it being our first time working in teams for an assignment. With my teammates Pao Pongsala and Lara Hattatoglu, we chose the first scenario which was testing applications for a voice-operated assistant. Here are the requirements:

Create the context for open-ended exploration of what users would expect to use as verbal commands for a simple task, such as navigating a spoken recipe, choosing music to play for a party, or a voice-based game.
Explore users' tolerance for recognition errors.
Note: You may not use an existing voice assistant, but must figure out how to deliver this experience in a believable way that allows you to make easy, real time modifications.

Design Concept

For our design concept, we wanted to focus on concepts that would be most useful for consumers and businesses as well as enhancing the quality of experiences for a design and the devices that have yet to be tested.

Usability - How well does the product work in terms of appropriately responding to the user’s commands.
Desirability - To what extent does the user want to use the product in their daily lives and recommend it to others.

Criteria for success:

If user is unable to identify that the product is a Wizard of Oz
If the user mentioned wanting to use this product and/or recommending it to others (desirability)
If the user was able to successfully completed all tasks given without help from the facilitator (usability)
If the user did not display major impatience through their tone, facial expression, etc. when testing for their tolerance.
If the user offered achievable and constructive feedback for future product iterations

Data & documentation:

Observations during the interaction:

Using the user's facial expression, tone, body language, hand gestures, and user’s interaction with the product to determine their satisfaction, level of patience, and ability to figure out how the product functions.

Task Completion:

Tracking whether the user is successfully able to complete the task will help gauge usability and how intuitive the product is to use.

Verbal Qualitative/Quantitative data:

During follow-up questions, rating scores and feedback on how the product could be improved will also give us insight into usability, desirability, and future areas of improvement.

Wizard Strategy:

Because we wanted our testing to feel seamless and believable, we repurposing an existing humidifier that one of us owned to appear like a voice-assistant device. We will had two major devices, a small iPhone to fit inside the humidifier, and this is connected to the Wizard’s laptop. With a website that automates text into voice messages, we were able to create pre-recorded responses to our tasks along with typical commands and error messages. The wizard will be on a phone call with the phone that is in the humidifier to hear the participant’s commands.

User Testing Plan:

Facilitator: Alida

Wizard: Lara

Note Taking: Pao

Cameraman: External person/or set up cameras around the room

Physical setup: The wizard will be on a live call, hidden, in another room where the user cannot see. The wizard will have their phone for the call and a laptop to play music and pre-recorded responses. The notetaker will sit on the other side of the table to write down their observations.
Context: How would the user play music from a smart device?
Test Plan: Below is a flow chart of a planned expectancy of responses:

Task 1: How would you go about asking the voice assistant to play the song?

Wizard: “Who is the artist of the song, Happy?”
Wizard: Plays the song Happy from Pharrell Williams’ album “GIRL”

Task 2: What would you do if you wanted to make the song quieter?

Wizard: turns down volume

Task 3: How about skipping to the next song on the album?

Wizard: go to the next song

Task 4: How would you temporarily stop the music? [checking for user tolerance]

1st round — Wizard: turns music volume down
2nd round — Wizard: “Sorry, I didn’t catch that. Could you repeat it again?”
3rd round — Wizard: actually pauses the music

Testing Video:

Analysis:

Task 1: “Kuki, play Happy by Pharell Williams”

Task 2: “Kuki, turn the volume down”

Task 3: ”Kuki, skip to the next song of the album”

Task 4: Tolerance for errors

- 1st time “Kuki, pause the music”

- 2nd time “Pause the music”

- 3rd time “Kuki, pause the music”

What went well:

Voice commands for the most part were recognized on the first attempt, except for last one
Speed is very quick on performing the actions

What did not go well:

The last one didn’t pick up on what they said.
The participant wasn’t sure if they were supposed to say “kuki” again, or just say the action

What to change about product:

They wondered if there were “buttons to press to perform actions?”
Overall satisfied with the voice assistant

1-5 scale how easy to use the VA:

4 because the last command didn’t pickup, otherwise it picked up on everything else and had a quick performance

In what settings/situations they’d use product:

Taking a shower and playing music
[hearing command over the sound of water] is not a concern if they speak loud enough

Would they use the product:

Yes, easy to use, basic commands. Provides exactly what they would want. Great speaker for the bathroom.

Would recommend the product for others:

Yes, easy to use, basic functionality you would need.

After the reveal:

Mentioned that they were skeptical and confused at first on how it would work as a voice assistant since they realized it was a humidifier.

Overall Thoughts

In beginning of this assignment, I was really confused on the purpose of the behavioral prototype, or actually understanding what it was even doing. After defining the main design concepts and strategies, it was really starting to come together. Along the way we ran into some major issues. For example, instead of a speaker, we made use of an old iphone, but the sound was muffled from the user's perspective. There was also a delay time in being able to switch scripts to music for the wizard since there were many moving pieces. It's almost similar to playing a game without being able to make a mistake. One of the solutions that we came up with was using iMessage to send a message to the wizard of when to play a certain script. IMessage in this case was perfect because it was simple, cheap, quick, and effective. The results for the text were immediate and on par to real time response. This was the true game changer to make our results accurate! This was a fun prototype that I look forward to using more in the future!