Hey, CarMax!
UX Researcher
During the spring and summer of my HCI Master’s program, my team (Hi, Mileage!) worked with the Conversational Experience team at CarMax to explore where a conversational agent could be implemented in CarMax’s current services and on-lot experiences. After conducting several rounds of generative research, we ended up developing a research tool and prototype to support a conversational agent for test drives and on-lot discovery.
Client
CarMax (via Carnegie Mellon University)
Team
Aziz Ghadiali • Carolyn Zhong • Christie Sohn • Will Kuhlke
Duration
Spring & Summer 2021
Project Background & Goals
From the start of the project, our team was challenged to determine the best scenarios and contexts for a conversational agent to be embedded in the car buying process at CarMax. To do so, we first ventured to understand the CarMax process, identify the shoppers’ needs, discover their pain points, and identify an area for innovation.
After finishing our initial months of generative research, our team narrowed down to the space of the test drive. From there, we started developing a research tool that would allow for a wizard of oz testing of a conversational agent on a test drive and a functional prototype.
After finishing our initial months of generative research, our team narrowed down to the space of the test drive. From there, we started developing a research tool that would allow for a wizard of oz testing of a conversational agent on a test drive and a functional prototype.
Generative Research Goals
- Learn about the CarMax car buying process from both the shopper and business perspective.
- Identify area in car buying process where a conversational agent can be embedded to create value for CarMax and their customers.
- Understand the current state of conversational technologies and future-looking capabilities.
Design Research Goals
- Identify best modality for shoppers to interact with a conversational agent while conducting their test drives.
- Minimize risk of interacting with a conversational agent while conducting a test drive.
- Find best balance of proactive and reactive engagement with shoppers while interacting with our conversational agent.
Research Methods
Throughout the course of this 8-month project, our team conducted several qualitative and quantitative studies to inform the design of our conversational agent. Early on in the project, we conducted generative research to first narrow down on a space in the car buying journey to focus our design and then, as we moved into the summer, focused on a conversation agent to assist the test drive.
In-Depth Interviews
- Early on in the project, our team conducted interviews with experts in conversation design, artificial intelligence, and interaction design to learn more about the interactivity of conversational agents.
- We interviewed ten Sales Consultants from CarMax in the generative stage of our research to learn more about their pain points, as well as customers. As we began testing on early iterations of our prototype, we interviewed more Sales Consultants to further understand their thoughts and feelings on how our conversational agent would be able to facilitate the test drive.
- Throughout all phases of research on this project, we interviewed CarMax customers to learn about their experience with existing services and the (on-going) iterations of our conversation agent prototype.
- As Lead UX Researcher, I created interview guides for our team to ensure that all interviews would cover the topics relevant to the research questions we were aiming to capture within each round of interviews.
Experiential Prototyping & Usability Testing
- After wrapping up our generative research, we focused on the test drive. As we iterated on our prototype, we conducted several rounds of usability testing.
- For in-person prototype testing, we created a Wizard of Oz setup that consisted of a visual UI in Figma and voice interface with ReactJS to simulate the experience of a conversational agent assisted test drive. Testing was conducted throughout all iterations of our prototype with non-CarMax customers and real (in-store) CarMax shoppers.
- For remote prototype testing, I designed an OBS overlay that allowed us to conduct virtual usability tests over Zoom. The OBS set-up used driving footage from the participants' city and the visual/voice UI used for in-person testing.
Wizard of Oz setup for in-person testing.
OBS setup for remote testing.
Directed Storytelling
- Our team used directed storytelling to better understand the various pain points in the car buying process at CarMax.
- For current CarMax customers and (general) used car buyers, we had participants walk us through their process of researching cars and utilizing CarMax services.
- For CarMax Sales Consultants, we had them walk us through the process of interacting with customers in-store.
Survey Research
- I was responsible to planning and designing our teams' surveys on Qualitrics throughout the project.
- After identifying opportunity spaces for our conversational agent, we created storyboards for each major concept to test with customers. We wanted to get as many customer impressions as possible, so we decided to pair dissimilar storyboards and show each customer one pair to gather their impressions (by only showing two storyboards, we were able to keep the survey to about 5 minutes)
- In order to achieve this, I designed the survey to randomize which pair of storyboards each customer would see and, within each of those pairs, randomized the order in which the individual storyboards appeared. I also used a validated scale to measure their impressions as accurate as possible given the constraint of using closed-ended responses being for, traditionally, open-ended questions.
Key Design Opportunities
Before we ultimately narrowed down to the space of the test drive, we used the insights gathered throughout our generative and evaluative research to identify the three key opportunity spaces for our conversational agent.
Conversational agent to increase the shoppers’ access to accurate and relevant information.
- Our insights told us that customers use the on-lot experience at CarMax to validate the research that they have already done.
- We found that shoppers have varying opinions on the presence of a sales consultant, so a conversational agent could act as another resource for customers to receive accurate information.
- A conversational agent has the ability to provide assurance and accuracy throughout the entire car buying journey that goes beyond the on-lot experience.
Conversational agent to assist sales consultants on the lot.
- The number of shoppers that visit CarMax and the number of cars on the lot make it difficult for sales consultants to provide the accurate information each shopper needs.
- A conversational agent that exists on the lot, separated from the sales consultants, can provide the information that shoppers need for wherever they are in their process.
- With the variety of cars available on the lot, it is impossible for sales consultants to know the details of each car. The conversational agent could help relieve this pressure of knowing everything about every car on the lot.
Conversational agent to maximize the value of test drives.
- The test drive is an opportunity for shoppers to physically experience the car and find assurance by validating what they know without the outside influence of sales consultants.
- The test drive plays a critical role in helping shoppers transition from understanding their needs to understanding their wants in a vehicle, which helps them make their final selection.
- The environment provided by the test drive allows for a shopper to ask questions and receive answers without the presence of a sales consultant. Situating a conversational agent within the test drive maximizes the "alone time" shoppers have with the car.
What I learned
- Having the opportunity to design and conduct various research studies taught me a great deal about how research is directly connected to design and, from there, how research can be conducted most effectively to help tackle complex design challenges.
- In having to learn about conversational agents and conversation design when starting this project, I picked up quickly on how to better digest information about new areas of design and how my existing knowledge could be utilized to enhance my understanding/perspective of the new subject matter.
- This project also allowed me to get creative with common research methods and practices to adapt them to our work. I leveraged a lot of what I have learned in game design practices to inform new ways of testing and evaluating concepts with users.
For more details about this project, please reach out directly to zachvancleef.ux@gmail.com