BASED IN USA
Evaluating Usability in the Walmart App: A Quantitative UX Study
WHAT I DID
Conducted SUS analysis with statistical validation, user interviews, and ran think-aloud sessions, synthesised findings into key insights and co-developed design recommendations.
ROLE AND TEAM
I worked as a UX Researcher alongside a team of six.
METHODS
-
User Interviews
-
Persona
-
Affinity Mapping
-
Think-Aloud Testing
-
SUS Rating
TOOLS
-
SUS Questionnaire
-
Figma
-
Excel Sheets
-
Data Analytics Platform
-
Google Form
INTRODUCTION
As part of a cross-functional team, we conducted a UX research study to evaluate the Walmart mobile app, focusing on international students aged 20–30. While Walmart is a household name in U.S. retail, we wanted to understand how well its app supports non-native users navigating everyday purchases. Our goal was to uncover usability issues across the shopping journey and propose improvements rooted in user feedback.
THE CHALLENGE
Walmart’s mobile app delivers convenience at scale—but not for everyone.
For international students navigating a new country, even simple tasks like tracking a delivery or finding help can turn into frustrating roadblocks.
What’s meant to be seamless often feels scattered, inconsistent, and unclear.
RESEARCH OBJECTIVE
We set out to understand how international students actually experience the Walmart app.
OUR GOAL
To uncover where the app breaks down—and why—during common user flows like shopping and getting support.
By combining quantitative metrics and real user stories, we aimed to reveal hidden friction and transform those insights into meaningful design improvements
TARGET AUDIENCE
We focused on international students aged 20–30, living in the U.S., who rely on Walmart for household and academic essentials. They frequently shop online but often face unique hurdles like payment limitations, unclear delivery updates, and support barriers.
METHODS AND APPROACH
To evaluate the Walmart app’s usability, we used a mixed-methods research approach that combined both quantitative metrics and qualitative insights:
1. Participant Recruitment
We recruited 12 participants, primarily international students and frequent Walmart app users, to ensure diverse usage patterns and real-world feedback.
2. Think-Aloud Usability Testing
Participants were asked to complete key shopping tasks—such as purchasing a dustbin or mirror—while verbalizing their thoughts. This method helped us uncover real-time usability issues like unclear labelling, slow load times, and unexpected changes in product availability.
3. Semi-Structured Interviews
Following the usability tests, we conducted one-on-one interviews to explore participants’ experiences in greater depth. We focused on pain points during specific flows such as checkout, tracking, and returns, and gathered insights into emotional responses and workarounds.
4. System Usability Scale (SUS) Survey
After completing the tasks, participants filled out the SUS—a 10-item standardized questionnaire that measures perceived usability. We calculated the average SUS score, standard deviation, and a 90% confidence interval to assess the app’s overall usability performance.
5. Data Analysis
We applied thematic coding to the qualitative data, grouping insights into themes such as navigation, search, customer support, and checkout. Quantitative SUS results were analysed using descriptive and inferential statistics to validate patterns observed during testing.
Interviews
To better understand user behavior and pain points with the Walmart app, we conducted semi-structured interviews with 12 international students between the ages of 20–30 who were frequent users of the app. This method allowed us to gather both quantitative feedback and qualitative insights, balancing structure with flexibility.The interviews were guided by a script but remained conversational, allowing us to probe deeper into participants’ thoughts and emotions.
We continued each session until saturation was reached and no new themes emerged. We explored a wide range of user interactions and experiences, including:
-
Use of Walmart+ memberships
-
Product search and navigation ease
-
Visibility and use of discounts or offers
-
Order placement flow and any friction points
-
Shipping and delivery experience
-
Preferred payment methods
-
Satisfaction with the returns process
Overall impressions of the app and future usage intentThe goal was to capture the end-to-end user journey and surface key pain points or moments of delight that could inform design recommendations.
Affinity Mapping
After conducting the interviews, we used affinity mapping to synthesize the qualitative data and uncover recurring patterns in user feedback. Each observation, quote, or pain point was written on individual sticky notes and grouped into themes based on similarity. MIRO LINK
Interview Findings Summary with Common Codes
-
Purpose: The application was used to place bulk orders, order items for house set-up
-
Usage: The application is mostly used weekly to place orders, to compare prices, and to check discounts or offers.
-
Products: Stationeries, groceries, and various household items were ordered using the application.
-
Payments: The card payment method was most preferred by the users.
-
Selecting Factors: The products were bought on the basis of prices, reviews, quality, and descriptions.
-
Tipping: 11 out of 12 people preferred not to tip as they are international students, and some due to bad delivery experiences.
-
Platform: Both the mobile application and website were used by the users to check prices or browse the product.
-
Customer Support: 7 out of 12 participants reached out to customer support, and getting the issue resolved was time-consuming. 3 users did not reach out to customer support as they did not have any issues, and 2 users did not reach out to customer support due to time constraints.
-
Challenges: Issues such as products low on stock, preferred products unavailable for product substitution, misplaced and wrong deliveries, auto cancellation of products, and difficulties with app installation.
-
Walmart Subscription: All users took the free Walmart+ trial subscription, and only 1 user chose to purchase the extended plan. 11 users did not extend their subscriptions due to bad delivery experiences and budget concerns.
-
Delivery: The users experience challenges like products not being delivered to the correct location, receiving damaged items with no replacement option, and poor communication with no response to feedback.
-
Return: Users faced challenges with receiving timely refunds for returned products, with delays of 4-5 days and instances where they were asked to return orders in person to the Walmart store, resulting in dissatisfaction with the refund process.
-
Experiences: Users were satisfied with the product price and quality. Most users were disappointed with delivery experiences, order substitution, returns, and getting a refund.
Think Aloud Session
To evaluate the user experience of the Walmart mobile application by observing how international students complete two real-world shopping tasks using the think-aloud protocol.
Tasks
Task 1: “You have moved into your house and it’s been a while. You want to order a dustbin and a mirror from Walmart.”
-
Average Completion Time: 3 minutes 32 seconds
-
Success Rate: 100% (12/12 participants)
Task 2: “Your order is delayed or has an issue. Find customer support and get help of any kind.”
-
Average Completion Time: 1 minute 55 seconds
-
Success Rate: 100% (12/12 participants)
Think Aloud Findings
-
Hard to Filter & Search: Filters are confusing, size units like “gallon” are unfamiliar, and search results often show unrelated or sponsored items.
-
Cluttered Product Info: Product images aren’t clear, and size or dimensions are hard to find.
-
Price Confusion: Final cost (tax, delivery, tip) isn’t shown clearly, and price filters are hard to locate.
-
Cart Frustration: No option to clear the cart at once—users must remove items one by one.
-
Support is Hard to Find: Customer support is buried, unclear, and not linked to past orders.
-
Stock Problems: Items appear available but turn out to be out of stock; labels like “Likely out of stock” are unclear.
-
No Personalization: Common or repeat purchases aren’t prioritized—users scroll a lot to find what they need.
-
Slow Help Experience: Support responses feel generic, and users have to repeat info multiple times.
SUS Rating
To evaluate the overall usability of the Walmart app beyond task success, we employed the System Usability Scale (SUS)—a reliable tool for measuring perceived ease of use and user satisfaction.
What is SUS Rating
The System Usability Scale is a standardized questionnaire consisting of 10 statements that users rate on a 5-point Likert scale. These statements alternate between positive and negative sentiments, capturing a balanced view of the user experience.
Each participant's responses are scored and converted into a single usability score ranging from 0 to 100.
Why We Used It
While all participants were able to complete the tasks (100% success rate), SUS allowed us to measure how users felt during the process—shedding light on frustrations not visible through completion rates alone.
It helped validate our qualitative findings, especially in areas like:
-
Difficulty using filters and finding product information
-
Confusing support navigation and terminology
-
Unclear pricing and cart management
Participant Data
Probability and confidence interval
We calculated the average SUS (System Usability Scale) score and the t-value. Since the t-value was negative, we used its absolute value for further steps. This gave us a probability of 0.0176, meaning there's only a small chance (1.76%) that the usability score is less than 80. So, we can say with 98.24% confidence that users perceive the technology’s usability to be higher than 80.
Next, we calculated the critical t-value and the margin of error, which came out to 4.85 points. Using this, we found the 90% confidence interval for the SUS score to be between 68.69 and 78.39.
Recommendations
Ordering Experience: Provide consistent and updated notifications about the status of the orders and returns. Provide valid reasons before/after automatic order cancellations. Provide an option to modify the delivery instructions after placing the order.Inform users beforehand through notifications if fragile products are being delivered in a plastic bag.
Customer Service : Keep the users posted with approximate wait times on customer care support calls. Provide alternate solutions to connect with customer service if the wait time is above the ideal time limit. Provide sufficient options as prompts during the initial chat with the customer service bot or give the freedom to users to write the problems.