Overview
Pocket Casts is an app designed by podcast fans for podcast fans. Its free and Pocket Cast Plus (upgrade paid) tiers aim to provide the best and easiest podcast listening experience.
This mobile usability study aimed to find if there is alignment to that claim by identifying areas of ease, interest, and excitement while also unveiling areas of frustration, confusion, and dissatisfaction.
Goals: To observe and assess Pocket Casts’ first-run experience (FRE) and the usability of its key features and functions, including:
- Importing podcasts from other podcast applications/URLs.
- Searching and subscribing to a podcast.
- Archiving and unarchiving episodes.
- Utilizing filters.
Role: UX Researcher, Test Facilitator
Methodology
Recruitment
12 participants were recruited for a moderated, guerrilla mobile usability study to evaluate the Pocket Casts app’s first-run journey and a few supporting tasks.
A team of four testing facilitators recruited three participants each from social, educational, and professional affiliations. The screened participants were advanced or frequent podcast users. Shannon administered three in-person tests with a more traditional guerrilla testing setup, which included an overhead camera (to record gestures and movements) and mobile screen-capturing using the software AirDroid Cast for in-app interaction details.
Testing Details
In-person mobile sessions were conducted with a microphone, overhead camera (and rig), and screen casting/capturing software AirDroid Cast via laptop computer and connection cable. Technical issues encountered include licensing issues with casting software, camera storage limitations, and spotty WIFI connections.
Before beginning their session, participants received an informed consent statement and agreed to have their screens and hand gestures recorded. Participants were informed and allowed to withdraw at any time and were prompted to think out loud and verbalize their thought process.
At the start of each session, participants were asked to download and install the Pocket Casts app on their mobile phones, create an account, and do a series of crucial tasks within the application. Later, the participants provided feedback about their experience and if they felt anything was lacking from the application.
Participants
Participants qualified for this study if they listened to podcasts regularly (more than a few times a month) but did not use the Pocket Casts app. During the testing sessions, participants warmed up by answering questions about listening habits and frustrations with their current podcast app(s).
| Test Admin: Shannon Kelly Participant 1 Name: Renee Duration: 52:34 Podcast usage: Driving, work, or exercise. Frustrations with current app: Spotify doesn’t have enough show notes and no bookmarks. Device Type: Android | Test Admin: Shannon Kelly Participant 2 Name: Jason Duration: 25:53 Podcast usage: Work or during chores. Frustrations with current app: Spotify always shows the same uninteresting podcasts. Device Type: Android | Test Admin: Shannon Kelly Participant 3 Name: Gabriela Duration: 34:39 Podcast usage: At home, during chores & driving. Frustrations with current app: Google podcasts often don’t sync between devices. Device Type: Android | Test Admin: Ashley Jelicks Participant 4 Name: Sigal Duration: 08:10 Podcast usage: Commuting to campus. Frustrations with current app: When the video version isn’t available on Spotify. Device Type: iPhone |
| Test Admin: Ashley Jelicks Participant 5 Name: Alice Duration: 07:58 Podcast usage: N/A Frustrations with current app: N/A Device Type: iPhone | Test Admin: Ashley Jelicks Participant 6 Name: Kasia Duration: 06:26 Podcast usage: N/A Frustrations with current app: N/A Device Type: iPhone | Test Admin: Katie Tobin Participant 7 Name: Hannah Duration: 16:19 Podcast usage: Driving. Frustrations with current app: Hard to find podcasts, especially the most recent, unplayed episodes. Device Type: iPhone | Test Admin: Katie Tobin Participant 8 Name: Richard Duration: 13:50 Podcast usage: Work & commute. Frustrations with current app: Difficult to find podcast section in Spotify. Device Type: iPhone |
| Test Admin: Katie Tobin Participant 9 Name: Meg Duration: 20:14 Podcast usage: Driving. Frustrations with current app: Some podcasts are unavailable on Spotify, and saving for later doesn’t always work. Device Type: iPhone | Test Admin: Morgan Ream Participant 10 Name: Taylor Duration: 11:39 Podcast usage: Work & commute. Frustrations with the current app: Car Bluetooth skips out when playing both Apple and Spotify. Device Type: iPhone | Test Admin: Morgan Ream Participant 11 Name: Mason Duration: 13:39 Podcast usage: Daily commute. Frustrations with current app: (Spotify) The latest podcast episode loads vs. where they left off listening. Device Type: iPhone | Test Admin: Morgan Ream Participant 12 Name: Amber Duration: 17:29 Podcast usage: 5 days/wk. Home. Frustrations with current app: Spotify lacks variety and is confusing. Device Type: iPhone |
The following outlines the 6 moderated tasks asked of participants during testing, as well as their honest responses and recommendations for improving the Pocket Casts mobile app. Skip tasks and see additional opportunities.
Task 1
Download & Create an account (FRE)
Scenario: Today, we are looking at the Pocket Casts app. Please find and install the app as you would normally. Once installed, open the app and follow the prompts to create an account. Stop once you reach the welcome screen.
Findings
Task Completion Success Rate: 12/12
Downloads Pocket Casts and signs up for an account (any method.)
Mean Time on Task: 1 minute, 27 seconds.
- Pocket Casts is easily found by name in both Apple and Google Play app stores.
- All account creation options (Google, Apple, or Standard) were quick and appreciated.
- Participants were hit with a different number of screens before entering the app.
- Most participants skipped or were not interested in upgrading before exploring the app.
“Pretty easy & typical sign-up process, especially if you’re logging in with an existing account.”
“Ok, it’s asking me to unlock features?”
Recommendations
- Reduce the number of steps to get into the app (during FRE) to 3-4 and keep the experience consistent across devices.
- Example: 1. Splash > 2. Login > 3. Enter app
- Eliminate upgrade offers until after the user has used the app more than once.
Task 2
Import Podcasts from other apps or URL
Scenario: You want to make sure your favorite podcasts are available in the Pocket Casts app. Use an import option to add a podcast from another platform or URL.
Findings:
Task Completion Success Rate: 1/12
Import a podcast from a separate application or OMPL URL file.
Mean Time on Task: 2 minutes, 54 seconds.
- Import from app options lacked variety and required too many steps.
- Only one user could import a podcast from Apple Podcasts, but to do so, they had to install the Apple Shortcuts app first.
- The import instructions were hard to follow because they included unfamiliar terms, even for frequent podcast enthusiasts.
“I have no idea what an OPML file is. Unless this is a developer thing, I don’t think many people are going to know.”
“Right off the bat, I would be like, ‘too much work!’…I’d rather search for a podcast.”
Recommendations
- Improve process with quick-import logins from other podcast accounts.
- Show the step-by-step process of finding OMPL URLs on specific platforms.
- If quick-import isn’t executional, eliminate the import feature upon FRE and limit the function to live in account settings only.
Task 3a
Search & Subscribe to a podcast
Scenario: Your friend highly recommended a podcast called “The Professional Book Nerds”, and you want to follow this podcast for future episodes. Show me how you would do this.
Findings
Task Completion Success Rate: 8/12
Successfully finds a podcast and subscribes.
Mean Time on Task: 1 minute, 15 seconds.
- At least 6 of 12 participants misspelled a phrase while searching, resulting in incorrect or inconclusive results.
- Often, participants weren’t sure if they subscribed to a podcast.
- Participants couldn’t sort or filter results in the Discover tab or search results.
“If I have a typo, nothing shows up.”
“I don’t know where to go to confirm that I followed it.”
”Seems pretty easy for me to add them. I’m assuming that’s what the checkmark means?”
Recommendations
- Upgrade the search function to recognize misspellings and suggest alternative search queries (like Google.)
- While browsing, make it more obvious when a user has subscribed with a bolder check icon and a dismissible/swipe-able confirmation message.
- Add filtering and sorting options on the Discover tab & Search results – by popularity, newest, genre, etc.
Task 3b
Archive an episode
Scenario: Imagine you’ve finished listening to an episode of the professional book nerds podcast but don’t wish to listen to the episode again. What would you do in the app to make sure you don’t hear the same episode again in the future?
Findings
Task Completion Success Rate: 10/12
Archives episode or choose “Marked Played”.
Mean Time on Task: 1 minute, 7 seconds.
- Some participants had difficulty finding an option to archive or mark played (at first) because the button wasn’t clearly visible.
- Most participants didn’t understand the difference between “Archive” and “Mark Played.”
- Only 3 participants discovered swipe or press-and-hold gestures to reveal archive options.
“Oh, [Mark played] archives it?”
“It seems pretty user-friendly. I had no prior knowledge of this app. I just held it down and clicked the three dots, which usually mean options, and then archived.”
Recommendations
- Upgrade the search function to recognize misspellings and suggest alternative search queries (like Google.)
- While browsing, make it more obvious when a user has subscribed with a bolder check icon and a dismissible/swipe-able confirmation message.
- Add filtering and sorting options on the Discover tab & Search results – by popularity, newest, genre, etc.
Task 3c
Unarchive an episode
Scenario: Oh no! You’ve mistakenly archived a professional book nerds episode that you haven’t listened to yet. How would you undo that mistake?
Findings
Task Completion Success Rate: 11/12
Unarchives an episode or finds a way to do so.
Mean Time on Task: 46 seconds.
- Almost all participants were able to find an unarchive option rather quickly.
- Those who weren’t able to find an option quickly didn’t see the “Show Archived” option above listed episodes.
- 1 participant who sought the help resources mentioned the irony of swiping on an episode when it’s disappeared and how “Show Archived” wasn’t listed in the help article.
”But [archive] made it disappear…
“Oh, 1 archived. Show archived…that wasn’t clear….but that wasn’t in the help document.”
“I’m still not clear on what the archive button does. What does that feature do exactly?”
Recommendations
- Make the “Show Archived” section stand out more, primarily when episodes are archived.
- Add this method of unarchiving in the how-to help article.
Task 4
Create a new filter
Scenario: Imagine you have a 30-minute drive to work and would like to listen to podcasts while you commute. You’d like to listen to episodes roughly 30 minutes long, so you’re not fumbling and switching episodes while you drive. Find a way to refine your podcasts to meet the 30-minute episode criteria.
Findings
Task Completion Success Rate: 8/12
Finds filters and chooses duration.
Mean Time on Task: 1 minute, 18 seconds.
- Some participants thought these filter options weren’t selectable.
- Although most thought filters were useful, some participants found they couldn’t click and listen to a filtered episode without saving the filter.
- Some participants expressed wanting to filter all content, not just subscribed podcasts.
“I wish that these were not grey so they were a little easier to see…when I clicked it before, I thought that these were not clickable options”
“So it’s going to have me set up a permanent filter…Not sure if I like that; maybe I just want to filter quickly?”
Recommendations
- Again, make buttons and clickable options higher contrast/stand out.
- Don’t require filters to be saved in order to browse filtered content.
- Change the filter call to action button to “Save This Filter” so users know what the next step is.
- Enable filtering for all podcasts; don’t limit filtering to those subscribed to.
Follow Up Comments
Participants were asked: Please share your thoughts on your overall experience with the Pocket Casts app. What features do you feel are lacking from this application?
“I was confused when it came to importing.”
“I’d like alternative (more) filter options.”
“…if I start listening, will it recommend others?”
”A lot of options. I would want to advertise this to advanced users… who want to customize their experience. I wouldn’t put this in front of a brand new podcast listener”
“First seeing it, it was kind of overwhelming seeing all the different types of podcasts…but actually kind of like the app… like the filters…the seconds listened and saved.”
Additional Opportunities
Brand Identity & Alignment
There is some brand confusion. Establish brand guidelines and be consistent across operating systems and light/dark modes.
- The upgrade screen is yellow and black.
- The logo is red.
- The internal app has blue, yellow, and green buttons.
“It doesn’t really offer a unique look or feel…this looks a little too plain-jane”
“This looks like it’s copying Spotify with the yellow to black fade”
On-boarding Tutorial for First-Run Experiences
While reducing the steps to get into the app, first-time users should be given a dismissible onboarding tutorial highlighting some of the key features that make the Pocket Casts app unique.
Additional Ideas
- Two participants expressed a way to hide or mark uninterested podcasts.
- Allow users to bookmark while listening and share an excerpt of an episode.
[To problematic content creators:] “Don’t talk to me or my son again.”
“I don’t want this podcast to keep following me and showing up.”
“A way to share a specific time-marker within an episode.”
Conclusion
Although some participants encountered interface issues and noted missing functionalities, most had a positive outlook on the app upon completing the study.
To improve user experience, it is recommended that features and buttons be streamlined and enhanced to enhance contrast. Furthermore, it is crucial to adhere to a clear and unique brand guideline that does not imitate competitors.
Test the proposed changes with a similar audience (as those to this study) and additionally with hard-of-seeing and screen magnification users. This will help identify contrast issues and ensure compliance with color ratios.
This application is designed to offer a personalized and convenient listening experience for all podcast enthusiasts. However, some experienced listeners have found it overwhelming at times. To promote this app effectively, it’s recommended to target frequent podcast users who seek a more tailored listening experience. Additionally, improving the onboarding process can help novice listeners adapt more easily.
Note: Pocket Casts did not commission this usability test.
Got a similar project?
Contact me to see how I can help!