Qualitative research for the rideCAT app, a mobile ticketing app for the Collier County public transportation system.
Overview
An independent qualitative research study to gain insight into the user experience of the rideCAT mobile ticketing app.
The rideCAT mobile ticketing app enables riders to purchase tickets using a credit or debit card, or via Apple Pay. Once purchased, passengers can activate their tickets on their phones or mobile devices before boarding and scan the barcode on the onboard validation unit. The goal of the app is to make riding transit faster and safer by removing the need for passengers to handle cash, physical tickets, or interact with ticketing infrastructure.
This cutting-edge technology will greatly improve our riders’ travel experience by providing seamless access to public transit and ticket purchasing anytime, anywhere”
— Michelle Arnold, director of public transit and neighborhood enhancement
Collier Area Transit (CAT)
Based in Naples, Florida, the mission of CAT is to provide safe, accessible, reliable, convenient, and courteous mobility services to customers.
OBJECTIVES
-
Gauge user satisfaction and uncover pain points of app users.
-
Discover barriers for riders not currently using the app.
-
Assess how the app is performing in relation to stakeholder goals.
METHODOLOGIES
-
Field Observations
-
In-Person Field Interviews
-
Paper Surveys
TEAM
I was the sole researcher.
TIMELINE
4 WEEKS
Problem Statement
The rideCAT mobile fare app was released in August 2020 as part of an initiative to streamline payments and reduce physical contact between drivers and passengers during the Covid-19 pandemic. CAT needed to find out how the app was performing with riders and stakeholders up to this point.
-
Gauge user satisfaction and uncover pain points of app users.
Getting current app user satisfaction was important to get a baseline and to identify pain points to work on.
-
Discover barriers for riders not currently using the app.
Uncovering obstacles for non-app users would help CAT gain more app users.
-
Assess how the app is performing in relation to stakeholders goals.
Measuring whether the app is performing in line with what the CAT administrators goals are and what the experience is for on-the-ground staff (customer service and bus drivers) was important to the overall mission of CAT to use technology to improve transportation for Collier County.
rideCAT Audience
RideCAT users are residents of Naples and the surrounding area who don’t have or can’t use independent transportation of their own.
CAT services Southwest Florida including connections to Marco Island, Sanibel Island, Immokalee, and connects to Lee County transit which services Cape Coral-Fort Myers. The rideCAT app was released to work with the main bus lines in Naples, though CAT also runs a seasonal beach bus and a para-transit bus that services door-to-door by request.
Bus riders use the services for commuting to work or school, to go shopping or for doctors appointments. They range in age, demographic and socio-economic status.
Bus drivers are also part of the rideCAT audience. Of all the stakeholders, bus drivers interact with the rideCAT app the most. It directly affects their job and route performance. Investigating how drivers experience the app was an important component of the research.
Roles & Responsibilities
My role was to work with stakeholders in the Collier Area Transit Department to determine goals of the study, run the study and analyze the findings.
-
Work with stakeholders to determine the best locations, routes and times to intercept riders for the study
-
Interview riders and administer surveys
-
Synthesize findings in relation to objectives and actionable steps forward
-
Present findings to stakeholders
Scope & Constraints
Interviews and surveys were done over nine days (between March 29 - April 29, 2021) at a major transfer hub and onboard buses. I was the solo researcher and all work was done pro bono with no budget, but some printing and translation resources were provided by CAT.
Some other challenges of the study were:
Covid-19: PPE was required onboard as well as adherence to distancing and sanitizing, but some riders were still weary of talking or touching anything. Masks made it difficult to communicate with riders at the transfer station, as the buses were idling and it was hard to hear. This caused people to move closer to try to understand each other and that was not ideal for social distancing.
Language barrier: The population included Spanish and Creole speakers. CAT provided a translator for two intercept sessions and I was able to make a Spanish language version of the paper survey.
Sample size: The study was designed to be a snapshot in time, and with only one researcher
1. Review Ten Year Development Plan
A development plan was released in October 2020 detailing improvements in all areas of the transportation system over the next 10 years. I reviewed this for insights into CAT’s goals and ideas for what to focus on in the research.
CAT has technology goals outlined in section 7.2.7 Technology Trends that aligned with my research and I was able to validate some of their goals.
2. Gather Stakeholder Insights
I interviewed CAT marketing, operations, planners and the director of operations to get an idea of where they were focusing efforts for the app and what they had discovered so far about the app usage.
CAT provided their route data as well as Masabi’s app data, which included stops and number of tickets used at stops over a 3 month period. This steered the research locations and times to reach the most riders, and specifically riders who used the app.
3. Analyze Existing Rider Feedback
Check social media feedback, as well as customer service and in-app survey feedback for insights into pain points and concerns.
CAT has a Facebook and Instagram account that is not very active, however there were some good questions that were raised when the app was first released that were important to keep in mind throughout the test. The in-app survey feedback also had some thoughtful comments for the study.
4. Write Field Interview Script and Design Paper Surveys
Questions were chosen to get an idea of barriers for non-app users and pain points and satisfaction for app users. The questions were the same for in-person interviews and paper surveys.
A filtering question was asked first, “Do you use the phone app to buy your bus ticket?”
-
Based on answers, non-app users were asked two questions:
-
Does the phone app sound like something you would like to use? Why or why not?
-
Is there anything that would prevent you from using the phone app?
-
-
For app users, there were three questions:
-
How disappointed would you be if you could no longer use the phone app?
(Not at all Disappointed, Somewhat Disappointed, Very Disappointed)-
Please explain why you answered that way.
-
-
If you had a magic wand, what would you change about the phone app?
-
How satisfied are you with the phone app?
(Unsatisfied, Kind of Unsatisfied, Neutral, Kind of Satisfied, Satisfied)-
Please explain why you answered that way.
-
-
Paper Surveys in English (left) and Spanish (right)
5. Observe riders, Conduct Field Interviews, Administer Paper Surveys
Observations were performed of riders purchasing tickets and using different fare payment methods onboard. Paper surveys were handed out onboard and collected soon after. Interviews were performed at the Government transfer station and onboard various routes.
-
Observations were made of riders purchasing tickets from the customer service desk and onboard buses. As well as using existing ticketing methods such as cash, pass cards and the rideCAT mobile app.
-
Paper surveys in English and Spanish were distributed on busy routes to get as much data as possible.
-
Field interviews were performed at the Government Transfer Station and on various routes.
26
App Users
(20 in-person interviews,
6 paper surveys)
56
Non-App Users
(18 in-person interviews,
38 paper surveys)
16
Bus Drivers
6. Synthesize data, compile a report and presentation
A report was presented to stakeholders of findings from the study and recommendations on how to address issues.
-
Gauge user satisfaction and uncover pain points of app users.
Users were generally satisfied with the rideCAT app. They said that it was easy and convenient to use, however there were pain points. App users pain points can be broken down into 2 categories, User Experience and Technical. There is some overlap between the categories when it comes to getting real-time service alerts and accurate route times.They described this as a priority and wish that this information and the fare app was combined into a single app.
The money doesn’t go in the farebox all of the time, and [I] hate to hold up the line of people waiting to board the bus.
More convenient than a card…always have my phone, I won’t forget it.”
Other user experience issues were not critical to the function of the app, such as a request for fewer screens to click through to get to the ticket, or having a longer window of time to purchase a single fare ticket. CAT will have to coordinate with Masabi to implement feature upgrades.
Technical problems were the bulk of pain points for app users. Riders mentioned the app freezing when they are trying to use it, or the validator not scanning. One rider mentioned having to restart the app after it glitched when he moved between apps on his phone. And as I mentioned before, service alerts were not updating, which caused a lot of frustration for app users.
-
Discover barriers for riders not currently using the app.
For Non-App users some of the obstacles were a lack of awareness of the app and assumptions about the specifics of the app. Some people I talked to didn’t know if they could use special fare passes on the app, like reduced fare tickets or if they could buy single fares.
Circumstantial barriers like not having a bank account or credit or debit cards can keep people from using the app. Insufficient devices such as using a non-smartphone, or currently using a smartphone that doesn’t support new apps, or a phone that doesn’t have enough memory or battery life present obstacles to app usage.
Behaviors and beliefs include riders who don’t use a cellphone at all, or who don’t use the bus often, and riders who are distrustful of digital devices for fear of scams, or they just prefer physical cards.
I prefer a physical card because I’m afraid of online scams
Nothing keeping me from using it, except would want to be able to put a few dollars in and not a whole $20.
-
Assess how the app is performing in relation to stakeholders goals.
All 16 drivers I spoke with liked the app. They said riders rarely had any issues and never asked for help. Compared to the farebox, where the driver has to intervene often when cash gets jammed and paper tickets won’t swipe. All of the operators were frustrated with these farebox malfunctions and some expressed that it put them in unsafe positions, one example being holding cash on their route. They also mentioned that it took up time from their route to troubleshoot those issues.
Drivers said that they didn’t know much about the app and when some non-app riders asked them about it, all they could do was direct them to the app store. They had no information to share with riders.
When the farebox gets jammed, it’s not safe...especially in Immokalee, there is no supervisor out there to fix it, so you end up holding cash the whole time—it’s not safe!
Most people know how to use it. If people ask about the app, all I can do is tell them to download from the App store.
Recommendations
I recommended focusing on 3 areas to address immediately:
Awareness and Adoption
There are some possibilities to supplement current marketing initiatives, such as creating larger signs and in multiple languages, providing drivers with step-by-step handouts to give to non-app users and emphasizing the security of the app. I also recommend in-person workshops to bridge the gap for riders who are less tech savvy.
Driver Safety and Advocacy
Drivers can become front-line advocates for the app if there are readily available resources to reference or hand out to riders. Job satisfaction and performance could increase with more app usage by alleviating some job stresses, such as dealing with faulty machines, handling cash, and physically interacting with riders. This could have many other benefits down the line as well.
Glitches and User Experience
The study further validates the need outlined in the Ten Year Development Plan for updates to hardware and examination of the systems currently in place. Fixing the cause of those glitches will address the user experience of the app. It would be ideal if CAT was able to work with Masabi to customize the app, however if it is not possible, then passing along rider feedback and feature requests is recommended.
I also presented some ideas on what can be measured going forward to keep tabs on CAT’s objectives.
Route Efficiency: Comparing pre- and post-app data on routes with high adoption of the app over time
Adoption Rate of the App: Setting a goal for a six month check in to assess progress
Driver Satisfaction: Could be measured to see if app adoption has any effect
Rider Satisfaction: This is a standard measurement, but requires an explanation to get actionable data
Rider Disappointment: Measuring loss is a good indicator of how essential a service is to someone’s life. Moving riders from “I don’t use the app” to “being very disappointed if they could no longer use it” will demonstrate more clearly how valuable it is to them.
Outcomes & Lessons
I learned how to streamline my interview and survey questions. For future projects, I will want to do more preliminary research to understand the population and to consider the constraints of field work.
RESEARCH METHODOLOGY
-
Edit questions
I originally had more demographic and trip questions for riders—where are you going today? What type of fare do you normally buy? How did you find out about the app? After the first hour of interviews, I realized that I had to be more direct and cut out these questions because the buses were loud, it was hard to communicate with masks on, and people were rushing to catch their bus.
I had to cut driver questions down as well. I went for a basic question that I elaborated on as I got answers. “What is your experience with riders using the rideCAT mobile fare app?” followup, “Do they ever ask for your help with the app?”, “Are there any technical problems with the app?”
Scrapped driver survey
-
Adjust timing
Changed the original time schedule based off of early tests, where the riders were tired or anxious to catch their bus for work. I thought adding more afternoon sessions would be better to catch crowds without those challenges. I had mixed results with this, but I think overall it was good to get that varying time spread of riders.
-
Understanding the population to get the best data
-
I had to rework the language on the survey and interviews to be extremely clear. Many people didn’t know the name of the app, so I had to refer to it very specifically as the 'phone app to buy your bus ticket." This was also to avoid confusion with CAT's other apps, planCAT for trip planning and maps, and MyStop for real-time updates.
-
Future research needs to include Spanish and Creole speakers for qualitative research. People were more likely to speak to someone in their native tongue face-to-face than to speak to me in English, their second language. I also think that the structure of the survey might have missed cultural or experiential cues. There were several participants who answered both the “NO” and “YES” questions, which made me wonder if I designed it in a confusing way or if they didn't have experience with that type of survey.
INTERVIEW TECHNIQUE AND AWARENESS
-
The language barrier and lack of knowledge of the app meant I ended up doing more marketing, handing out flyers and explaining how the app worked. This was helpful to see how much non-app users understood the purpose of the app and what their barriers might be.
-
I developed a rhythm of allowing passengers to get settled on the bus before approaching them about the study. Then trying my best to explain the app and the reason for the study, while the bus moved and while talking through a mask, a couple feet away from riders.