top of page
IMG_1436 2.heic

Wikipedia App Usability Study

This was a remote, moderated usability study for the Wikipedia app designed for feature phones in India, conducted in the middle of the pandemic.
The most challenging part of the study was creating a research setup that would enable us to look at users' phones that didn't have screen-sharing capabilities.The study was conducted in 3 different regional languages, across 3 states.

Background
 

In recent years, the innovation of smart feature phones has brought the Internet to millions of low-income Indian users who would not otherwise have been able to go online. These phones have voice calling, messaging and basic internet functionalities. Jiophones are the most popular type of feature phones in India.

 

In 2019, the Wikimedia foundation noticed that a lot of Indian users were accessing Wikipedia through browsers on feature phones. The organization was curious to understand who these users were, what were their motivations and challenges in accessing Wikipedia and how a better experience could be designed for them. 

With this question, the Wikimedia organization approached the agency I worked for. In an attempt to find answers, a foundational research study was conducted and personas were delivered.

Using this research as a framework, in 2020, Wikimedia built an exclusive Wikipedia app for feature phones that was available in multiple regional languages and approached us again to conduct usability testing. This is a story of what happened next.

The Wikipedia app landing page on a Jiophone

Determining Study Goals

After talking to the stakeholders, we came up with 3 main goals:

 1. To evaluate the users’ understanding and usability of different features.

 2. To test users’ understanding of key workflows: 

      a. reading/navigating within the article

      b. searching for content 

      c. changing the article language

 3. To understand users’ motivations and challenges in using the app.

Determining Research Sample

Since we wanted to evaluate the experiences of regional language readers, I initially decided to recruit 9 users -  3 each for each of the regional languages we had picked - Hindi, Marathi and Telugu. These languages were picked to cover Northern, Western and Southern parts of India respectively, also matching our teams’ linguistic skills.

However, I was doubtful of the quality of remote sessions and worried about the no-show rates, so I ultimately recruited 20 users in total.

Determining Research Methods

Challenges: Due to the ongoing pandemic, we decided to conduct a remote, moderated UT. But, Jiophone doesn’t have the capability of screen sharing or video calls, which meant we couldn’t test over Zoom on Jiophones directly.

 

How I still managed to conduct a remote UT: To overcome this, I thought of a simple set-up that the users could quickly put together to show us what they were doing on their screens - this setup consisted of any household item like a jar or a container being used as a support to hold a smartphone that would run a Zoom call and capture the activities on the JioPhone placed right below it, while the user thought out loud.

We requested users to borrow smartphones from their family members / friends / acquaintances if possible. When this wasn’t a possibility for some users, we shipped a smartphone to their homes. Participants were also compensated for data charges.

I made short tutorial videos on how to install and operate Zoom and how to put together the setup for the study and sent it to participants through WhatsApp, a day prior to the session.

Snippets from the tutorial videos illustrating  (L)  how to assemble the setup (R) how to install and operate zoom

Recruiting Users

First, I decided to recruit users by posting an ad with a screening survey in Facebook groups that were exclusively created by and for Jiophone users. But we didn’t receive many useful responses (most were from minors) and decided to recruit users through a recruiting agency. 

Challenge: It was difficult to filter out genuine Wikipedia users from the pool as people often wanted to participate for incentives. We conducted a screening round asking potential participants about their Wikipedia browsing habits and reading history.

Conducting the Study

Participants were given one short task at a time, read out loud by the moderator.

They were asked to think out loud as they performed the tasks.

I invited members from the Wikipedia team to be a part of our remote sessions, so they could witness how the users were interacting with the app. However, this was not a possibility for Marathi or Telugu sessions due to the language barrier.

Challenges:

1. The biggest challenge here was not being able to see the users’ faces and their expressions.

 

2. Participants tried to be agreeable, often not voicing their frustrations while they struggled in a task. It was hard to convince them that we really want to hear how they feel about the app.

The study setup perfectly put together by the participants 

Key Deliverables

1. Stoplight Charts: To show whether a participant succeeded in a task, was unclear in the task, or failed the task with clear definition of each category. 

 

2. Challenges as per severity: We ranked users’ overall challenges in 3 categories based on severity: Highest priority - most critical to fix, medium priority, and lowest priority. 

3. Highlight reels: I created short, subtitled videos of 2-4 minutes to show the Wikipedia team  how exactly the users interacted with the app and how they felt about it. These were particularly beneficial in creating a lasting impact on stakeholders, which was a challenge as we were an external agency conducting research.

4. Detailed usability report explaining all the findings in depth.

Impact

I believe that research impact is made at three levels, as described by Tao Dong here. 

In this study, the research impact was realized at the first two levels as follows:

 

1. Research Questions Answered: The three distinct research questions about testing the users’ understanding and usability of different features, testing the usability of key workflows and understanding the user’s challenges and motivations in using the app were answered through the key deliverables and the findings were accepted by the stakeholders. 

 

2. Decisions informed by the research: The usability issues discovered in this research led to changes in the UI of the app, better onboarding instructions, changes in the navigation of language settings of the app and in the enhancement of the search feature.

That was a lot of reading. Here's a donut for making it till the end! 

bottom of page