Usability Study

Improving UX of the Audio guide
in the Thyssen-Bornemisza Museum

Audioguiarte, a company offering multimedia solutions to museums, developed a web app for delivering audio guides for the visitors of the Thyssen Bornemisza National Museum, an art museum in Madrid, Spain. Audioguiarte came to us to ask for recommendations to improve the audio guide app’s usability and design, so the app can fit the needs of users and the museum more perfectly.
MY ROLE

My roles included:
- Writing the user test moderator script
- Conducting and moderating two virtual user tests 
- Analyzing and consolidating usability problems
- Creating mock-up recommendations and writing report
- Creating and presenting final presentation deck

TEAM

Our team of four worked collaboratively to submit a written report with mockup recommendations as our final deliverable to Audioguiarte.

PROJECT DURATION

Spring 2021, 8 Weeks

What was the problem?

Our client

Audioguiarte

Audioguiarte, a company based in Spain offering customized audio and multimedia solutions to museums, developed a mobile web app for the Thyssen Bornemisza National Museum in Spain. The app contains audio guide content that can be accessed on one’s own mobile phone’s browser.

The development of the app is to guarantee "zero contact,” as conventional audio guides apparatus is prohibited due to Covid-19 and hygiene concern. To access the app, museum vistors can enter the URL or scan the QR code shown in the museum.

The Problems

There was only one month for Audioguiarte to develop the new web app. Audioguiarte wants our group of four researchers to evaluate the app in terms of usability and design, so the app can meet the institutions’ needs and is satisfactory for the potential users.

What was the process?

Step 1

Setting Study Goals

1

2

3

Meeting the client

After the meeting, we learned the context of the app: Audioguiarte wanted to improve the audio guide app's usability and design.

Defining goals: app on mobile

We aimed to evaluate users’ ability to find audio guides regarding artwork and museums.
As this app was intended to be used easily on mobile phones, so the study would be taken on mobile phones.

Defining target users

Since the audio guide was intended for anybody who would visit the museum, there was no targeted user profile.

Step 2

Design the Study

1. Recruit participants

1. Prepare screener

A screener was issued to make sure the participants spoke English and have basic mobile phone skills.

2. Select participants

A total of 26 people responded to our screening questionnaire. We selected 8 participants to make sure that we had a nice balance of participants among different genders, age ranges, and from a wide range of locations.

3. Confirm the participants

We sent the same message to the selected participants to confirm the test time and make sure they got the same instruction to prepare for the technology needed in advance of the test.

2. Design the Scenarios

Moderated Remote User Testing

We conducted the usability test via “moderated remote usability testing,"  which we gave the participants a set of tasks, then observe their behaviors virtually. Their comments and their actions on screen were recorded.

Limitations

This study was meant to be conducted inside the museum. However, due to the COVID-19 pandemic and financial constraints, the study was conducted remotely.

Thus, the test is not conducted in a a natural setting. The team acknowledges that other technical issues (such as connectivity problems) may occur in the natural environment in which this app would be used.



How we solve this

We formed a strategy that we presented a picture to the participants virtually as if they really were inside the museum, viewing the same information they would face in the museum.

3. Develop Test’s Tasks

Task 1

Asking for first impressions on the homepage


Task 2

Navigating in Audioguiarte

Imagine you are facing this painting in the museum, find information about it from Audioguiarte.

Task 3

Complex navigation

Let say your favorite painting is on Floor [number] room [number]. What would you do to find more information about this room?

4. Prepare Test Materials

Moderator Script

Read by the moderators to attain consistency in data across tests.

Informed Consent Form

Signed by the participant, getting consent for audio, video and screen recording.

Pre-test Questionnaire

Filled by the participants before the tests to get to know them better and support the findings.

Post-test Questionnaire

Filled by the participants after the test, measuring their impressions

5. Set up Technology Environment

Before each session, we sent out links of UserZoom (for moderating and recording the tests) and Zoom (for presenting visual materials) to participants, asking them to open them on computers and mobile devices before test session.

6. Pilot Test

A pilot test was conducted with a fellow researcher outside the team. The technical mechanics of the tests were also tested.

Step 3

Conduct the Test

We conducted 8 user tests, 40 min sessions each. Each researcher conducted two tests by using the standardized script. Here is a summary of our user tests:

Stimluated the Museum’s Environment

We presented the painting to the participants through Zoom. We recorded participants’ audio and their mobile screen sharing and moderated the tests through UserZoom Go.

Think-Aloud Approach

Participants were instructed to think out loud as they were performing the tests and follow-up task questions. This allowed us to collect qualitative insights.

Pre- and Post- Test Surveys

Pre-test and post-test surveys were sent out to get quantitative data from the participants

Step 4

Analyze the Data

Participants Profile

This study employed people who would visit museums at least once in a year.

Participants’ age profile variated between 18 to 39. No gender declaration was asked from the users.

Data Analysis

We gathered the raw data at the same place, compiling the researcher’s notes, observations, questionnaires' results and video recordings.

Then, we identified and prioritised the most important usability issues by the number of occurrences.

What were the results?

Findings

Overall Findings

75%

62.5%

75%

of the participants have rated visual design as their favorite aspect of the app.

of the participants have rated the app as either “easy to use” or “very easy to use” in the post- test questionnaire.

of the participants have reported that they would recommend this app to their friends or families.

4 Major Problem Areas

A

Navigation

B

Information Architecture

C

Visual Design

D

Content

B. Information Architecture

1. Users were confused over the structure of the menu items

“On the information page, I wish I could click the author, the room number," view more information related.”

Problems

The participants could not find the page Room, which is inside the main label Permanent Collection, showing that users could not relate the concept of rooms with the permanent collection.

The label Audio Descriptions confused users.

Recommendations

Combine the two pages Highlighted Works and 50 Selected Works into one page and rename it as 50 Highlighted Works. This action would reduce users’ options, lower users’ cognitive load and also reduce redundancy in the menu.

Remove the main label Permanent Collection

Remove the page Audio Descriptions as the content inside the page does not have any noticeable differences, so as to reduce ambiguity in the menu.

2. Users did not understand the naming of the menu items

“I need to click on each collection (tab) to see the differences.” 
“Why is
Children's Tour outside of the Thematic Tours?”

Problems

Many participants were confused about the Spanish used on the navigation menu. 

The naming confused users and reduces users’ confidence in the interface especially.

Recommendations

Name all the labels in English after users choose the English language pages.

Rename Children’s Tour as For Children to differentiate the content’s targeted audience.

C. Visual Design

Lack of visual cues between items with different content hierarchy

“I thought room 7's audio guide was Carpaccio's audio guide.”

Problem

The participants did not realize the difference between room explanations and individual art explanations while scrolling through.

Recommendation

Increase the width of cards for rooms and lower the width of cards for paintings, contrasting the differences in advance of users clicking on it.

Problem

The participants often failed to understand that the audio they were listening to was about the room that the painting was located in, but not the painting itself.

Recommendation

Change the background for room audio, helping users understand that they are at different content.

D. Content

1. Inconsistent use of imagery

“I need to click on each collection (tab) to see the differences.” 
“Why is
Children's Tour outside of the Thematic Tours?”

Problem

The lack of painting's thumbnails on the thematic tour pages has made it hard for the participants to relate to the paintings.

Recommendation

Adding thumbnails of pictures to all the paintings in the app

2. Lack of supporting text

“I need to click on each collection (tab) to see the differences.” 
“Why is
Children's Tour outside of the Thematic Tours?”

Problem

Many participants have listened to the wrong audio but believed they were in the correct place. The participants reported that a short caption or transcript would alleviate the problem for them.

Recommendation

Add brief introductions of paintings and the year of creation to the painting description pages. This can give users a comprehensive and brief impression of the artwork before listening to the audio. 

Conclusion & Lessons Learned

Client's feedback

She mentioned that she had already known some usability issues, however, we provided further alternative insights she had not even thought about, and also actionable suggestions that they can implement immediately.  

We presented our usability findings to our client, Audioguiarte. The representative from the Audioguiarte listened to our presentation, and she thought that our findings were extremely helpful for Audioguiarte to improve the app in the future.

Next Steps

Conduct further studies on the information architecture recommendations above mentioned

I suggest doing a treejack test on newly designed information architecture and navigation systems. If so, participants would be asked to find specific information based on the new navigation system. Also, I would do a hybrid card sorting or closed card sorting to revamp the sitemap of the app, helping us to get a sense of how users would categorize different paintings’ collections, floors’ audio guides, painting’s audio guides, and thematic tours in the app.

Actual usage behavior of the app in the actual museum’s environment

As we could not conduct our usability tests in the natural museum environment due to the Covid-19 pandemic, I strongly hope that I would be able to find participants or actual visitors of the museum to conduct the test after reopening the museum. Moreover, although we are confident that our findings and recommendations are supported by the testing’s results and our expertise in user experiences, more user research and usability studies, no matter on our recommendations and the current app, are always useful to provide strong insights into the mental model of the target users.

Adding new features to provide efficient ways to access audio guides in the museum

As we found out that users often need to click through pages to find the audio guide of a specific painting, having a QR code or an assigned number for each painting in the museum could help provide a direct way to get the information they need. What users need to do is scanning the QR code or typing in the number put next to the painting.
 
Other than that, we suggest adding an interactive map feature to assist visitors traveling inside the museum when they choose to do a thematic tour in the app. The paintings in the thematic tour in the app are scattered around the entire museum. If visitors were in the museum, they would need to find their own ways to walk to different paintings when they listened to the audio guide app. Hence, an interactive app that can show users where to go next and how to navigate inside the museum as they are listening to the thematic tours.

My Lessons Learned

Learning to resolve group’s conflict

Working in a group can be challenging. There were times that our groupmates had contradicting ideas on what tasks participants were going to do in the tests, what to include in our final report, and what suggestions we would provide. For example, the groupmates have argued what usability issues we would include in the main findings, as they thought the findings they have found are more important than the others. Then, I suggested using a table to summarize our observational data and count the number of occurrences of the issues to prioritize which findings to be put in the main report. 

Proactively stepping up to strive for good work

Whenever there were groupmates having issues in completing their parts of work, I would step in and help them finish their parts, such as writing other parts of findings, formatting the report, and doing mockups. Moreover, whenever I found there were mistakes/problems in the working area that my groupmates did, I found that immediately speak up was essential to achieve good work. Offering actionable suggestions and my helping hands would make groupmates feeling less offended and that they were being supported.

Over-communication is the key to virtual collaboration

This group project was all done virtually. Our groupmates were living in three very different timezones, which were EST (UTC -05:00), CET (UTC +01:00), TWT (UTC +08:00). We had always encountered difficulties to coordinate a time that we could all do a meeting. When coordinating a time to meet, it is always important to consider other people’s time and list out what time to meet based on every time zone. For example, when we messaged groupmates the meeting time, I would now use“ Name: 6pm EST | Name: 12am CET | Name: 6am TWT“ to show people what time they were going to meet according to their timezone. Otherwise, people would misinterpret the meeting time as their own timezones or even forget the time.

1. Users were confused over the structure of the menu items

“On the information page, I wish I could click the author, the room number," view more information related.”

Problem

Many participants were confused about the Spanish used in the labels on the navigation menu.

Recommendations

Combine the two pages Highlighted Works and 50 Selected Works into one page and rename it as 50 Highlighted Works. This action would reduce users’ options, lower  users’ cognitive load and also reduce redundancy in the menu.

Previous project

User Research & App Design

Dayly - The Estée Lauder Companies' Skincare App

Skincare App for The Estee Lauder Companies

Next project

Data Analysis

Ingenium Museums Websites Analysis

Ingenium