Before the COVID-19 outbreak, I got a chance to join a UX research session held by Agoda, Asia Pacific’s most influential global online reservation provider for hotels, vacation rentals, and flights. They also happen to be one of our clients, who we do WordPress development for.

One of our operating team values at Fresh is to share knowledge boldly and humbly, which is something that came to mind during the presentation. We have our own UX & CX Research service at Fresh, a time-tested approach to the craft, but it’s interesting to see what other organizations in the industry are doing and learn about the similarities and differences between our approaches.

In the session, I learned some very interesting things about how Agoda’s internal team used research to better address user needs on their website. Reflecting on my learnings, I realized that research isn’t always appreciated as a necessary component of UX design, and is often viewed as an added expense rather than an integral part of the process. But as illustrated in a recent post by Aravind Ravi – Creating a UX Research System with Airtable – when framed in a way that stakeholders can understand, research is incredibly valuable.

Companies – regardless of whether they’re a startup or a more established organization – may choose not to invest in user research because:

1. They believe they already know what users want
2. They believe user research is expensive and daunting
3. They believe user research methods are complicated
4. They aren’t convinced more data will improve their product

The truth is, hearing what users have to say about your product increases your chances of success whether you’re a startup or a veteran leader in your industry.

At Fresh, we have an established approach. But it’s interesting to look at what worked at Agoda, examine the similarities and differences between our approaches, and identify the core of why research matters. Communicating “the why” is essential to convincing clients who are on the fence about the value of research to invest.

Agoda’s Challenge: Growing an International Research Team

In the talk, Aaron Moy (then head of UX research at Agoda) shared insights about his team’s experience scaling its practice and finding the balance between a product, user needs, and business requirements. At first, UX research at Agoda was underdeveloped, and as the company scaled, they realized they needed to improve their process. The main areas they focused on in growing their UX research practice were:

1. People – the power of talent
2. Method – the importance of an approach
3. Impact – the value of measurement

People: Utilizing Your Talent

At first, there were very few people on Agoda’s UX research team. Individual members had to manage research across multiple products, which resulted in many pain points. They were stretched too thin.

To solve the problem, the UX work was eventually embedded into smaller phases of product design. Agoda paired UX designers and researchers throughout the product development process, as illustrated in the diagram below:

At Fresh, all of our UX designers have a background in research, which allows them to consult at multiple stages of the project. But our methodology has also been developed, refined, and improved over more than a decade. When working with a smaller team and without a proven process, it can be helpful to pair individuals with different skill sets together, collaborating at various stages to ensure that research and design work are done in tandem.

Method: Choosing a Unified Approach

One key project Agoda took on to improve their research process was the redesign of their website. In the presentation, Aaron Moy said the original Agoda website needed an extensive overhaul. The team considered a variety of methods to collect feedback from users. As illustrated in the diagram below, the number of research methods available to UX practitioners is daunting:

Image Credit: Nielsen Norman Group

In the end, Agoda’s team narrowed their approach down to several focused UX research approaches, with the aim of developing a holistic view of user pain points. Their goal was to use design thinking to understand their users, define their current challenges, ideate potential solutions, create a prototype, and then test with users to eventually find an authentic solution.

Built into each phase were different methods. As an example, to create empathy, focus groups, interviews, and customer feedback would all be appropriate. Breaking research down into its parts allows you to choose the best approach based on constraints like timeline, budget, and talent resources. In our experience at Fresh, research isn’t one-size-fits-all. Tailoring the plan to a client and their unique needs is vital.

Impact: Measuring Change

Finally, the Agoda team chose to pursue goals with proven business value that could be collectively understood and agreed upon throughout the organization. By focusing on KPIs that were easy to track and maintain, they could more easily measure the overall product success and increase stakeholder satisfaction.

When Agoda decided what methods to apply to their user research, they chose to study a mix of:

• What people do
• What people say
• Qualitative data
• Quantitative data

They further divided their research into three stages:

1. Foundational Research: A pathfinding stage where teams explore the problem and analyze it from multiple angles.
2. Generational Research: A stage where teams develop a solution and examines its value via user testing. 
3. Evolutional Research: A stage where teams refine the product until it’s right.

The process is similar in many ways to what we do at Fresh:

Whereas Agoda’s process is slightly more general (less specific in terms of what happens at the foundational, generational, and evolutional stages), we place a strong emphasis on architecture and wireframing in virtually every UX design project, as indicated in the process graphic above. The idea is ensuring that, first and foremost, the design has structural integrity and that it closely mirrors the user’s cognitive model. We test and revise repeatedly at the wireframe and mockup stage, conducting smaller sprints to ensure that design decisions are tied to research findings and are vetted with actual users before any final production is complete.

One more thing to note about Agoda’s approach: instead of interviewing customers out of the gate, they looked at the available data to form initial questions and hypotheses. We do something similar. But why look at the existing data first? The answer: it’s low-cost, fast, and sets the context for the rest of the research effort.

Things like web and app analytics, system logs, transactional data (in the case of e-commerce or any site mechanism that asks for a user’s information), and customer support logs can give you a sense of the pain points users are experiencing before you ask them direct questions. Existing data can help you form a plan.

The value of establishing context and setting benchmarks cannot be overstated, and like Agoda, we recommend it.

The Role of Usability Testing and 1:1 Interviews

Usability testing helped Agoda better understand how users interact with their site. Usability testing can be fast and inexpensive, and the data gathered directly informs design improvements. It’s one of the best ways to build user understanding and empathy early on.

Here are three of Agoda’s takeaways on how to get the most out of usability testing, views shared by many thought leaders in the UX industry:

• Target a 100% completion rate. Different respondents will provide different insights. Some respondents may need more encouragement to follow through on a specific task. For your sample to be representative, you need to get as close to a 100% completion rate as possible. In other words, if an individual user needs a prompt, give it to them, without giving the task away.
• Limit usability testing to five users. At Fresh, in line with both Agoda and the Nielsen Norman Group, we’ve found that the best results come from testing no more than five users and running as many small tests as resources allow. There are diminishing returns beyond testing five users, as the same insights tend to surface without enough nuance to reward investing in finding more users and compensating them for their time.

Image Credit: Nielsen Norman Group

In a project for the University of Washington Research office, we redesigned a site catering to a wide range of academics and staff members. We conducted user testing, but a smaller number of testers – with a wider range of tests – still surfaces the common challenges that users across the institution face. You don’t need to test every person who has ever used a product, because likely, they’ll have experienced the same challenges as the people you do test, as long as the test conducted is comprehensive.

• Use responses to adjust features. The goal of usability testing is to find problems and implement solutions, so the feedback you get from testing is only valuable if you act on it. From Agoda’s usability testing during their website redesign, they were able to spot a variety of problems and implement solutions to improve the site. This is a somewhat obvious takeaway, but the point is that research conducted in a vacuum is useless. Research should be used to inform, validate, and improve a design.

Agoda’s End Result: Proof in the Numbers

After several rounds of research, iterations, and reviews, Agoda went live with a grid structure design made of tiles to improve the site’s scalability in the future.

Agoda’s homepage

User data revealed engagement on the homepage increased and the site saw lower bounce and exit rates. Time spent on the site increased, new content tiles had a better rate of conversion, and more users were progressing down the funnel to the search page. Here are some specific stats:

• 1.5% increase in search
• 4000 clicks/day on the discount section, which was a 29.1% new user conversion
• 3.4% increase in signup conversion

In addition to the improvement in metrics, research team members no longer felt lost when they wanted to get their content up on the homepage because the new process they had defined was clear and quick to adopt and use. Agoda also leveraged the earlier survey results to craft a more user-friendly content hierarchy.

Referring back to the importance of benchmarking (i.e. gathering data before a redesign is completed, and often before you even talk to users), is that it allows you to measure whether search actually improved, or whether the increased conversion rate satisfies your priority KPIs.

At Fresh, we stand behind the value of thorough benchmarking. Without it, research data isn’t actionable or useful.

One Size Doesn’t Fit All: The Value of Customizing Your Research Approach

Let’s return to some of the reasons companies might not want to invest in research:

1. They believe they already know what users want

Companies do have decades of historical wisdom about their users, but companies also have blind spots. Assumptions are dangerous, as are attachments to legacy product features that could benefit from being analyzed and improved. Stakeholder interviews are an essential piece of the research puzzle because a company’s knowledge and experience matter. But most companies can benefit from an even deeper understanding of their users, which is why we include research as a part of our process.

2. They believe user research is expensive and daunting

As stated previously, research is not one-size-fits-all. A company with a small budget can still get a solid body of data about its users and products. At Fresh, we emphasize customizing research plans to fit our clients’ needs, knowing that research without boundaries can become never-ending, without yielding much return beyond a certain point.

3. They believe user research methods are complicated

Some research methods are complicated, but some are very straightforward. With a basic test plan – including background questions, time-based tasks, and task difficulty ratings – you can reveal a massive number of insights about user pain points with a product. Complex, multi-year studies are complicated to execute and unrealistic for many, but they’re not always necessary either. Research can be as simple as having conversations with users, gathering quantitative and qualitative data, and using that information to make design decisions.

4. They aren’t convinced more data will improve their product

More data isn’t always a bonus, but good data is. This comes back to the previous point about clients having decades of historical wisdom about their products and services, regarding what has worked and what hasn’t. They might have a lot of data to use, but the methods for gathering might not be the best, yielding data that’s skewed. In this sense, they’re right – more of the same data wouldn’t improve their product. But new data, gathered with tested approaches, would be beneficial.

Agoda discovered that when the design was driven by data, users were given a seat at the table, a voice in determining the ultimate trajectory of the product. That, in essence, is why UX research matters.