Biases in usability testing: How to minimize biases in software testing

In user experience (UX) design, qualitative usability tests and tests with test subjects are indispensable for gaining an in-depth understanding of user behavior, detecting errors at an early stage and improving the user-friendliness of digital products. While quantitative methods deliver numbers, qualitative tests in user testing, usability testing and software testing go one step further: they explore the thoughts, emotions and behaviors of users.

One question we often hear in relation to usability testing is: “Can we really trust the statements made by our test subjects in usability tests or do they distort the results?”.

In our blog post, we shed light on why qualitative usability tests are so valuable, how cognitive biases can influence the reliability of these statements and what role test management plays in this.

Explanation Biases A bias is a systematic distortion or bias in perception, judgment or decision-making. Biases are often caused by prejudices, stereotypes or unconscious thought patterns that influence our interpretation of information. They can occur in many areas, such as psychology (e.g. confirmation bias) or data analysis (e.g. selection bias).

Why do we need qualitative usability tests in software testing? 

Qualitative usability tests in the context of user testing and software testing are more than just a method for finding errors. They make it possible to gain in-depth insights into the behavior and experiences of users. Through these tests, undiscovered problems can be uncovered and the causes behind users' decisions can be understood. Thanks to detailed feedback from just a few participants, targeted improvements can be made. The flexibility of these tests makes it possible to focus on the essential aspects and gather insights that quantitative methods often cannot provide.

Test subjects with visual impairments: Experiences from a usability test 

---

Advantages and disadvantages 
of user testing and usability tests  

Advantages	Disadvantages
In-depth insights into user behavior Motivations, thought processes and emotions of users	Subjectivity of the results Test subjects must be representative of the entire target group
Uncovering unexpected problems Open test structure makes it easier to find unexpected errors	Limited generalizability Samples may not be transferable to the entire user population
Flexibility Explorative approach enables reaction to spontaneously arising problems or questions	Time/resource-intensive analysis Evaluation of the extensive observations and notes
Small number of participants Just 5 test subjects reveal the most common usability problems (Law of Diminishing Returns)	Measurement of criticality  Severity or frequency of a problem cannot be precisely determined
User-centered approach Concrete feedback to improve the user experience	Dependence on moderation Strong dependence on the skills of the moderators

How do biases influence testing with test subjects?

In the world of user experience (UX), qualitative usability tests are invaluable. They provide deeper insights into user behavior and experiences by going beyond mere numbers and data points. Instead of just measuring the “what”, these tests explore the “why” behind user behavior through interviews, observations and open-ended questions. User tests are therefore an important quality assurance method for all digital applications, such as apps, websites and IoT products.

Usability tests for IoT products: Planning and setup of the setup

But how reliable are the statements made by the test subjects during these tests? They reflect opinions and assessments, but how reliable are they? In software testing and usability testing, it is important to minimize bias in order to obtain accurate results. Therefore, it is important to understand both the benefits of qualitative usability testing and the potential biases caused by cognitive biases.

Despite their strengths, qualitative usability tests have their challenges. Cognitive biases (systematic errors in thinking) can affect the reliability of respondents' statements. These biases influence how respondents perceive and communicate their experiences, which can lead to inaccurate or distorted feedback.

Here are some common cognitive biases that can occur in usability tests:

                                         

Strategies for minimizing bias in usability tests 

In view of the possible cognitive biases, the question arises as to whether we can fully trust the statements made by test persons in usability tests in software testing. The short answer is: No, not always. It is important to interpret respondent feedback in the context of these possible biases. However, this does not mean that their statements are worthless - rather, they should be viewed with a healthy skepticism and strategies applied to reduce them.

The following strategies could be helpful:

                                         

The role of test management in software and user testing

The test moderator therefore plays a decisive role during a usability test. To minimize bias during usability tests conducted at Appmatics, we ensure high interview quality by experienced moderators. As test moderators, we not only take into account the verbal statements of the test subjects, but also their observed behavior. For example, if a respondent claims that a task was easy even though there were obvious difficulties, the moderator must question this discrepancy. This incongruence can provide valuable clues as to where and why the user may have had difficulties. It could indicate that a certain aspect of the user interface or interaction design needs to be revised, even if the respondent claims to have had no problems.

One of the most important tasks of a test administrator is to distinguish between what is said and what is shown and to analyze what biases may be at play. This requires both a deep understanding of cognitive biases and experience in observing and interpreting user behavior.

                                         

Conclusion: Recognizing and minimizing biases in usability testing

Qualitative usability tests and user testing are crucial methods in UX design and software testing to gain in-depth insights into the behavior and perception of users. Despite the challenges of cognitive biases and distortions, respondent statements provide valuable feedback. By understanding these biases and using appropriate strategies, UX experts can significantly improve the validity of user and software tests.  A keen eye and a deep understanding of human psychology makes it easier to identify biases and incorporate the observed behavior into the analysis. Test management therefore plays a central role in recognizing biases and ensuring that the insights gained are based on actual user needs.