The Internet of Things (IoT) is becoming increasingly present in our everyday lives. Whether through smart home devices or the automation of operating facilities (Industry 4.0) - IoT technologies, which are networked with each other via the Internet, have become an integral part of our everyday lives.
For this reason, there is also an ever-increasing demand for usability tests to be carried out with IoT devices. As with every digital product, the user experience plays a major role in its success.
Translated with DeepL
Our experience with IoT devices in usability testing
While testing IoT devices and regular apps or websites is similar on many levels, there are certain factors that need to be considered when usability testing IoT devices that could otherwise affect the quality of the study results.
Two exciting project examples that our UX research team had the opportunity to work on were a smart control grill and a smart home remote control. For these two products and their respective mobile apps, we conducted usability tests with typical average users. In these two and other projects with IoT devices, we were able to gather numerous experiences and learnings that we would like to share with you in this blog post.
Selection of a suitable setting
Working with an IoT product often requires a large laboratory in which all the equipment must be accommodated. Consequently, usability testing with IoT devices should not be carried out remotely if possible, in order to save the laboratory set-up in the private environment of the test persons and the additional transport of required technology and products. This also ensures that any technology can be well positioned in the room. On site, it is easiest and most efficient to observe how the test persons behave at the object of study. In addition, the test persons can fully concentrate on the IoT device to be tested - without external disturbing factors (e.g. noise, internet connection, other persons present in the household, etc.).
Planning and construction of the technical set-up
In order to be able to document important, physical interactions with the IoT device in the long term, it is necessary that, in addition to screen mirroring and the recording of gestures and facial expressions, the movements of the test person are also extensively recorded.
In our usability tests of products on mobile devices (e.g. apps), we use a document camera to record the typing behaviour on the smartphone or a small IoT device (e.g. a remote control). A second camera records the activities of the test person with the physical environment. For IoT products that are larger or contain relevant areas, such as an additional display, the technology setup is supplemented as needed with additional cameras for recording. In the example of the Smart Control Grill, the grill surface and grill controls were recorded with two additional cameras.
Plan enough time
An IoT usability test can generally take more time than testing regular products.
For one thing, planning and setting up the technology setup requires more effort. For each IoT product, an individual technology plan should be created that covers all the important functions of the device. Thus, more time has to be spent to build and test the extensive setup.
Furthermore, UX researchers have to deal with the IoT devices in addition to the app. As a result, the preparation of the usability test can take longer than usual, depending on the scope and complexity of the IoT device. If a product is in the early stages of development, there is also a higher risk that technical problems will complicate the preparation and implementation. Here it is particularly important to get in touch with relevant people early on to discuss problems of a technical nature so that they can be prevented or preempted during the meetings.
For these reasons, it may be useful to allow more time for preparation and implementation in advance. Pre-tests also help to assess the time frame for individual sessions and to weigh up which tasks and questions are clear and which should be reconsidered. Moreover, pre-tests can be used to make corrections to the setting. For example, camera positioning and settings of the test items can be undertaken. In the pre-test with the smart remote control, it became clear that the camera, which was supposed to record the typing behaviour on the remote control, was positioned too close and the radius had to be increased. In the pre-test, the subjects moved much more expansively and held the smart remote control in the direction of the experimenter to point out different controls.
Before and during the sessions
Depending on the usability test, it may be necessary to do additional preparation or follow-up between sessions. For example, the IoT devices need to be decoupled or reset to factory settings after each usability test.
If possible, it makes sense to have a helping hand (e.g. by the protocolist) between the individual sessions. This person can then take care of preparing the lab while the researcher or moderator prepares for the next usability test.
It is possible that the handling of IoT products is unfamiliar to some test persons and that they need more time to complete tasks. Especially in such scenarios, UX researchers are required to be patient and give the participants enough time to familiarise themselves with the app and the device. Some people may need assistance with certain use cases that need to be given in order to proceed with the test. These points should be documented by the UX research team as there is room for optimisation there.
The creation of a checklist that includes all additional steps for usability testing with IoT products can be helpful to keep an overview during usability testing.
Conclusion
Usability testing with a new technology can seem extremely challenging and complex at first glance. However, with thoughtful planning, preparation and also practical practice, overcoming this hurdle is less and generates valuable results to optimise products with high future potential and, above all, make them more accessible to people who are not yet familiar with the world of IoT products.
Author: Lining Bao
