Go to previous pageGo to next page

1.4.1. Testing techniques

Testing Aims

The aim of the usability testing was to evaluate how usable the system being developed was and how closely it matched the needs and expectations of users. The testing was carried out at regular intervals, each summer, during the life time of the project, with the results feeding into back into the prototypes being developed. The aims of the testing were:

  1. Is the device suitable for mobile use in terms of it size, weight, display clarity?
  2. Is the application performance fast enough for the users?
  3. Is the data transfer rate sufficient for the users?
  4. Does the location determining technology provide a position with in a time acceptable to user?
  5. Is the information logically organized and grouped? Can the user easily locate the information they are looking for?
  6. Are the graphics used and the icons recognizable to users and do they facilitate ease of use/understanding?
  7. Are the maps sufficiently, clear to use for orientation, navigation and obtaining information?
  8. Can users successfully navigate through the application? Is the system status clear to users at all times? Is the navigation intuitive?
  9. Could the user obtain information to answer their questions? (pull services)
  10. Is the information timely and useful to the users activity? (push services)
  11. Is the user able to get support / help in using application when required?
  12. Is the user able to personalise the applications?
  13. Are there tasks that users will want to perform that are not currently supported by application and services?

These concerns were examined through a four complementary testing techniques. Techiques were selected because of their suitability to a mobile system, they ability to support quantitative analysis, and because they invited new ideas from the participants.

Shadowing

The aim of shadowing (also called monitoring or participant observation) is to observe a user with the application in the envisaged context of use. In shadowing, a researcher follows a user during their time with the system, observing and recording any problems and questions the user has. The researcher never offers any information but only answers the questions of the shadowed visitors.
A visitor being shadowed in TexelA visitor being shadowed in Texel
In the WebPark testing the research was provided with a form to fill it that allowed different types of problem to be easily identified. An example of the shadowing form used in the first testing is provided below.

A shadowing formA shadowing form

Questionaire

The main advantage of the questionaire was that it provided a structured method to quantify the usability of the system and compare these result over subsequent years. Questionnaires differ from surveys in that they are written lists, not ad hoc unstructured interviews. They require a high degree of question structuring to ensure a logical narrative thread for the user to follow. This must be matched with simple, unambiguous questions to maximise the number of respondents. The principles of (Schnell et al. 1998) were adhered to when designing the questionaire. These are:

  • The layout of the questionnaire needs to be clear and concise to lead the respondent through the questions and to make it easy for the to differentiate between the instructions/explanations on the one hand and the questions themselves on the other
  • The first few questions should be given as an introduction to lead attention and thoughts to the topic in general
  • Several questions should be given for each thematic group with filter questions at the beginning of a new question complex to make it possible to jump to the next question that applies for the respondent
  • Personal questions to provided a demographic background should be asked at the end of the questionnaire in order not to provoke an early break off by the respondent

A copy of the questionaire used for WebPark can be found here.

Interviews

Interviews are ad hoc conversations with users, where a set list of questions is asked and the user`s responses recorded. Interviews differ from questionnaires in that they are interactive and are not rigorously structured. The general form of the interview followed the structure of the questions outlied previously for the aims. One of the main advantages of interviews is that they suggest new ideas that can be employed in the subsequent design revisions.

Thinking aloud

The Thinking Aloud protocol is a technique used during usability testing. During the course of a test, where the participant is performing a task as part of a user scenario, the participant is asked to vocalize their thoughts, feelings, and opinions while interacting with the application. The participant, the product to be tested and a scenario in which the tests are to be perform are selected. The participant is asked to perform the tasks using the product, and explain what they're thinking about while working with the product's interface. The chief advantage of thinking aloud is that it provides a picture of the users mental model of the new technology.



Go to previous page
Go to next page