Cost of a Usability Evaluation: Bootstrap Discovery Behaviour Model

The international debate on the costs of usability evaluation is mainly focused on the return on investment (ROI) model of Nielsen and Landauer (1993). In this study, the ROI model properties and limits are discussed in order to identify the base of an alternative model that considers a large number of variables for the estimation of the number of participants needed for a usability evaluation. Using the bootstrapping statistical inference (Efron,1979), we propose a model, named Bootstrap Discovery Behaviour (BDB), suitable to take into account: a) the interface properties, as the properties at zero condition of evaluation; and b) the probability that the population discovery behaviour is represented by all the possible discovery behaviour of a sample. The data of two experimental groups, one of users and one of experts, are involved in the evaluation of a website. Applying the BDB model to the problems identified by the two groups we found that 13 experts and 20 users are needed to obtain the 80% of usability problems, instead of 6 experts and 7 users required according to the estimation of the discovery likelihood provided by the ROI model. The power of the BDB model rests on the most predictive validity for accurate predictions about a participants’ future discovery behaviour of usability problem, as regards the ROI model.