Investigating performance characteristics of novice evaluators in heuristic usability evaluations

Abstract

Heuristic Evaluation (HE) of usability of web systems has received special attention in the literature. Traditional heuristics are not sufficient to evaluate such systems. In HEs, it is recommended that expert evaluators be employed in HEs. An evaluator became expert after several years of job in usability area. However, many companies resort to novice evaluators in HEs. The effect these evaluators cause in HE is not well known, and deeper research studies about this still remain as a gap in the literature. This study aims to investigate if a set of heuristic focused on usability of web systems can help novice evaluators to find more usability problems than traditional heuristics in a HE of a web system. Results from the HE were compared to results of a test with real users. Obtained results showed the effect of using different sets of heuristics and the quality of reports of novice evaluators. The conclusions of the study showed the importance of investigating the effects of novice evaluators in HE, as well as the extent to which using different sets of heuristics can help improve the overlap between HE results performed by novice evaluators and problems encountered by real users.