A. E. Blandford , S. J. Buckingham Shum & R. M. Young
of the Programmable Users Models (PUM) technique (Young, Green and Simon, 1989; Blandford and Young, 1995b). In contrast, the third (Buckingham Shum) has no personal stake in the technique, but has collaborated with different HCI modelling groups to facilitate the transfer of their techniques to practitioners. As such, he was able to take an outsider?s view of the outcome of the study. Ideally, any study into the process and results of applying such a technique by practitioners would be conducted in an independent way, without direct involvement of the developers. This is more possible for a mature technique, such as Cognitive Walkthrough or GOMS, (e.g. John and Packer, 1995) for which there is tested training material and enough documented experience of use to allow a designer to work alone and for data to be analysed by the researchers. However, when the technique is less well encapsulated, and the training material is less mature, it is necessary for the original developers to be involved, to interpret the output from the study and to recognise some of the deeper understandings or misconceptions of the practitioners. PUM is still at this stage.
Consequently, the main aim of the work reported here is relatively modest: to test an approach to transferring the ?PUM Instruction Language? (IL) to trainee software engineers. Three sub-questions we were addressing while doing this work were:
? Does using the IL make a discernible difference to the kinds of usability assessments made?
? How do trainee software engineers use IL after one day?s training? and
? How could the training be improved?
The first sub-question is an important one, particularly in relation to understanding how much of the leverage of a particular technique derives from the technique itself, and how much from the skill of the analysts using it. The second question, answered through video analysis of IL in use, is pertinent to understanding what actually happens when one learns to use IL; the video data allows us to address issues such as what concepts beginners struggle with, how IL constructs may contribute to design insight, and what kinds of process support can be provided to accelerate the acquisition of IL expertise. As John and Packer (1995) note, you can only distinguish with any certainty between technique and craft skill when you have non-originators using a technique and when you have some process data, to enable you to assess what analysts did, and how they gained insights into the design problem. One of the aims of this study is to address both of these points. One cautionary note is that the process data in this study was taken after students had been exposed to the technique for only about six hours, so still represents the analysis of novices.
This study also raises generic issues about the transfer of theoretically-based HCI design techniques to practitioners. In the discussion following our report of the empirical study, we focus on several issues which confront all those seeking to develop and transfer such approaches to practitioners, namely the level of theoretical training required, the nature and encapsulation of craft skill, and the different degrees of formality with which a notation can be deployed.
1.1. Related studies
Other studies have investigated the process skills required to apply formal modelling techniques and other HCI design formalisms. These include studies on Cognitive Complexity Theory and Cognitive Walkthroughs.
(Kieras, 1988) has described work to make Cognitive Complexity Theory (Kieras & Polson, 1985), as based on GOMS modelling (Card, Moran & Newell, 1983), a practical user interface design technique. Kieras recognises that CCT and GOMS, despite their formality, leave decisions to the analyst which can make a significant difference to the validity or usefulness of the analysis. He therefore provides some guidance on such issues as making ?judgement calls? on user cognition in the absence of reliable data, deciding what tasks to analyse, when to conduct a GOMS task analysis, and GOMS-based criteria for evaluating and improving the design. Kieras proposes and exemplifies an enhanced notation called ?Natural GOMS Language? (NGOMSL), designed to assist methodologically in conducting more systematic analyses, and to be more readable.
The Cognitive Walkthrough (CW), the latest version of which is described in (Wharton, et al., 1994) is not a modelling technique, but a structured methodology for critiquing a user interface. However, it is directly relevant to the present paper, since it is based on a cognitive model of learning called CE+ (Polson & Lewis, 1990), but seeks to deliver the power of this model in the form of a checklist of eight principles based on CE+ without actually requiring CW users to know any more about it. Several studies of CW have now been reported, which together are an excellent example of the use of empirical data to drive the design of a technique to meet the skills of its targeted users and the