The INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. We report here a study of intensive care physicians reconciling 2 perspectives on their patients. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, or changing the assigned categories) and the actual rule-set.
Each of the 3 experts achieved a very high degree of consensus (鈭?7%) between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). We then had the experts produce a common rule-set and then refine their several sets of annotations against it; this again resulted in inter-expert agreements of 鈭?7%. The resulting rule-set can then be used in applications with considerable confidence.
This study has shown that under some circumstances, it is possible for domain experts to achieve a high degree of correlation between 2 perspectives of the same task. The experts agreed that the immediate feedback provided by INSIGHT was a significant contribution to this successful outcome.