Abstracting essay evaluation structure by using multidimensional alpha coefficient

When we abstract the structure of three-way essay evaluation data, the aggregated matrix tends to show more unidimensional feature than that of each rater if the equal weights are used. Multidimensional Alpha Coefficient (MAC) proposed by Yanai (1994) gives independent sets of scoring weights which vary dimensionality of the aggregated matrices. The present study tried to extract the evaluation structure from three-way essay data by using the method for preservation of the original multidimensionality. The essay data of Taira (1995) evaluated by seven raters were used. Some criteria, including the factor analysis with Procrustes rotation method, were set to choose the most suitable weights. The result showed that the weights for the sixth solution of MAC was regarded as the best. The aggregated matrix yielded three factors, consisting of one more dimension compared with the former analysis. The revised path diagram showed much clearer causal relationships. The emotional factors on writing and reading had an effect on the ability of story making, whereas Writing Habits influenced only upon the Loyalty to the Task Condition.