Validity of a Japanese Vocabulary Test: Cognitive Analysis with Rule Space Methodology

Rule Space Methodology (RSM) has been widely used as a promising method for providing diagnostic information in various testing areas. The purpose of the present study was to utilize RSM to verify the validity of the Japanese vocabulary test developed by Taira, Ono, and Hayashibe (1992). Rule Space analysis was applied to the responses of 1,500 high school student examinees; a satisfactory classification rate of over 80% was obtained, using an incidence matrix that included 23 attributes. The 17 major knowledge states identified were used to draw a network of knowledge states in a 2-dimensional cognitive space, with axes representing overall ability level and the unusualness of the response pattern. The results indicate that students who master the important attributes among those that are being measured would receive a considerable increase in their score, whereas those who master attributes regarded as test-wiseness would get almost no increase at all. The results of this study suggest that Rule Space Methodology could be used for verifying the validity of a scale, as well as for diagnostic purposes.