Empathizing is defined as the drive to identify the mental states of others for predicting their behavior and responding with an appropriate emotion. Systemizing is defined as the drive to analyze a system in terms of the rules that govern the system in order to predict its behavior. Using voxel-based morphometry and questionnaires in a large sample of normal, righthanded young adults, we investigated the regional gray matter volume (rGMV) correlates of empathizing and systemizing and additionally those of the D score, which is the difference between systemizing and empathizing, to reveal the comprehensive picture of those correlates. Negative rGMV correlates of empathizing and positive rGMV correlates of the D score (formed by the negative correlation between rGMV and empathizing), were found primarily in nodes in the default mode network, mirror neuron system, dorsal anterior cingulate cortex, and the lateral part of the prefrontal cortex together with other areas. Positive rGMV correlates of systemizing and of the D score (formed by the positive correlation between rGMV and systemizing) were found primarily in nodes in the external attention system, middle cingulate cortex, and other regions. Negative rGMV correlates of systemizing were found in an area close to the left posterior insula and putamen. These findings reconcile some previously inconsistent findings, provide other new findings and suggest that these areas contribute to empathizing-systemizing. Furthermore, the negative/positive rGMV correlates of empathizing and positive/ negative rGMV correlates of systemizing overlapped substantially. This may be in line with the notion that empathizing and systemizing compete neurally in the brain.