Our impressions of other people are formed mainly from the two possible factors of facial attractiveness and trustworthiness. Previous studies have shown the importance of orbitofrontal-hippocampal interactions in the better remembering of attractive faces, and psychological data have indicated that faces giving an impression of untrustworthiness are remembered more accurately than those giving an impression of trustworthiness. However, the neural mechanisms of the latter effect are largely unknown. To investigate this issue, we investigated neural activities with event-related fMRI while the female participants rated their impressions of the personalities of men in terms of trustworthiness. After the rating, memory for faces was tested to identify successful encoding activity. As expected, faces that gave bad impressions were remembered better than those that gave neutral or good impressions. In fMRI data, right insular activity reflected an increasing function of bad impressions, and bilateral hippocampal activities predicted subsequent memory success. Additionally, correlation between these insular and hippocampal regions was significant only in the encoding of faces associated with a bad impression. Better memory for faces associated with an impression of bad personality could reflect greater interaction between the avoidance-related insular region and the encoding-related hippocampal region.