On March 11, 2011, Japan experienced an earthquake of magnitude 9.0 and subsequent enormous tsunamis. This disaster destroyed many coastal cities and caused nearly 20,000 casualties. In the aftermath of the disaster, many tsunami survivors who lost their homes were forced to live in small temporary apartments. Although all tsunami survivors were at risk of deteriorating health, the elderly people were particularly at a great risk with regard to not only their physical health but also their mental health. In the present study, we performed a longitudinal cohort study to investigate and analyze health conditions and cognitive functions at 28, 32, and 42 months after the disaster in the elderly people who were forced to reside in temporary apartments in Kesennuma, a city severely damaged by the tsunamis. The ratio of people considered to be cognitively impaired significantly increased during the research period. On the other hand, the mean scores of the Kessler Psychological Distress Scale-6 and Athens Insomnia Scale improved based on the comparison between the data at 24 and 42 months. The multiple logistic regression analysis revealed that frequency of "out-of-home activities" and "walking duration" were independently associated with an increase in the ratio of people with cognitive impairment. We concluded that the elderly people living in temporary apartments were at a high risk of cognitive impairment and "out-of-home activities" and "walking" could possibly maintain the stability of cognitive functions.