Disasters such as earthquakes, hurricanes, and flooding are responsible for large-scale infrastructure damages and loss of human lives. Immediately after disaster strikes, one of the most critical and difficult tasks is accurately assessing the extent and severity of the disaster. This task is especially challenging in areas isolated by the disaster; in such cases, remote sensing information provides the best alternative to tackle this problem. This paper presents a damage mapping framework using remote sensing imagery acquired from previous disasters. The proposed deep learning-based framework is trained to learn features related to building damage using imagery from previous disasters that were collected from different regions around the world. Then, it is tested to recognize damage from a different urban environment.