In this paper, we present a robust Phase-Only Correlation (POC)-based motion estimation method for video sequences. Robust motion estimation is indispensable for many applications such as mesh-based video coding, stereo vision, and super-resolution imaging. In our proposed method, the motion vector of a point in a video frame is adaptively switched between motion vectors obtained by two motion estimation methods: (i) POC-based full search and (ii) POC-based hierarchical search. This approach can reliably detect both global and local motion in video sequences. We evaluate the robustness of our proposed method in mesh-based motion compensation. Experimental results show that our proposed method performs significantly better than conventional full search using Sum of Absolute Differences (SAD).