Recently, lithotripsy (kidney stone treatment) using HIFU (High Intensity Focused Ultrasound) was developed by researchers in therapeutic ultrasound field. The lithotripsy crushes kidney stones powder, therefore, do not harm to the surrounding tissues of the kidney stones. However, it is necessary to continuously emit high intensity ultrasound waves on a target kidney stone during treatment. Therefore, HIFU transducers should follow a target kidney stone which moves due to respiration and heartbeat. In this paper, ultrasound-based visual servoing system is described. Two ultrasound probes and a HIFU transducer are mounted on the end effector of a xyz stage machine. The two ultrasound probes visually keep track of the target kidney stone within the body, and servo the xyz stage machine. The required techniques, such as conversion of a frame of radio frequency (RF) echo signals into an ultrasound bright mode image, visual tracking, pose estimation and control, are explained.