Removal of image obstacles for vehicle-mounted surrounding monitoring cameras by real-time video inpainting

One of the practical problems with surrounding view cameras (SMCs) of a vehicle is the degradation of image quality due to obstacles by substances adherent to their lens surface, such as raindrops and mud. Such image degradation could be improved by image restoration techniques that have been studied in the field of computer vision. However, to assist the driver, real time processing and fidelity of the recovered image are essential, which disqualifies most of the existing methods. In this study, we propose to adopt a recently developed video-inpainting method that can restore high-fidelity images in real time. It estimates optical flows using a CNN and use them to match occluded regions in the current frame to unoccluded regions in previous frames, restoring the former. Although the direct application does not lead to satisfactory results due to the peculiarities ofthe SMC videos, we show that two improvements make it possible to obtain good results that are useful in practice. One is to use a model-based flow estimation method to obtain target flows for training the CNN, and the other is to improve how the estimated flows are used to match the current and previous frames. We conducted experiments using real images mainly of parking spaces in urban areas. The results, including subjective evaluation, show the effectiveness of our approach.