Purpose: Real‐time tumor position/shape measurement and dynamic beam tracking techniques allow accurate and continuous irradiation to moving tumor, but there can be a delay of several hundred milliseconds between observation and irradiation. A time‐variant seasonal autoregressive (TVSAR) model has been proposed for compensating the delay by predicting respiratory tumor motion with sub‐millimeter accuracy for a second latency. This is the‐state‐of‐the‐art model for almost regular breathing prediction so far. In this study, we propose an extended prediction method based on TVSAR to be usable for various breathing patterns, by predicting the residual component obtained from conventional TVSAR. Methods: An essential core of the method is to take into account the residual component that is not predictable by only TVSAR. The residual component involves baseline shift, amplitude variation, and so on. In this study, the time series of the residual obtained for every new sample are predicted by using autoregressive (AR) model. The order and parameters of the AR model is adaptively determined for each residual component by using an information criterion. Eleven data sets of 3‐D lung tumor motion, observed at Georgetown University Hospital by using Cyberknife Synchrony system, were used for evaluation of the prediction performance. Results: Experimental results indicated that the proposed method is superior to those of conventional and the state‐of‐the‐art methods for 0 to 1 s ahead prediction. The average prediction error of the proposed method was 0.920 plus/minus 0.348 mm for 0.5 s forward prediction. Conclusion: We have developed the new prediction method based on TVSAR model with adaptive residual prediction. The new method can predict various respiratory motions including not only regular but also a variety of irregular breathing patterns and thus can compensate the bad effect of the delay in dynamic irradiation system for moving tumor tracking. A part of this work has been financially supported by Varian Medical Systems Inc., Palo Alto, CA and Japan Society for the Promotion of Science (JSPS), Japan.