On-line AdaTron learning of unlearnable rules

We study the on-line AdaTron learning of linearly nonseparable rules by a simple perceptron. Training examples are provided by a perceptron with a nonmonotonic transfer function that reduces to the usual monotonic relation in a certain limit. We find that, although the on-line AdaTron learning is a powerful algorithm for the learnable rule, it does not give the best possible generalization error for unlearnable problems. Optimization of the learning rate is shown to greatly improve the performance of the AdaTron algorithm, leading to the best possible generalization error for a wide range of the parameter that controls the shape of the transfer function.