A Weight Ternary Ensemble Vision Transformer Toward Memory Size Reduction

Vision Transformer (ViT) is an image recognition model with advanced accuracy. However, there is a problem with the large parameter size. A ternary representation that introduces zero to the binary representation of {-1, + 1} can be applied to reduce the parameter size. We define it as weight ternary ViT, Although the weight ternary ViT can reduce the parameter size, the recognition accuracy decreases compared to ViT with float32 accuracy. Therefore, we introduce an ensemble method that computes multiple weight ternary ViTs in parallel. Recognition accuracy can be improved by majority voting through ensemble formation. In this paper, we propose a method for adding a normalization layer to advance the training of a weight ternary ViT and a training algorithm. Next, we will describe the training method of an ensemble weight ternary ViT. We evaluate the ensemble weight ternary ViT using CIFAR10 benchmark images, an image classification task. As a result, we achieve recognition accuracy equivalent to the original float32 precision ViT by using ten weight ternary ViTs with 80% zero representation applied. The parameter size is reduced by 87.5%.