Subword-Based Compact Reconstruction for Open-Vocabulary Neural Word Embeddings

Shota Sasaki, Jun Suzuki, Kentaro Inui

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


The methodology of neural word embeddings has become an important fundamental resource for tackling many applications in the artificial intelligence (AI) research field. They have successfully been proven to capture high-quality syntactic and semantic relationships in a vector space. Despite their significant impact, neural word embeddings have several disadvantages. In this paper, we focus on two issues regarding well-trained word embeddings: (i) the massive memory requirement and (ii) the inapplicability of out-of-vocabulary (OOV) words. To overcome these two issues, we propose a method of reconstructing pre-trained word embeddings by using subword information that effectively represents a large number of subword embeddings in a considerably small fixed space while preventing quality degradation from the original word embeddings. The key techniques of our method are twofold: memory-shared embeddings and a variant of the key-value-query self-attention mechanism. Our experiments show that our reconstructed subword-based word embeddings can successfully imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets and can simultaneously predict effective embeddings of OOV words. We also demonstrate the effectiveness of our reconstruction method when it is applied to downstream tasks, such as named entity recognition and natural language inference tasks.

Original languageEnglish
Pages (from-to)3551-3564
Number of pages14
JournalIEEE/ACM Transactions on Audio Speech and Language Processing
Publication statusPublished - 2021
Externally publishedYes


  • Neural word embeddings
  • open vocabulary
  • subwords

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Acoustics and Ultrasonics
  • Computational Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Subword-Based Compact Reconstruction for Open-Vocabulary Neural Word Embeddings'. Together they form a unique fingerprint.

Cite this