TY - GEN
T1 - Unsupervised Token-wise Alignment to Improve Interpretation of Encoder-Decoder Models
AU - Kiyono, Shun
AU - Takase, Sho
AU - Suzuki, Jun
AU - Okazaki, Naoaki
AU - Inui, Kentaro
AU - Nagata, Masaaki
N1 - Funding Information:
This work is a product of collaborative research program of Tohoku University and NTT Communication Science Laboratories. We are grateful to anonymous reviewers for their insightful comments. We thank Sosuke Kobayashi for providing helpful comments. We also thank Qingyu Zhou for providing a dataset and information for a fair comparison.
Publisher Copyright:
© 2018 Association for Computational Linguistics
PY - 2018
Y1 - 2018
N2 - Developing a method for understanding the inner workings of black-box neural methods is an important research endeavor. Conventionally, many studies have used an attention matrix to interpret how Encoder-Decoder-based models translate a given source sentence to the corresponding target sentence. However, recent studies have empirically revealed that an attention matrix is not optimal for token-wise translation analyses. We propose a method that explicitly models the token-wise alignment between the source and target sequences to provide a better analysis. Experiments show that our method can acquire token-wise alignments that are superior to those of an attention mechanism.
AB - Developing a method for understanding the inner workings of black-box neural methods is an important research endeavor. Conventionally, many studies have used an attention matrix to interpret how Encoder-Decoder-based models translate a given source sentence to the corresponding target sentence. However, recent studies have empirically revealed that an attention matrix is not optimal for token-wise translation analyses. We propose a method that explicitly models the token-wise alignment between the source and target sequences to provide a better analysis. Experiments show that our method can acquire token-wise alignments that are superior to those of an attention mechanism.
UR - http://www.scopus.com/inward/record.url?scp=85082307481&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85082307481&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85082307481
T3 - EMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Proceedings of the 1st Workshop
SP - 74
EP - 81
BT - EMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP
PB - Association for Computational Linguistics (ACL)
T2 - 1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Y2 - 1 November 2018
ER -