Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough?

Masato Mita; Tomoya Mizumoto; Masahiro Kaneko; Ryo Nagata; Kentaro Inui

Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough?

Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, Kentaro Inui

INF - System Information Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Citations (Scopus)

Abstract

This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the performance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models' rankings considerably vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models.

Original language	English
Title of host publication	Long and Short Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	1309-1314
Number of pages	6
ISBN (Electronic)	9781950737130
Publication status	Published - 2019
Event	2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 - Minneapolis, United States Duration: 2019 Jun 2 → 2019 Jun 7

Publication series

Name	NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume	1

Conference

Conference	2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019
Country/Territory	United States
City	Minneapolis
Period	19/6/2 → 19/6/7

Cite this

Mita, M., Mizumoto, T., Kaneko, M., Nagata, R., & Inui, K. (2019). Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough? In Long and Short Papers (pp. 1309-1314). (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1). Association for Computational Linguistics (ACL).

Mita, Masato ; Mizumoto, Tomoya ; Kaneko, Masahiro et al. / Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough?. Long and Short Papers. Association for Computational Linguistics (ACL), 2019. pp. 1309-1314 (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference).

@inproceedings{2a4ac3f741bf465ba618ca64488dc8b5,

title = "Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough?",

abstract = "This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the performance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models' rankings considerably vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models.",

author = "Masato Mita and Tomoya Mizumoto and Masahiro Kaneko and Ryo Nagata and Kentaro Inui",

note = "Funding Information: We are grateful to the members of the Tohoku University Natural Language Processing Laboratory as well as the anonymous reviewers for their insightful comments and suggestions. Publisher Copyright: {\textcopyright} 2019 Association for Computational Linguistics; 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 ; Conference date: 02-06-2019 Through 07-06-2019",

year = "2019",

language = "English",

series = "NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1309--1314",

booktitle = "Long and Short Papers",

address = "United States",

}

Mita, M, Mizumoto, T, Kaneko, M, Nagata, R & Inui, K 2019, Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough? in Long and Short Papers. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, Association for Computational Linguistics (ACL), pp. 1309-1314, 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019, Minneapolis, United States, 19/6/2.

Cross-corpora evaluation and analysis of grammatical error correction models - Is single-corpus evaluation enough? / Mita, Masato; Mizumoto, Tomoya; Kaneko, Masahiro et al.
Long and Short Papers. Association for Computational Linguistics (ACL), 2019. p. 1309-1314 (NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review