A multilingual parallel corpora collection effort for Indian languages

Shashank Siripragada; Jerin Philip; Vinay P. Namboodiri; C. V. Jawahar

A multilingual parallel corpora collection effort for Indian languages

Shashank Siripragada, Jerin Philip, Vinay P. Namboodiri, C. V. Jawahar

Department of Computer Science

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

37 Citations (SciVal)

Abstract

We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.

Original language	English
Title of host publication	LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings
Editors	Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Publisher	European Language Resources Association (ELRA)
Pages	3743-3751
Number of pages	9
ISBN (Electronic)	9791095546344
Publication status	Published - 31 May 2020
Event	12th International Conference on Language Resources and Evaluation, LREC 2020 - Marseille, France Duration: 11 May 2020 → 16 May 2020

Publication series

Name	LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

Conference

Conference	12th International Conference on Language Resources and Evaluation, LREC 2020
Country/Territory	France
City	Marseille
Period	11/05/20 → 16/05/20

Bibliographical note

Funding Information:
We gratefully acknowledge the online corpora provided by ILCI, WAT-ILMPC, the online publicly available data that we have used in this work and the online open source tools that have facilitated this work.

Publisher Copyright:
© European Language Resources Association (ELRA), licensed under CC-BY-NC

Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.

Funding

We gratefully acknowledge the online corpora provided by ILCI, WAT-ILMPC, the online publicly available data that we have used in this work and the online open source tools that have facilitated this work.

Keywords

Indian languages
Machine Translation
Parallel Corpus

ASJC Scopus subject areas

Language and Linguistics
Education
Library and Information Sciences
Linguistics and Language

Access to Document

https://www.aclweb.org/anthology/2020.lrec-1.462Licence: CC BY

Cite this

Siripragada, S., Philip, J., Namboodiri, V. P., & Jawahar, C. V. (2020). A multilingual parallel corpora collection effort for Indian languages. In N. Calzolari, F. Bechet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings (pp. 3743-3751). (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings). European Language Resources Association (ELRA). https://www.aclweb.org/anthology/2020.lrec-1.462

A multilingual parallel corpora collection effort for Indian languages. / Siripragada, Shashank; Philip, Jerin; Namboodiri, Vinay P. et al.
LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. ed. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Asuncion Moreno; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2020. p. 3743-3751 (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Siripragada, S, Philip, J, Namboodiri, VP & Jawahar, CV 2020, A multilingual parallel corpora collection effort for Indian languages. in N Calzolari, F Bechet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk & S Piperidis (eds), LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings, European Language Resources Association (ELRA), pp. 3743-3751, 12th International Conference on Language Resources and Evaluation, LREC 2020, Marseille, France, 11/05/20. <https://www.aclweb.org/anthology/2020.lrec-1.462>

Siripragada S, Philip J, Namboodiri VP, Jawahar CV. A multilingual parallel corpora collection effort for Indian languages. In Calzolari N, Bechet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors, LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. European Language Resources Association (ELRA). 2020. p. 3743-3751. (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings).

Siripragada, Shashank ; Philip, Jerin ; Namboodiri, Vinay P. et al. / A multilingual parallel corpora collection effort for Indian languages. LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. editor / Nicoletta Calzolari ; Frederic Bechet ; Philippe Blache ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis. European Language Resources Association (ELRA), 2020. pp. 3743-3751 (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings).

@inproceedings{14f055d0b54a4e4ab1da69888c6d90dd,

title = "A multilingual parallel corpora collection effort for Indian languages",

abstract = "We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.",

keywords = "Indian languages, Machine Translation, Parallel Corpus",

author = "Shashank Siripragada and Jerin Philip and Namboodiri, {Vinay P.} and Jawahar, {C. V.}",

note = "Funding Information: We gratefully acknowledge the online corpora provided by ILCI, WAT-ILMPC, the online publicly available data that we have used in this work and the online open source tools that have facilitated this work. Publisher Copyright: {\textcopyright} European Language Resources Association (ELRA), licensed under CC-BY-NC Copyright: Copyright 2020 Elsevier B.V., All rights reserved.; 12th International Conference on Language Resources and Evaluation, LREC 2020 ; Conference date: 11-05-2020 Through 16-05-2020",

year = "2020",

month = may,

day = "31",

language = "English",

series = "LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings",

publisher = "European Language Resources Association (ELRA)",

pages = "3743--3751",

editor = "Nicoletta Calzolari and Frederic Bechet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis",

booktitle = "LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings",

}

TY - GEN

T1 - A multilingual parallel corpora collection effort for Indian languages

AU - Siripragada, Shashank

AU - Philip, Jerin

AU - Namboodiri, Vinay P.

AU - Jawahar, C. V.

N1 - Funding Information: We gratefully acknowledge the online corpora provided by ILCI, WAT-ILMPC, the online publicly available data that we have used in this work and the online open source tools that have facilitated this work. Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC Copyright: Copyright 2020 Elsevier B.V., All rights reserved.

PY - 2020/5/31

Y1 - 2020/5/31

N2 - We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.

AB - We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performance in 10 Indian languages. Alongside, we report on the methods of constructing such corpora using tools enabled by recent advances in machine translation and cross-lingual retrieval using deep neural network based methods.

KW - Indian languages

KW - Machine Translation

KW - Parallel Corpus

UR - http://www.scopus.com/inward/record.url?scp=85094320442&partnerID=8YFLogxK

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85094320442

T3 - LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

SP - 3743

EP - 3751

BT - LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

T2 - 12th International Conference on Language Resources and Evaluation, LREC 2020

Y2 - 11 May 2020 through 16 May 2020

ER -

A multilingual parallel corpora collection effort for Indian languages

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this