SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Holy Lovenia; Rahmad Mahendra; Salsabil Maulana Akbar; Lester James V. Miranda; Jennifer Santoso; Elyanah Aco; Akhdan Fadhilah; Jonibek Mansurov; Joseph Marvin Imperial; Onno P. Kampman; Joel Ruben Antony Moniz; Muhammad Ravi Shulthan Habibi; Frederikus Hudi; Railey Montalan; Ryan Ignatius; Joanito Agili Lopo; William Nixon; Börje F. Karlsson; James Jaya; Ryandito Diandaru; Yuze Gao; Patrick Amadeus; Bin Wang; Jan Christian Blaise Cruz; Chenxi Whitehouse; Ivan Halim Parmonangan; Maria Khelli; Wenyu Zhang; Lucky Susanto; Reynard Adha Ryanda; Sonny Lazuardi Hermawan; Dan John Velasco; Muhammad Dehan Al Kautsar; Willy Fitra Hendria; Yasmin Moslem; Noah Flynn; Muhammad Farid Adilazuarda; Haochen Li; Johanes Lee; R. Damanhuri; Shuo Sun; Muhammad Reza Qorib; Amirbek Djanibekov; Wei Qi Leong; Quyet V. Do; Niklas Muennighoff; Tanrada Pansuwan; Ilham Firdausi Putra; Yan Xu; Ngee Chia Tai; Ayu Purwarianti; Sebastian Ruder; William Tjhi; Peerat Limkonchotiwat; Alham Fikri Aji; Sedrick Keh; Genta Indra Winata; Ruochen Zhang; Fajri Koto; Zheng Xin Yong; Samuel Cahyawijaya

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, Ryandito DiandaruYuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Ngee Chia Tai, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, Samuel Cahyawijaya

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

1 Citation (SciVal)

Abstract

Southeast Asia (SEA) is a region characterized by rich linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, the performance of contemporary AI models for SEA languages is compromised by a significant lack of representation of texts, images, and auditory datasets from SEA. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the predominance of English training data, which raises concerns regarding potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub to bridge the resource gap by providing standardized corpora and benchmarks in nearly 1,000 SEA languages across three modalities. We assess the performance of AI models on 36 indigenous languages across 13 tasks included in SEACrowd, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.

Original language	English
Title of host publication	EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
Editors	Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Publisher	Association for Computational Linguistics (ACL)
Pages	5155-5203
Number of pages	49
ISBN (Electronic)	9798891761643
Publication status	Published - 16 Nov 2024
Event	2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024 - Hybrid, Miami, USA United States Duration: 12 Nov 2024 → 16 Nov 2024

Publication series

Name	EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Conference

Conference	2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024
Country/Territory	USA United States
City	Hybrid, Miami
Period	12/11/24 → 16/11/24

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems
Linguistics and Language

Access to Document

https://aclanthology.org/2024.emnlp-main.296/Licence: CC BY

Cite this

Lovenia, H., Mahendra, R., Akbar, S. M., Miranda, L. J. V., Santoso, J., Aco, E., Fadhilah, A., Mansurov, J., Imperial, J. M., Kampman, O. P., Moniz, J. R. A., Habibi, M. R. S., Hudi, F., Montalan, R., Ignatius, R., Lopo, J. A., Nixon, W., Karlsson, B. F., Jaya, J., ... Cahyawijaya, S. (2024). SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference (pp. 5155-5203). (EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference). Association for Computational Linguistics (ACL). https://aclanthology.org/2024.emnlp-main.296/

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages. / Lovenia, Holy; Mahendra, Rahmad; Akbar, Salsabil Maulana et al.
EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. ed. / Yaser Al-Onaizan; Mohit Bansal; Yun-Nung Chen. Association for Computational Linguistics (ACL), 2024. p. 5155-5203 (EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Lovenia, H, Mahendra, R, Akbar, SM, Miranda, LJV, Santoso, J, Aco, E, Fadhilah, A, Mansurov, J, Imperial, JM, Kampman, OP, Moniz, JRA, Habibi, MRS, Hudi, F, Montalan, R, Ignatius, R, Lopo, JA, Nixon, W, Karlsson, BF, Jaya, J, Diandaru, R, Gao, Y, Amadeus, P, Wang, B, Cruz, JCB, Whitehouse, C, Parmonangan, IH, Khelli, M, Zhang, W, Susanto, L, Ryanda, RA, Hermawan, SL, Velasco, DJ, Al Kautsar, MD, Hendria, WF, Moslem, Y, Flynn, N, Adilazuarda, MF, Li, H, Lee, J, Damanhuri, R, Sun, S, Qorib, MR, Djanibekov, A, Leong, WQ, Do, QV, Muennighoff, N, Pansuwan, T, Putra, IF, Xu, Y, Tai, NC, Purwarianti, A, Ruder, S, Tjhi, W, Limkonchotiwat, P, Aji, AF, Keh, S, Winata, GI, Zhang, R, Koto, F, Yong, ZX & Cahyawijaya, S 2024, SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages. in Y Al-Onaizan, M Bansal & Y-N Chen (eds), EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 5155-5203, 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Hybrid, Miami, USA United States, 12/11/24. <https://aclanthology.org/2024.emnlp-main.296/>

Lovenia H, Mahendra R, Akbar SM, Miranda LJV, Santoso J, Aco E et al. SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages. In Al-Onaizan Y, Bansal M, Chen YN, editors, EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2024. p. 5155-5203. (EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference).

Lovenia, Holy ; Mahendra, Rahmad ; Akbar, Salsabil Maulana et al. / SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages. EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. editor / Yaser Al-Onaizan ; Mohit Bansal ; Yun-Nung Chen. Association for Computational Linguistics (ACL), 2024. pp. 5155-5203 (EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference).

@inproceedings{dff24f8a8cf1468c951f03bfef4efc2b,

title = "SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages",

abstract = "Southeast Asia (SEA) is a region characterized by rich linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, the performance of contemporary AI models for SEA languages is compromised by a significant lack of representation of texts, images, and auditory datasets from SEA. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the predominance of English training data, which raises concerns regarding potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub to bridge the resource gap by providing standardized corpora and benchmarks in nearly 1,000 SEA languages across three modalities. We assess the performance of AI models on 36 indigenous languages across 13 tasks included in SEACrowd, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.",

author = "Holy Lovenia and Rahmad Mahendra and Akbar, {Salsabil Maulana} and Miranda, {Lester James V.} and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Imperial, {Joseph Marvin} and Kampman, {Onno P.} and Moniz, {Joel Ruben Antony} and Habibi, {Muhammad Ravi Shulthan} and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Lopo, {Joanito Agili} and William Nixon and Karlsson, {B{\"o}rje F.} and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Cruz, {Jan Christian Blaise} and Chenxi Whitehouse and Parmonangan, {Ivan Halim} and Maria Khelli and Wenyu Zhang and Lucky Susanto and Ryanda, {Reynard Adha} and Hermawan, {Sonny Lazuardi} and Velasco, {Dan John} and {Al Kautsar}, {Muhammad Dehan} and Hendria, {Willy Fitra} and Yasmin Moslem and Noah Flynn and Adilazuarda, {Muhammad Farid} and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Qorib, {Muhammad Reza} and Amirbek Djanibekov and Leong, {Wei Qi} and Do, {Quyet V.} and Niklas Muennighoff and Tanrada Pansuwan and Putra, {Ilham Firdausi} and Yan Xu and Tai, {Ngee Chia} and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Aji, {Alham Fikri} and Sedrick Keh and Winata, {Genta Indra} and Ruochen Zhang and Fajri Koto and Yong, {Zheng Xin} and Samuel Cahyawijaya",

year = "2024",

month = nov,

day = "16",

language = "English",

series = "EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "5155--5203",

editor = "Yaser Al-Onaizan and Mohit Bansal and Yun-Nung Chen",

booktitle = "EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference",

note = "2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024 ; Conference date: 12-11-2024 Through 16-11-2024",

}

TY - GEN

T1 - SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

AU - Lovenia, Holy

AU - Mahendra, Rahmad

AU - Akbar, Salsabil Maulana

AU - Miranda, Lester James V.

AU - Santoso, Jennifer

AU - Aco, Elyanah

AU - Fadhilah, Akhdan

AU - Mansurov, Jonibek

AU - Imperial, Joseph Marvin

AU - Kampman, Onno P.

AU - Moniz, Joel Ruben Antony

AU - Habibi, Muhammad Ravi Shulthan

AU - Hudi, Frederikus

AU - Montalan, Railey

AU - Ignatius, Ryan

AU - Lopo, Joanito Agili

AU - Nixon, William

AU - Karlsson, Börje F.

AU - Jaya, James

AU - Diandaru, Ryandito

AU - Gao, Yuze

AU - Amadeus, Patrick

AU - Wang, Bin

AU - Cruz, Jan Christian Blaise

AU - Whitehouse, Chenxi

AU - Parmonangan, Ivan Halim

AU - Khelli, Maria

AU - Zhang, Wenyu

AU - Susanto, Lucky

AU - Ryanda, Reynard Adha

AU - Hermawan, Sonny Lazuardi

AU - Velasco, Dan John

AU - Al Kautsar, Muhammad Dehan

AU - Hendria, Willy Fitra

AU - Moslem, Yasmin

AU - Flynn, Noah

AU - Adilazuarda, Muhammad Farid

AU - Li, Haochen

AU - Lee, Johanes

AU - Damanhuri, R.

AU - Sun, Shuo

AU - Qorib, Muhammad Reza

AU - Djanibekov, Amirbek

AU - Leong, Wei Qi

AU - Do, Quyet V.

AU - Muennighoff, Niklas

AU - Pansuwan, Tanrada

AU - Putra, Ilham Firdausi

AU - Xu, Yan

AU - Tai, Ngee Chia

AU - Purwarianti, Ayu

AU - Ruder, Sebastian

AU - Tjhi, William

AU - Limkonchotiwat, Peerat

AU - Aji, Alham Fikri

AU - Keh, Sedrick

AU - Winata, Genta Indra

AU - Zhang, Ruochen

AU - Koto, Fajri

AU - Yong, Zheng Xin

AU - Cahyawijaya, Samuel

PY - 2024/11/16

Y1 - 2024/11/16

N2 - Southeast Asia (SEA) is a region characterized by rich linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, the performance of contemporary AI models for SEA languages is compromised by a significant lack of representation of texts, images, and auditory datasets from SEA. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the predominance of English training data, which raises concerns regarding potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub to bridge the resource gap by providing standardized corpora and benchmarks in nearly 1,000 SEA languages across three modalities. We assess the performance of AI models on 36 indigenous languages across 13 tasks included in SEACrowd, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.

AB - Southeast Asia (SEA) is a region characterized by rich linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, the performance of contemporary AI models for SEA languages is compromised by a significant lack of representation of texts, images, and auditory datasets from SEA. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the predominance of English training data, which raises concerns regarding potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub to bridge the resource gap by providing standardized corpora and benchmarks in nearly 1,000 SEA languages across three modalities. We assess the performance of AI models on 36 indigenous languages across 13 tasks included in SEACrowd, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.

UR - http://www.scopus.com/inward/record.url?scp=85213125303&partnerID=8YFLogxK

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85213125303

T3 - EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

SP - 5155

EP - 5203

BT - EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

A2 - Al-Onaizan, Yaser

A2 - Bansal, Mohit

A2 - Chen, Yun-Nung

PB - Association for Computational Linguistics (ACL)

T2 - 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024

Y2 - 12 November 2024 through 16 November 2024

ER -

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this