Include: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou; Negar Foroutan; Anna Sotnikova; Zeming Chen; Sree Harsha Nelaturu; Shivalika Singh; Rishabh Maheshwary; Micol Altomare; Mohamed A. Haggag; A Snegha; Alfonso Amayuelas; Azril Hafizi Amirudin; Viraat Aryabumi; Danylo Boiko; Michael Chang; Jenny Chim; Gal Cohen; Aditya Kumar Dalmia; Abraham Diress; Sharad Duwal; Daniil Dzenhaliou; Daniel Fernando Erazo Florez; Fabian Farestam; Joseph Marvin Imperial; Shayekh Bin Islam; Perttu Isotalo; Maral Jabbarishiviari; Börje F. Karlsson; Eldar Khalilov; Christopher Klamm; Fajri Koto; Dominik Krzemiński; Gabriel Adriano de Melo; Syrielle Montariol; Yiyang Nan; Joel Niklaus; Johan Samir Obando Ceron; Debjit Paul; Esther Ploeger; Jebish Purbey; Swati Rajwal; Selvan Sunitha Ravi; Sara Rydell; Roshan Santhosh; Drishti Sharma; Marjana Prifti Skenduli; Arshia Soltani Moakhar; Bardia Soltani Moakhar; Ran Tamir; Ayush Kumar Tarun; Azmine Toushik Wasi; Thenuka Ovin Weerasinghe; Serhan Yilmaz; Mike Zhang; Imanol Schlag; Marzieh Fadaee; Sara Hooker; Antoine Bosselut

Include: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, A Snegha, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad DuwalDaniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut

Research output: Working paper / Preprint › Preprint

11 Downloads (Pure)

Abstract

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

Original language	English
Publisher	arXiv
Publication status	Published - 29 Nov 2024

Keywords

cs.CL

Access to Document

2411.19799v1

Cite this

Romanou, A, Foroutan, N, Sotnikova, A, Chen, Z, Nelaturu, SH, Singh, S, Maheshwary, R, Altomare, M, Haggag, MA, Snegha, A, Amayuelas, A, Amirudin, AH, Aryabumi, V, Boiko, D, Chang, M, Chim, J, Cohen, G, Dalmia, AK, Diress, A, Duwal, S, Dzenhaliou, D, Florez, DFE, Farestam, F, Imperial, JM, Islam, SB, Isotalo, P, Jabbarishiviari, M, Karlsson, BF, Khalilov, E, Klamm, C, Koto, F, Krzemiński, D, de Melo, GA, Montariol, S, Nan, Y, Niklaus, J, Ceron, JSO, Paul, D, Ploeger, E, Purbey, J, Rajwal, S, Ravi, SS, Rydell, S, Santhosh, R, Sharma, D, Skenduli, MP, Moakhar, AS, Moakhar, BS, Tamir, R, Tarun, AK, Wasi, AT, Weerasinghe, TO, Yilmaz, S, Zhang, M, Schlag, I, Fadaee, M, Hooker, S & Bosselut, A 2024 'Include: Evaluating Multilingual Language Understanding with Regional Knowledge' arXiv.

@techreport{f5ba8392c4c24cd8b8fdfb2a583fc02c,

title = "Include: Evaluating Multilingual Language Understanding with Regional Knowledge",

abstract = "The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.",

keywords = "cs.CL",

author = "Angelika Romanou and Negar Foroutan and Anna Sotnikova and Zeming Chen and Nelaturu, {Sree Harsha} and Shivalika Singh and Rishabh Maheshwary and Micol Altomare and Haggag, {Mohamed A.} and A Snegha and Alfonso Amayuelas and Amirudin, {Azril Hafizi} and Viraat Aryabumi and Danylo Boiko and Michael Chang and Jenny Chim and Gal Cohen and Dalmia, {Aditya Kumar} and Abraham Diress and Sharad Duwal and Daniil Dzenhaliou and Florez, {Daniel Fernando Erazo} and Fabian Farestam and Imperial, {Joseph Marvin} and Islam, {Shayekh Bin} and Perttu Isotalo and Maral Jabbarishiviari and Karlsson, {B{\"o}rje F.} and Eldar Khalilov and Christopher Klamm and Fajri Koto and Dominik Krzemi{\'n}ski and {de Melo}, {Gabriel Adriano} and Syrielle Montariol and Yiyang Nan and Joel Niklaus and Ceron, {Johan Samir Obando} and Debjit Paul and Esther Ploeger and Jebish Purbey and Swati Rajwal and Ravi, {Selvan Sunitha} and Sara Rydell and Roshan Santhosh and Drishti Sharma and Skenduli, {Marjana Prifti} and Moakhar, {Arshia Soltani} and Moakhar, {Bardia Soltani} and Ran Tamir and Tarun, {Ayush Kumar} and Wasi, {Azmine Toushik} and Weerasinghe, {Thenuka Ovin} and Serhan Yilmaz and Mike Zhang and Imanol Schlag and Marzieh Fadaee and Sara Hooker and Antoine Bosselut",

year = "2024",

month = nov,

day = "29",

language = "English",

publisher = "arXiv",

type = "WorkingPaper",

institution = "arXiv",

}

TY - UNPB

T1 - Include

T2 - Evaluating Multilingual Language Understanding with Regional Knowledge

AU - Romanou, Angelika

AU - Foroutan, Negar

AU - Sotnikova, Anna

AU - Chen, Zeming

AU - Nelaturu, Sree Harsha

AU - Singh, Shivalika

AU - Maheshwary, Rishabh

AU - Altomare, Micol

AU - Haggag, Mohamed A.

AU - Snegha, A

AU - Amayuelas, Alfonso

AU - Amirudin, Azril Hafizi

AU - Aryabumi, Viraat

AU - Boiko, Danylo

AU - Chang, Michael

AU - Chim, Jenny

AU - Cohen, Gal

AU - Dalmia, Aditya Kumar

AU - Diress, Abraham

AU - Duwal, Sharad

AU - Dzenhaliou, Daniil

AU - Florez, Daniel Fernando Erazo

AU - Farestam, Fabian

AU - Imperial, Joseph Marvin

AU - Islam, Shayekh Bin

AU - Isotalo, Perttu

AU - Jabbarishiviari, Maral

AU - Karlsson, Börje F.

AU - Khalilov, Eldar

AU - Klamm, Christopher

AU - Koto, Fajri

AU - Krzemiński, Dominik

AU - de Melo, Gabriel Adriano

AU - Montariol, Syrielle

AU - Nan, Yiyang

AU - Niklaus, Joel

AU - Ceron, Johan Samir Obando

AU - Paul, Debjit

AU - Ploeger, Esther

AU - Purbey, Jebish

AU - Rajwal, Swati

AU - Ravi, Selvan Sunitha

AU - Rydell, Sara

AU - Santhosh, Roshan

AU - Sharma, Drishti

AU - Skenduli, Marjana Prifti

AU - Moakhar, Arshia Soltani

AU - Moakhar, Bardia Soltani

AU - Tamir, Ran

AU - Tarun, Ayush Kumar

AU - Wasi, Azmine Toushik

AU - Weerasinghe, Thenuka Ovin

AU - Yilmaz, Serhan

AU - Zhang, Mike

AU - Schlag, Imanol

AU - Fadaee, Marzieh

AU - Hooker, Sara

AU - Bosselut, Antoine

PY - 2024/11/29

Y1 - 2024/11/29

N2 - The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

AB - The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

KW - cs.CL

M3 - Preprint

BT - Include

PB - arXiv

ER -

Include: Evaluating Multilingual Language Understanding with Regional Knowledge

Abstract

Keywords

Access to Document

Fingerprint

Cite this