Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance

Rabab Alkhalifa; Iman Bilal; Hsuvas Borkakoty; Jose Camacho-Collados; Romain Deveaud; Alaa El-Ebshihy; Luis Espinosa-Anke; Gabriela Gonzalez-Saez; Petra Galuščáková; Lorraine Goeuriot; Elena Kochkina; Maria Liakata; Daniel Loureiro; Philippe Mulhem; Florina Piroi; Martin Popel; Christophe Servan; Harish Tayyar Madabushi; Arkaitz Zubiaga

Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance

Rabab Alkhalifa, Iman Bilal, Hsuvas Borkakoty, Jose Camacho-Collados, Romain Deveaud, Alaa El-Ebshihy, Luis Espinosa-Anke, Gabriela Gonzalez-Saez, Petra Galuščáková, Lorraine Goeuriot, Elena Kochkina, Maria Liakata, Daniel Loureiro, Philippe Mulhem, Florina Piroi, Martin Popel, Christophe Servan, Harish Tayyar Madabushi, Arkaitz Zubiaga

Research output: Contribution to journal › Conference article › peer-review

2 Citations (SciVal)

80 Downloads (Pure)

Abstract

We describe the first edition of the LongEval CLEF 2023 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

Original language	English
Pages (from-to)	2181-2203
Number of pages	23
Journal	CEUR Workshop Proceedings
Volume	3497
Publication status	Published - 21 Sept 2023
Event	24th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF-WN 2023 - Thessaloniki, Greece Duration: 18 Sept 2023 → 21 Sept 2023

Bibliographical note

Funding Information:
This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1) and The Alan Turing Institute (grant no. EP/N510129/1) through project funding and its Enrichment PhD Scheme for Iman Bilal. This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz),

Funding Information:
supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

Keywords

Evaluation
Temporal Generalisability
Temporal Persistence

ASJC Scopus subject areas

General Computer Science

Access to Document

paper-184Final published version, 1.59 MBLicence: CC BY

https://ceur-ws.org/Vol-3497/Licence: CC BY

Cite this

Alkhalifa, R., Bilal, I., Borkakoty, H., Camacho-Collados, J., Deveaud, R., El-Ebshihy, A., Espinosa-Anke, L., Gonzalez-Saez, G., Galuščáková, P., Goeuriot, L., Kochkina, E., Liakata, M., Loureiro, D., Mulhem, P., Piroi, F., Popel, M., Servan, C., Madabushi, H. T., & Zubiaga, A. (2023). Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance. CEUR Workshop Proceedings, 3497, 2181-2203. https://ceur-ws.org/Vol-3497/

Alkhalifa, R, Bilal, I, Borkakoty, H, Camacho-Collados, J, Deveaud, R, El-Ebshihy, A, Espinosa-Anke, L, Gonzalez-Saez, G, Galuščáková, P, Goeuriot, L, Kochkina, E, Liakata, M, Loureiro, D, Mulhem, P, Piroi, F, Popel, M, Servan, C, Madabushi, HT & Zubiaga, A 2023, 'Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance', CEUR Workshop Proceedings, vol. 3497, pp. 2181-2203. <https://ceur-ws.org/Vol-3497/>

@article{0de016760de8473d8e83cf99756cd83d,

title = "Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance",

abstract = "We describe the first edition of the LongEval CLEF 2023 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.",

keywords = "Evaluation, Temporal Generalisability, Temporal Persistence",

author = "Rabab Alkhalifa and Iman Bilal and Hsuvas Borkakoty and Jose Camacho-Collados and Romain Deveaud and Alaa El-Ebshihy and Luis Espinosa-Anke and Gabriela Gonzalez-Saez and Petra Galu{\v s}{\v c}{\'a}kov{\'a} and Lorraine Goeuriot and Elena Kochkina and Maria Liakata and Daniel Loureiro and Philippe Mulhem and Florina Piroi and Martin Popel and Christophe Servan and Madabushi, {Harish Tayyar} and Arkaitz Zubiaga",

note = "Funding Information: This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1) and The Alan Turing Institute (grant no. EP/N510129/1) through project funding and its Enrichment PhD Scheme for Iman Bilal. This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), Funding Information: supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ. ; 24th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF-WN 2023 ; Conference date: 18-09-2023 Through 21-09-2023",

year = "2023",

month = sep,

day = "21",

language = "English",

volume = "3497",

pages = "2181--2203",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

}

TY - JOUR

T1 - Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance

AU - Alkhalifa, Rabab

AU - Bilal, Iman

AU - Borkakoty, Hsuvas

AU - Camacho-Collados, Jose

AU - Deveaud, Romain

AU - El-Ebshihy, Alaa

AU - Espinosa-Anke, Luis

AU - Gonzalez-Saez, Gabriela

AU - Galuščáková, Petra

AU - Goeuriot, Lorraine

AU - Kochkina, Elena

AU - Liakata, Maria

AU - Loureiro, Daniel

AU - Mulhem, Philippe

AU - Piroi, Florina

AU - Popel, Martin

AU - Servan, Christophe

AU - Madabushi, Harish Tayyar

AU - Zubiaga, Arkaitz

N1 - Funding Information: This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1) and The Alan Turing Institute (grant no. EP/N510129/1) through project funding and its Enrichment PhD Scheme for Iman Bilal. This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), Funding Information: supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

PY - 2023/9/21

Y1 - 2023/9/21

N2 - We describe the first edition of the LongEval CLEF 2023 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

AB - We describe the first edition of the LongEval CLEF 2023 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

KW - Evaluation

KW - Temporal Generalisability

KW - Temporal Persistence

UR - http://www.scopus.com/inward/record.url?scp=85175649590&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85175649590

SN - 1613-0073

VL - 3497

SP - 2181

EP - 2203

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - 24th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF-WN 2023

Y2 - 18 September 2023 through 21 September 2023

ER -

Extended Overview of the CLEF-2023 LongEval Lab on Longitudinal Evaluation of Model Performance

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this