LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023

Rabab Alkhalifa; Iman Bilal; Hsuvas Borkakoty; Jose Camacho-Collados; Romain Deveaud; Alaa El-Ebshihy; Luis Espinosa-Anke; Gabriela Gonzalez-Saez; Petra Galuščáková; Lorraine Goeuriot; Elena Kochkina; Maria Liakata; Daniel Loureiro; Harish Tayyar Madabushi; Philippe Mulhem; Florina Piroi; Martin Popel; Christophe Servan; Arkaitz Zubiaga

doi:10.1007/978-3-031-28241-6_58

LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023

Rabab Alkhalifa, Iman Bilal, Hsuvas Borkakoty, Jose Camacho-Collados, Romain Deveaud, Alaa El-Ebshihy, Luis Espinosa-Anke, Gabriela Gonzalez-Saez, Petra Galuščáková, Lorraine Goeuriot, Elena Kochkina, Maria Liakata, Daniel Loureiro, Harish Tayyar Madabushi, Philippe Mulhem, Florina Piroi, Martin Popel, Christophe Servan, Arkaitz Zubiaga

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

6 Citations (SciVal)

41 Downloads (Pure)

Abstract

In this paper, we describe the plans for the first LongEval CLEF 2023 shared task dedicated to evaluating the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. The task is motivated by recent research showing that the performance of these models drops as the test data becomes more distant, with respect to time, from the training data. LongEval differs from traditional shared IR and classification tasks by giving special consideration to evaluating models aiming to mitigate performance drop over time. We envisage that this task will draw attention from the IR community and NLP researchers to the problem of temporal persistence of models, what enables or prevents it, potential solutions and their limitations.

Original language	English
Title of host publication	Advances in Information Retrieval
Editors	Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Annalina Caputo, Udo Kruschwitz
Place of Publication	Cham, Switzerland
Publisher	Springer Nature Switzerland
Pages	499-505
Number of pages	7
ISBN (Electronic)	9783031282416
ISBN (Print)	9783031282409
DOIs	https://doi.org/10.1007/978-3-031-28241-6_58
Publication status	Published - 2 Apr 2023
Event	45th European Conference on Information Retrieval, ECIR 2023 - Dublin, Ireland Duration: 2 Apr 2023 → 6 Apr 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13982 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	45th European Conference on Information Retrieval, ECIR 2023
Country/Territory	Ireland
City	Dublin
Period	2/04/23 → 6/04/23

Funding

Acknowledgements. This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1) and The Alan Turing Institute (grant no. EP/N510129/1) through project funding and its Enrichment PhD Scheme for Iman Bilal. This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2018101) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

Keywords

Evaluation
Information retrieval
Temporal generalisability
Temporal persistence
Text classification

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-28241-6_58

alkhalifa2023longevalAccepted author manuscript, 223 KB

Cite this

Alkhalifa, R., Bilal, I., Borkakoty, H., Camacho-Collados, J., Deveaud, R., El-Ebshihy, A., Espinosa-Anke, L., Gonzalez-Saez, G., Galuščáková, P., Goeuriot, L., Kochkina, E., Liakata, M., Loureiro, D., Tayyar Madabushi, H., Mulhem, P., Piroi, F., Popel, M., Servan, C., & Zubiaga, A. (2023). LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023. In J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, A. Caputo, & U. Kruschwitz (Eds.), Advances in Information Retrieval (pp. 499-505). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13982 LNCS). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-28241-6_58

LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023. / Alkhalifa, Rabab; Bilal, Iman; Borkakoty, Hsuvas et al.
Advances in Information Retrieval. ed. / Jaap Kamps; Lorraine Goeuriot; Fabio Crestani; Maria Maistro; Hideo Joho; Brian Davis; Cathal Gurrin; Annalina Caputo; Udo Kruschwitz. Cham, Switzerland: Springer Nature Switzerland, 2023. p. 499-505 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13982 LNCS).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Alkhalifa, R, Bilal, I, Borkakoty, H, Camacho-Collados, J, Deveaud, R, El-Ebshihy, A, Espinosa-Anke, L, Gonzalez-Saez, G, Galuščáková, P, Goeuriot, L, Kochkina, E, Liakata, M, Loureiro, D, Tayyar Madabushi, H, Mulhem, P, Piroi, F, Popel, M, Servan, C & Zubiaga, A 2023, LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023. in J Kamps, L Goeuriot, F Crestani, M Maistro, H Joho, B Davis, C Gurrin, A Caputo & U Kruschwitz (eds), Advances in Information Retrieval. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13982 LNCS, Springer Nature Switzerland, Cham, Switzerland, pp. 499-505, 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, 2/04/23. https://doi.org/10.1007/978-3-031-28241-6_58

Alkhalifa R, Bilal I, Borkakoty H, Camacho-Collados J, Deveaud R, El-Ebshihy A et al. LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023. In Kamps J, Goeuriot L, Crestani F, Maistro M, Joho H, Davis B, Gurrin C, Caputo A, Kruschwitz U, editors, Advances in Information Retrieval. Cham, Switzerland: Springer Nature Switzerland. 2023. p. 499-505. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2023 Mar 16. doi: 10.1007/978-3-031-28241-6_58

Alkhalifa, Rabab ; Bilal, Iman ; Borkakoty, Hsuvas et al. / LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023. Advances in Information Retrieval. editor / Jaap Kamps ; Lorraine Goeuriot ; Fabio Crestani ; Maria Maistro ; Hideo Joho ; Brian Davis ; Cathal Gurrin ; Annalina Caputo ; Udo Kruschwitz. Cham, Switzerland : Springer Nature Switzerland, 2023. pp. 499-505 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9997a1ecb1d64e9a9550c9b30eaa9419,

title = "LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023",

abstract = "In this paper, we describe the plans for the first LongEval CLEF 2023 shared task dedicated to evaluating the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. The task is motivated by recent research showing that the performance of these models drops as the test data becomes more distant, with respect to time, from the training data. LongEval differs from traditional shared IR and classification tasks by giving special consideration to evaluating models aiming to mitigate performance drop over time. We envisage that this task will draw attention from the IR community and NLP researchers to the problem of temporal persistence of models, what enables or prevents it, potential solutions and their limitations.",

keywords = "Evaluation, Information retrieval, Temporal generalisability, Temporal persistence, Text classification",

author = "Rabab Alkhalifa and Iman Bilal and Hsuvas Borkakoty and Jose Camacho-Collados and Romain Deveaud and Alaa El-Ebshihy and Luis Espinosa-Anke and Gabriela Gonzalez-Saez and Petra Galu{\v s}{\v c}{\'a}kov{\'a} and Lorraine Goeuriot and Elena Kochkina and Maria Liakata and Daniel Loureiro and {Tayyar Madabushi}, Harish and Philippe Mulhem and Florina Piroi and Martin Popel and Christophe Servan and Arkaitz Zubiaga",

year = "2023",

month = apr,

day = "2",

doi = "10.1007/978-3-031-28241-6_58",

language = "English",

isbn = "9783031282409",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Nature Switzerland",

pages = "499--505",

editor = "Jaap Kamps and Lorraine Goeuriot and Fabio Crestani and Maria Maistro and Hideo Joho and Brian Davis and Cathal Gurrin and Annalina Caputo and Udo Kruschwitz",

booktitle = "Advances in Information Retrieval",

note = "45th European Conference on Information Retrieval, ECIR 2023 ; Conference date: 02-04-2023 Through 06-04-2023",

}

TY - GEN

T1 - LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023

AU - Alkhalifa, Rabab

AU - Bilal, Iman

AU - Borkakoty, Hsuvas

AU - Camacho-Collados, Jose

AU - Deveaud, Romain

AU - El-Ebshihy, Alaa

AU - Espinosa-Anke, Luis

AU - Gonzalez-Saez, Gabriela

AU - Galuščáková, Petra

AU - Goeuriot, Lorraine

AU - Kochkina, Elena

AU - Liakata, Maria

AU - Loureiro, Daniel

AU - Tayyar Madabushi, Harish

AU - Mulhem, Philippe

AU - Piroi, Florina

AU - Popel, Martin

AU - Servan, Christophe

AU - Zubiaga, Arkaitz

PY - 2023/4/2

Y1 - 2023/4/2

N2 - In this paper, we describe the plans for the first LongEval CLEF 2023 shared task dedicated to evaluating the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. The task is motivated by recent research showing that the performance of these models drops as the test data becomes more distant, with respect to time, from the training data. LongEval differs from traditional shared IR and classification tasks by giving special consideration to evaluating models aiming to mitigate performance drop over time. We envisage that this task will draw attention from the IR community and NLP researchers to the problem of temporal persistence of models, what enables or prevents it, potential solutions and their limitations.

AB - In this paper, we describe the plans for the first LongEval CLEF 2023 shared task dedicated to evaluating the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. The task is motivated by recent research showing that the performance of these models drops as the test data becomes more distant, with respect to time, from the training data. LongEval differs from traditional shared IR and classification tasks by giving special consideration to evaluating models aiming to mitigate performance drop over time. We envisage that this task will draw attention from the IR community and NLP researchers to the problem of temporal persistence of models, what enables or prevents it, potential solutions and their limitations.

KW - Evaluation

KW - Information retrieval

KW - Temporal generalisability

KW - Temporal persistence

KW - Text classification

UR - http://www.scopus.com/inward/record.url?scp=85151052865&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-28241-6_58

DO - 10.1007/978-3-031-28241-6_58

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85151052865

SN - 9783031282409

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 499

EP - 505

BT - Advances in Information Retrieval

A2 - Kamps, Jaap

A2 - Goeuriot, Lorraine

A2 - Crestani, Fabio

A2 - Maistro, Maria

A2 - Joho, Hideo

A2 - Davis, Brian

A2 - Gurrin, Cathal

A2 - Caputo, Annalina

A2 - Kruschwitz, Udo

PB - Springer Nature Switzerland

CY - Cham, Switzerland

T2 - 45th European Conference on Information Retrieval, ECIR 2023

Y2 - 2 April 2023 through 6 April 2023

ER -

LongEval: Longitudinal Evaluation of Model Performance at CLEF 2023

Abstract

Publication series

Conference

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this