Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model

Iiro Rastas; Yann Ryan; Iiro Tiihonen; Mohammadreza Qaraei; Liina Repo; Rohit Babbar; Eetu Mäkelä; Mikko Tolonen; Filip Ginter

doi:10.18653/v1/2022.lchange-1.7

Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model

Iiro Rastas, Yann Ryan, Iiro Tiihonen, Mohammadreza Qaraei, Liina Repo, Rohit Babbar, Eetu Mäkelä, Mikko Tolonen, Filip Ginter

Department of Computer Science

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

14 Citations (SciVal)

Abstract

In this paper, we describe a BERT model trained on the Eighteenth Century Collections Online (ECCO) dataset of digitized documents. The ECCO dataset poses unique modelling challenges due to the presence of Optical Character Recognition (OCR) artifacts. We establish the performance of the BERT model on a publication year prediction task against linear baseline models and human judgement, finding the BERT model to be superior to both and able to date the works, on average, with less than 7 years absolute error. We also explore how language change over time affects the model by analyzing the features the model uses for publication year predictions as given by the Integrated Gradients model explanation method.

Original language	English
Title of host publication	LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop
Editors	Nina Tahmasebi, Syrielle Montariol, Andrey Kutuzov, Simon Hengchen, Haim Dubossarsky, Lars Borin
Publisher	Association for Computational Linguistics (ACL)
Pages	68-77
Number of pages	10
ISBN (Electronic)	9781955917421
DOIs	https://doi.org/10.18653/v1/2022.lchange-1.7
Publication status	Published - 25 May 2022
Event	3rd International Workshop on Computational Approaches to Historical Language Change, LChange 2022 - Dublin, Ireland Duration: 26 May 2022 → 27 May 2022

Publication series

Name	LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop

Conference

Conference	3rd International Workshop on Computational Approaches to Historical Language Change, LChange 2022
Country/Territory	Ireland
City	Dublin
Period	26/05/22 → 27/05/22

Bibliographical note

Funding Information:
The research was supported by the Academy of Finland under the project High Performance Computing for the Detection and Analysis of Historical Discourses. Computational resources were provided by CSC – IT Center for Science.

Funding

The research was supported by the Academy of Finland under the project High Performance Computing for the Detection and Analysis of Historical Discourses. Computational resources were provided by CSC – IT Center for Science.

ASJC Scopus subject areas

Language and Linguistics
Computational Theory and Mathematics
Computer Science Applications
Software

Access to Document

10.18653/v1/2022.lchange-1.7Licence: CC BY

Cite this

Rastas, I., Ryan, Y., Tiihonen, I., Qaraei, M., Repo, L., Babbar, R., Mäkelä, E., Tolonen, M., & Ginter, F. (2022). Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model. In N. Tahmasebi, S. Montariol, A. Kutuzov, S. Hengchen, H. Dubossarsky, & L. Borin (Eds.), LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop (pp. 68-77). (LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.lchange-1.7

Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model. / Rastas, Iiro; Ryan, Yann; Tiihonen, Iiro et al.
LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop. ed. / Nina Tahmasebi; Syrielle Montariol; Andrey Kutuzov; Simon Hengchen; Haim Dubossarsky; Lars Borin. Association for Computational Linguistics (ACL), 2022. p. 68-77 (LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Rastas, I, Ryan, Y, Tiihonen, I, Qaraei, M, Repo, L, Babbar, R, Mäkelä, E, Tolonen, M & Ginter, F 2022, Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model. in N Tahmasebi, S Montariol, A Kutuzov, S Hengchen, H Dubossarsky & L Borin (eds), LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop. LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 68-77, 3rd International Workshop on Computational Approaches to Historical Language Change, LChange 2022, Dublin, Ireland, 26/05/22. https://doi.org/10.18653/v1/2022.lchange-1.7

Rastas I, Ryan Y, Tiihonen I, Qaraei M, Repo L, Babbar R et al. Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model. In Tahmasebi N, Montariol S, Kutuzov A, Hengchen S, Dubossarsky H, Borin L, editors, LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2022. p. 68-77. (LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop). doi: 10.18653/v1/2022.lchange-1.7

Rastas, Iiro ; Ryan, Yann ; Tiihonen, Iiro et al. / Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model. LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop. editor / Nina Tahmasebi ; Syrielle Montariol ; Andrey Kutuzov ; Simon Hengchen ; Haim Dubossarsky ; Lars Borin. Association for Computational Linguistics (ACL), 2022. pp. 68-77 (LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop).

@inproceedings{835d4847581c4ae59ddf2d44164577c9,

title = "Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model",

abstract = "In this paper, we describe a BERT model trained on the Eighteenth Century Collections Online (ECCO) dataset of digitized documents. The ECCO dataset poses unique modelling challenges due to the presence of Optical Character Recognition (OCR) artifacts. We establish the performance of the BERT model on a publication year prediction task against linear baseline models and human judgement, finding the BERT model to be superior to both and able to date the works, on average, with less than 7 years absolute error. We also explore how language change over time affects the model by analyzing the features the model uses for publication year predictions as given by the Integrated Gradients model explanation method.",

author = "Iiro Rastas and Yann Ryan and Iiro Tiihonen and Mohammadreza Qaraei and Liina Repo and Rohit Babbar and Eetu M{\"a}kel{\"a} and Mikko Tolonen and Filip Ginter",

note = "Funding Information: The research was supported by the Academy of Finland under the project High Performance Computing for the Detection and Analysis of Historical Discourses. Computational resources were provided by CSC – IT Center for Science. ; 3rd International Workshop on Computational Approaches to Historical Language Change, LChange 2022 ; Conference date: 26-05-2022 Through 27-05-2022",

year = "2022",

month = may,

day = "25",

doi = "10.18653/v1/2022.lchange-1.7",

language = "English",

series = "LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "68--77",

editor = "Nina Tahmasebi and Syrielle Montariol and Andrey Kutuzov and Simon Hengchen and Haim Dubossarsky and Lars Borin",

booktitle = "LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop",

}

TY - GEN

T1 - Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model

AU - Rastas, Iiro

AU - Ryan, Yann

AU - Tiihonen, Iiro

AU - Qaraei, Mohammadreza

AU - Repo, Liina

AU - Babbar, Rohit

AU - Mäkelä, Eetu

AU - Tolonen, Mikko

AU - Ginter, Filip

N1 - Funding Information: The research was supported by the Academy of Finland under the project High Performance Computing for the Detection and Analysis of Historical Discourses. Computational resources were provided by CSC – IT Center for Science.

PY - 2022/5/25

Y1 - 2022/5/25

N2 - In this paper, we describe a BERT model trained on the Eighteenth Century Collections Online (ECCO) dataset of digitized documents. The ECCO dataset poses unique modelling challenges due to the presence of Optical Character Recognition (OCR) artifacts. We establish the performance of the BERT model on a publication year prediction task against linear baseline models and human judgement, finding the BERT model to be superior to both and able to date the works, on average, with less than 7 years absolute error. We also explore how language change over time affects the model by analyzing the features the model uses for publication year predictions as given by the Integrated Gradients model explanation method.

AB - In this paper, we describe a BERT model trained on the Eighteenth Century Collections Online (ECCO) dataset of digitized documents. The ECCO dataset poses unique modelling challenges due to the presence of Optical Character Recognition (OCR) artifacts. We establish the performance of the BERT model on a publication year prediction task against linear baseline models and human judgement, finding the BERT model to be superior to both and able to date the works, on average, with less than 7 years absolute error. We also explore how language change over time affects the model by analyzing the features the model uses for publication year predictions as given by the Integrated Gradients model explanation method.

UR - http://www.scopus.com/inward/record.url?scp=85137449209&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.lchange-1.7

DO - 10.18653/v1/2022.lchange-1.7

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85137449209

T3 - LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop

SP - 68

EP - 77

BT - LChange 2022 - 3rd International Workshop on Computational Approaches to Historical Language Change 2022, Proceedings of the Workshop

A2 - Tahmasebi, Nina

A2 - Montariol, Syrielle

A2 - Kutuzov, Andrey

A2 - Hengchen, Simon

A2 - Dubossarsky, Haim

A2 - Borin, Lars

PB - Association for Computational Linguistics (ACL)

T2 - 3rd International Workshop on Computational Approaches to Historical Language Change, LChange 2022

Y2 - 26 May 2022 through 27 May 2022

ER -

Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model

Abstract

Publication series

Conference

Bibliographical note

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this