Flesch or Fumble?: Evaluating Readability Standard Alignment of Instruction-Tuned Language Models

Joseph Marvin Imperial; Harish Tayyar Madabushi

Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models

Joseph Marvin Imperial, Harish Tayyar Madabushi

National University, Philippines

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

21 Downloads (Pure)

Abstract

Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives$-$tasks that teachers perform$-$using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5$-$which have shown promising results.

Original language	English
Title of host publication	Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)
Place of Publication	Singapore
Publisher	Association for Computational Linguistics
Pages	205–223
Number of pages	19
Publication status	Published - 31 Dec 2023

Acknowledgements

We thank the anonymous reviewers for their constructive feedback of this work. We also thank Mark Townsend for the assistance with configuring the experiments with the Hex GPU cloud of the Department of Computer Science at the University of Bath.

Funding

JMI is supported by the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI (ART-AI) [EP/S023437/1] of the University of Bath and the Study Grant Program of National University Philippines.

Funders	Funder number
UKRI Centre for Doctoral Training in Accountable Responsible and Transparent Artificial Intelligence	EP/S023437/1
National University, Philippines

Keywords

cs.CL

Access to Document

Cite this

Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models. / Imperial, Joseph Marvin ; Tayyar Madabushi, Harish.
Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). Singapore: Association for Computational Linguistics, 2023. p. 205–223.

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

@inproceedings{5e11eac62e414ba1a9f94c28dcfc0a29,

title = "Flesch or Fumble?: Evaluating Readability Standard Alignment of Instruction-Tuned Language Models",

abstract = "Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives$-$tasks that teachers perform$-$using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5$-$which have shown promising results.",

keywords = "cs.CL",

author = "Imperial, {Joseph Marvin} and {Tayyar Madabushi}, Harish",

year = "2023",

month = dec,

day = "31",

language = "English",

pages = "205–223",

booktitle = "Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)",

publisher = "Association for Computational Linguistics",

}

TY - GEN

T1 - Flesch or Fumble?

T2 - Evaluating Readability Standard Alignment of Instruction-Tuned Language Models

AU - Imperial, Joseph Marvin

AU - Tayyar Madabushi, Harish

PY - 2023/12/31

Y1 - 2023/12/31

N2 - Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives$-$tasks that teachers perform$-$using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5$-$which have shown promising results.

AB - Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives$-$tasks that teachers perform$-$using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5$-$which have shown promising results.

KW - cs.CL

M3 - Chapter in a published conference proceeding

SP - 205

EP - 223

BT - Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

PB - Association for Computational Linguistics

CY - Singapore

ER -