Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group

A challenge for clinical trial design

Michael Hughes, Andrew Tracey, Monica Bhushan, Kuntal Chakravarty, Christopher P Denton, Shirish Dubey, Serena Guiducci, Lindsay Muir, Voon Ong, Louise Parker, John D Pauling, Athiveeraramapandian Prabu, Christine Rogers, Christopher Roberts, Ariane L Herrick

Research output: Contribution to journalArticle

6 Citations (Scopus)
24 Downloads (Pure)

Abstract

Introduction: The reliability of clinician grading of systemic sclerosis-related digital ulcers has been reported to be poor to moderate at best, which has important implications for clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group digital ulcer definitions among UK clinicians with an interest in systemic sclerosis.

Methods: Raters graded (through a custom-built interface) 90 images (80 unique and 10 repeat) of a range of digital lesions collected from patients with systemic sclerosis. Lesions were graded on an ordinal scale of severity: 'no ulcer', 'healed ulcer' or 'digital ulcer'.

Results: A total of 23 clinicians - 18 rheumatologists, 3 dermatologists, 1 hand surgeon and 1 specialist rheumatology nurse - completed the study. A total of 2070 (1840 unique + 230 repeat) image gradings were obtained. For intra-rater reliability, across all images, the overall weighted kappa coefficient was high (0.71) and was moderate (0.55) when averaged across individual raters. Overall inter-rater reliability was poor (0.15).

Conclusion: Although our proposed digital ulcer definitions had high intra-rater reliability, the overall inter-rater reliability was poor. Our study highlights the challenges of digital ulcer assessment by clinicians with an interest in systemic sclerosis and provides a number of useful insights for future clinical trial design. Further research is warranted to improve the reliability of digital ulcer definition/rating as an outcome measure in clinical trials, including examining the role for objective measurement techniques, and the development of digital ulcer patient-reported outcome measures.

Original languageEnglish
Pages (from-to)170-174
Number of pages5
JournalJournal of Scleroderma and Related Disorders
Volume3
Issue number2
Early online date27 Mar 2018
DOIs
Publication statusPublished - 1 Jun 2018

Fingerprint

Ulcer
Clinical Trials
Systemic Scleroderma
Rheumatology
Hand
Outcome Assessment (Health Care)
Research

Cite this

Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group : A challenge for clinical trial design. / Hughes, Michael; Tracey, Andrew; Bhushan, Monica; Chakravarty, Kuntal; Denton, Christopher P; Dubey, Shirish; Guiducci, Serena; Muir, Lindsay; Ong, Voon; Parker, Louise; Pauling, John D; Prabu, Athiveeraramapandian; Rogers, Christine; Roberts, Christopher; Herrick, Ariane L.

In: Journal of Scleroderma and Related Disorders, Vol. 3, No. 2, 01.06.2018, p. 170-174.

Research output: Contribution to journalArticle

Hughes, M, Tracey, A, Bhushan, M, Chakravarty, K, Denton, CP, Dubey, S, Guiducci, S, Muir, L, Ong, V, Parker, L, Pauling, JD, Prabu, A, Rogers, C, Roberts, C & Herrick, AL 2018, 'Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group: A challenge for clinical trial design', Journal of Scleroderma and Related Disorders, vol. 3, no. 2, pp. 170-174. https://doi.org/10.1177/2397198318764796
Hughes, Michael ; Tracey, Andrew ; Bhushan, Monica ; Chakravarty, Kuntal ; Denton, Christopher P ; Dubey, Shirish ; Guiducci, Serena ; Muir, Lindsay ; Ong, Voon ; Parker, Louise ; Pauling, John D ; Prabu, Athiveeraramapandian ; Rogers, Christine ; Roberts, Christopher ; Herrick, Ariane L. / Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group : A challenge for clinical trial design. In: Journal of Scleroderma and Related Disorders. 2018 ; Vol. 3, No. 2. pp. 170-174.
@article{69b22d547064429eaabeb659916aa803,
title = "Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group: A challenge for clinical trial design",
abstract = "Introduction: The reliability of clinician grading of systemic sclerosis-related digital ulcers has been reported to be poor to moderate at best, which has important implications for clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group digital ulcer definitions among UK clinicians with an interest in systemic sclerosis.Methods: Raters graded (through a custom-built interface) 90 images (80 unique and 10 repeat) of a range of digital lesions collected from patients with systemic sclerosis. Lesions were graded on an ordinal scale of severity: 'no ulcer', 'healed ulcer' or 'digital ulcer'.Results: A total of 23 clinicians - 18 rheumatologists, 3 dermatologists, 1 hand surgeon and 1 specialist rheumatology nurse - completed the study. A total of 2070 (1840 unique + 230 repeat) image gradings were obtained. For intra-rater reliability, across all images, the overall weighted kappa coefficient was high (0.71) and was moderate (0.55) when averaged across individual raters. Overall inter-rater reliability was poor (0.15).Conclusion: Although our proposed digital ulcer definitions had high intra-rater reliability, the overall inter-rater reliability was poor. Our study highlights the challenges of digital ulcer assessment by clinicians with an interest in systemic sclerosis and provides a number of useful insights for future clinical trial design. Further research is warranted to improve the reliability of digital ulcer definition/rating as an outcome measure in clinical trials, including examining the role for objective measurement techniques, and the development of digital ulcer patient-reported outcome measures.",
author = "Michael Hughes and Andrew Tracey and Monica Bhushan and Kuntal Chakravarty and Denton, {Christopher P} and Shirish Dubey and Serena Guiducci and Lindsay Muir and Voon Ong and Louise Parker and Pauling, {John D} and Athiveeraramapandian Prabu and Christine Rogers and Christopher Roberts and Herrick, {Ariane L}",
year = "2018",
month = "6",
day = "1",
doi = "10.1177/2397198318764796",
language = "English",
volume = "3",
pages = "170--174",
journal = "Journal of Scleroderma and Related Disorders",
issn = "2397-1983",
publisher = "Sage Publications",
number = "2",

}

TY - JOUR

T1 - Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group

T2 - A challenge for clinical trial design

AU - Hughes, Michael

AU - Tracey, Andrew

AU - Bhushan, Monica

AU - Chakravarty, Kuntal

AU - Denton, Christopher P

AU - Dubey, Shirish

AU - Guiducci, Serena

AU - Muir, Lindsay

AU - Ong, Voon

AU - Parker, Louise

AU - Pauling, John D

AU - Prabu, Athiveeraramapandian

AU - Rogers, Christine

AU - Roberts, Christopher

AU - Herrick, Ariane L

PY - 2018/6/1

Y1 - 2018/6/1

N2 - Introduction: The reliability of clinician grading of systemic sclerosis-related digital ulcers has been reported to be poor to moderate at best, which has important implications for clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group digital ulcer definitions among UK clinicians with an interest in systemic sclerosis.Methods: Raters graded (through a custom-built interface) 90 images (80 unique and 10 repeat) of a range of digital lesions collected from patients with systemic sclerosis. Lesions were graded on an ordinal scale of severity: 'no ulcer', 'healed ulcer' or 'digital ulcer'.Results: A total of 23 clinicians - 18 rheumatologists, 3 dermatologists, 1 hand surgeon and 1 specialist rheumatology nurse - completed the study. A total of 2070 (1840 unique + 230 repeat) image gradings were obtained. For intra-rater reliability, across all images, the overall weighted kappa coefficient was high (0.71) and was moderate (0.55) when averaged across individual raters. Overall inter-rater reliability was poor (0.15).Conclusion: Although our proposed digital ulcer definitions had high intra-rater reliability, the overall inter-rater reliability was poor. Our study highlights the challenges of digital ulcer assessment by clinicians with an interest in systemic sclerosis and provides a number of useful insights for future clinical trial design. Further research is warranted to improve the reliability of digital ulcer definition/rating as an outcome measure in clinical trials, including examining the role for objective measurement techniques, and the development of digital ulcer patient-reported outcome measures.

AB - Introduction: The reliability of clinician grading of systemic sclerosis-related digital ulcers has been reported to be poor to moderate at best, which has important implications for clinical trial design. The aim of this study was to examine the reliability of new proposed UK Scleroderma Study Group digital ulcer definitions among UK clinicians with an interest in systemic sclerosis.Methods: Raters graded (through a custom-built interface) 90 images (80 unique and 10 repeat) of a range of digital lesions collected from patients with systemic sclerosis. Lesions were graded on an ordinal scale of severity: 'no ulcer', 'healed ulcer' or 'digital ulcer'.Results: A total of 23 clinicians - 18 rheumatologists, 3 dermatologists, 1 hand surgeon and 1 specialist rheumatology nurse - completed the study. A total of 2070 (1840 unique + 230 repeat) image gradings were obtained. For intra-rater reliability, across all images, the overall weighted kappa coefficient was high (0.71) and was moderate (0.55) when averaged across individual raters. Overall inter-rater reliability was poor (0.15).Conclusion: Although our proposed digital ulcer definitions had high intra-rater reliability, the overall inter-rater reliability was poor. Our study highlights the challenges of digital ulcer assessment by clinicians with an interest in systemic sclerosis and provides a number of useful insights for future clinical trial design. Further research is warranted to improve the reliability of digital ulcer definition/rating as an outcome measure in clinical trials, including examining the role for objective measurement techniques, and the development of digital ulcer patient-reported outcome measures.

U2 - 10.1177/2397198318764796

DO - 10.1177/2397198318764796

M3 - Article

VL - 3

SP - 170

EP - 174

JO - Journal of Scleroderma and Related Disorders

JF - Journal of Scleroderma and Related Disorders

SN - 2397-1983

IS - 2

ER -