The Role of Benchmarking in Symbolic Computation: (Position Paper)

James Davenport

doi:10.1109/SYNASC.2018.00050

The Role of Benchmarking in Symbolic Computation: (Position Paper)

James Davenport

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

41 Downloads (Pure)

Abstract

There is little doubt that, in the minds of most symbolic computation researchers, the ideal paper consists of a problem statement, a new algorithm, a complexity analysis and preferably a few validating examples. There are many such great papers. This paradigm has served computer algebra well for many years, and indeed continues to do so where it is applicable. However, it is much less applicable to sparse problems, where there are many NP-hardness results, or to many problems coming from algebraic geometry, where the worst-case complexity seems to be rare.

We argue that, in these cases, the field should take a leaf out of the practices of the SAT-solving community, and adopt systematic benchmarking, and benchmarking contests, as a way measuring (and stimulating) progress. This would involve a change of culture.

Original language	English
Title of host publication	Proceedings of the 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2018
Publisher	IEEE
ISBN (Electronic)	9781728106250
ISBN (Print)	9781728106267
DOIs	https://doi.org/10.1109/SYNASC.2018.00050
Publication status	Published - 2018

Keywords

Computer algebra
Benchmarking
Performance

Access to Document

10.1109/SYNASC.2018.00050

SYNASC2018-JHDpositionV7Accepted author manuscript, 266 KBLicence: CC BY-NC-ND

Projects

1 Finished

SC-square - Satisfiability Checking and Symbolic Computation: uniting two communities to solve real problems

Davenport, J.

EU - Horizon 2020

1/07/16 → 31/08/18

Project: EU Commission

Cite this

Davenport, J. (2018). The Role of Benchmarking in Symbolic Computation: (Position Paper). In Proceedings of the 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2018 IEEE. https://doi.org/10.1109/SYNASC.2018.00050

@inproceedings{aee4c0b24a2f4bfea0d09fe44bafc18a,

title = "The Role of Benchmarking in Symbolic Computation: (Position Paper)",

abstract = "There is little doubt that, in the minds of most symbolic computation researchers, the ideal paper consists of a problem statement, a new algorithm, a complexity analysis and preferably a few validating examples. There are many such great papers. This paradigm has served computer algebra well for many years, and indeed continues to do so where it is applicable. However, it is much less applicable to sparse problems, where there are many NP-hardness results, or to many problems coming from algebraic geometry, where the worst-case complexity seems to be rare.We argue that, in these cases, the field should take a leaf out of the practices of the SAT-solving community, and adopt systematic benchmarking, and benchmarking contests, as a way measuring (and stimulating) progress. This would involve a change of culture.",

keywords = "Computer algebra, Benchmarking, Performance",

author = "James Davenport",

note = "To be published, on past form, in 2019, but will display a 2018 publication date.",

year = "2018",

doi = "10.1109/SYNASC.2018.00050",

language = "English",

isbn = "9781728106267",

booktitle = "Proceedings of the 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2018",

publisher = "IEEE",

address = "USA United States",

}

TY - GEN

T1 - The Role of Benchmarking in Symbolic Computation

T2 - (Position Paper)

AU - Davenport, James

N1 - To be published, on past form, in 2019, but will display a 2018 publication date.

PY - 2018

Y1 - 2018

N2 - There is little doubt that, in the minds of most symbolic computation researchers, the ideal paper consists of a problem statement, a new algorithm, a complexity analysis and preferably a few validating examples. There are many such great papers. This paradigm has served computer algebra well for many years, and indeed continues to do so where it is applicable. However, it is much less applicable to sparse problems, where there are many NP-hardness results, or to many problems coming from algebraic geometry, where the worst-case complexity seems to be rare.We argue that, in these cases, the field should take a leaf out of the practices of the SAT-solving community, and adopt systematic benchmarking, and benchmarking contests, as a way measuring (and stimulating) progress. This would involve a change of culture.

AB - There is little doubt that, in the minds of most symbolic computation researchers, the ideal paper consists of a problem statement, a new algorithm, a complexity analysis and preferably a few validating examples. There are many such great papers. This paradigm has served computer algebra well for many years, and indeed continues to do so where it is applicable. However, it is much less applicable to sparse problems, where there are many NP-hardness results, or to many problems coming from algebraic geometry, where the worst-case complexity seems to be rare.We argue that, in these cases, the field should take a leaf out of the practices of the SAT-solving community, and adopt systematic benchmarking, and benchmarking contests, as a way measuring (and stimulating) progress. This would involve a change of culture.

KW - Computer algebra

KW - Benchmarking

KW - Performance

U2 - 10.1109/SYNASC.2018.00050

DO - 10.1109/SYNASC.2018.00050

M3 - Conference contribution

SN - 9781728106267

BT - Proceedings of the 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2018

PB - IEEE

ER -