How to compare usability of techniques for the specification of virtual agents' behavior?: An experimental pilot study with human subjects

J. Gemrot, C. Brom, M. Bída, J. Bryson

Research output: Chapter in Book/Report/Conference proceedingOther chapter contribution

  • 3 Citations

Abstract

Reactive or dynamic planning is currently the dominant paradigm for controlling virtual agents in 3D videogames. Various reactive planning techniques are employed in the videogame industry while many reactive planning systems and languages are being developed in the academia. Claims about benefits of different approaches are supported by the experience of videogame programmers and the arguments of researchers, but rigorous empirical data corroborating alleged advantages of different methods are lacking. Here, we present results of a pilot study in which we compare the usability of an academic technique designed for programming intelligent agents' behavior with the usability of an unaltered classical programming language. Our study seeks to replicate the situation of professional game programmers considering using an unfamiliar academic system for programming in-game agents. We engaged 30 computer science students attending a university course on virtual agents in two programming assignments. For each, the students had to code high-level behavior of a 3D virtual agent solving a game-like task in the Unreal Tournament 2004 environment. Each student had to use Java for one task and the POSH reactive planner with a graphical editor for the other. We collected quantitative and qualitative usability data. The results indicate that POSH outperforms Java in terms of usability for one of the assigned tasks but not the other. This implies that the suitability of an AI systems-engineering approach is task sensitive. We also discuss lessons learnt about the evaluation process itself, proposing possible improvements in the experimental design. We conclude that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.
LanguageEnglish
Title of host publication Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers
Place of PublicationBerlin
PublisherSpringer
Pages38-62
Number of pages25
Volume7471 LNAI
ISBN (Print)978-3-642-32325-6
DOIs
StatusPublished - 2012

Publication series

NameLecture Notes in Computer Science
PublisherSpringer

Fingerprint

Students
Specifications
Planning
Intelligent agents
Computer programming
Systems engineering
Design of experiments
Computer programming languages
Computer science
Industry

Cite this

Gemrot, J., Brom, C., Bída, M., & Bryson, J. (2012). How to compare usability of techniques for the specification of virtual agents' behavior? An experimental pilot study with human subjects. In Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers (Vol. 7471 LNAI, pp. 38-62). (Lecture Notes in Computer Science). Berlin: Springer. DOI: 10.1007/978-3-642-32326-3_3

How to compare usability of techniques for the specification of virtual agents' behavior? An experimental pilot study with human subjects. / Gemrot, J.; Brom, C.; Bída, M.; Bryson, J.

Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers. Vol. 7471 LNAI Berlin : Springer, 2012. p. 38-62 (Lecture Notes in Computer Science).

Research output: Chapter in Book/Report/Conference proceedingOther chapter contribution

Gemrot, J, Brom, C, Bída, M & Bryson, J 2012, How to compare usability of techniques for the specification of virtual agents' behavior? An experimental pilot study with human subjects. in Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers. vol. 7471 LNAI, Lecture Notes in Computer Science, Springer, Berlin, pp. 38-62. DOI: 10.1007/978-3-642-32326-3_3
Gemrot J, Brom C, Bída M, Bryson J. How to compare usability of techniques for the specification of virtual agents' behavior? An experimental pilot study with human subjects. In Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers. Vol. 7471 LNAI. Berlin: Springer. 2012. p. 38-62. (Lecture Notes in Computer Science). Available from, DOI: 10.1007/978-3-642-32326-3_3
Gemrot, J. ; Brom, C. ; Bída, M. ; Bryson, J./ How to compare usability of techniques for the specification of virtual agents' behavior? An experimental pilot study with human subjects. Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers. Vol. 7471 LNAI Berlin : Springer, 2012. pp. 38-62 (Lecture Notes in Computer Science).
@inbook{77ce8c5e784b4b50a2751855eb92ac5f,
title = "How to compare usability of techniques for the specification of virtual agents' behavior?: An experimental pilot study with human subjects",
abstract = "Reactive or dynamic planning is currently the dominant paradigm for controlling virtual agents in 3D videogames. Various reactive planning techniques are employed in the videogame industry while many reactive planning systems and languages are being developed in the academia. Claims about benefits of different approaches are supported by the experience of videogame programmers and the arguments of researchers, but rigorous empirical data corroborating alleged advantages of different methods are lacking. Here, we present results of a pilot study in which we compare the usability of an academic technique designed for programming intelligent agents' behavior with the usability of an unaltered classical programming language. Our study seeks to replicate the situation of professional game programmers considering using an unfamiliar academic system for programming in-game agents. We engaged 30 computer science students attending a university course on virtual agents in two programming assignments. For each, the students had to code high-level behavior of a 3D virtual agent solving a game-like task in the Unreal Tournament 2004 environment. Each student had to use Java for one task and the POSH reactive planner with a graphical editor for the other. We collected quantitative and qualitative usability data. The results indicate that POSH outperforms Java in terms of usability for one of the assigned tasks but not the other. This implies that the suitability of an AI systems-engineering approach is task sensitive. We also discuss lessons learnt about the evaluation process itself, proposing possible improvements in the experimental design. We conclude that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.",
author = "J. Gemrot and C. Brom and M. B{\'i}da and J. Bryson",
year = "2012",
doi = "10.1007/978-3-642-32326-3_3",
language = "English",
isbn = "978-3-642-32325-6",
volume = "7471 LNAI",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "38--62",
booktitle = "Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers",

}

TY - CHAP

T1 - How to compare usability of techniques for the specification of virtual agents' behavior?

T2 - An experimental pilot study with human subjects

AU - Gemrot,J.

AU - Brom,C.

AU - Bída,M.

AU - Bryson,J.

PY - 2012

Y1 - 2012

N2 - Reactive or dynamic planning is currently the dominant paradigm for controlling virtual agents in 3D videogames. Various reactive planning techniques are employed in the videogame industry while many reactive planning systems and languages are being developed in the academia. Claims about benefits of different approaches are supported by the experience of videogame programmers and the arguments of researchers, but rigorous empirical data corroborating alleged advantages of different methods are lacking. Here, we present results of a pilot study in which we compare the usability of an academic technique designed for programming intelligent agents' behavior with the usability of an unaltered classical programming language. Our study seeks to replicate the situation of professional game programmers considering using an unfamiliar academic system for programming in-game agents. We engaged 30 computer science students attending a university course on virtual agents in two programming assignments. For each, the students had to code high-level behavior of a 3D virtual agent solving a game-like task in the Unreal Tournament 2004 environment. Each student had to use Java for one task and the POSH reactive planner with a graphical editor for the other. We collected quantitative and qualitative usability data. The results indicate that POSH outperforms Java in terms of usability for one of the assigned tasks but not the other. This implies that the suitability of an AI systems-engineering approach is task sensitive. We also discuss lessons learnt about the evaluation process itself, proposing possible improvements in the experimental design. We conclude that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.

AB - Reactive or dynamic planning is currently the dominant paradigm for controlling virtual agents in 3D videogames. Various reactive planning techniques are employed in the videogame industry while many reactive planning systems and languages are being developed in the academia. Claims about benefits of different approaches are supported by the experience of videogame programmers and the arguments of researchers, but rigorous empirical data corroborating alleged advantages of different methods are lacking. Here, we present results of a pilot study in which we compare the usability of an academic technique designed for programming intelligent agents' behavior with the usability of an unaltered classical programming language. Our study seeks to replicate the situation of professional game programmers considering using an unfamiliar academic system for programming in-game agents. We engaged 30 computer science students attending a university course on virtual agents in two programming assignments. For each, the students had to code high-level behavior of a 3D virtual agent solving a game-like task in the Unreal Tournament 2004 environment. Each student had to use Java for one task and the POSH reactive planner with a graphical editor for the other. We collected quantitative and qualitative usability data. The results indicate that POSH outperforms Java in terms of usability for one of the assigned tasks but not the other. This implies that the suitability of an AI systems-engineering approach is task sensitive. We also discuss lessons learnt about the evaluation process itself, proposing possible improvements in the experimental design. We conclude that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.

UR - http://www.scopus.com/inward/record.url?scp=84865291432&partnerID=8YFLogxK

UR - http://dx.doi.org/10.1007/978-3-642-32326-3_3

U2 - 10.1007/978-3-642-32326-3_3

DO - 10.1007/978-3-642-32326-3_3

M3 - Other chapter contribution

SN - 978-3-642-32325-6

VL - 7471 LNAI

T3 - Lecture Notes in Computer Science

SP - 38

EP - 62

BT - Agents for Educational Games and Simulations International Workshop, AEGS 2011, Taipei, Taiwan, May 2, 2011. Revised Papers

PB - Springer

CY - Berlin

ER -