Real-World Scanpaths Exhibit Long-Term Temporal Dependencies: Considerations for Contextual AI for AR Applications

Charlie S. Burlingham, Naveen Sendhilnathan, Xiuyun Wu, T. Scott Murdison, Michael J. Proulx

Research output: Chapter or section in a book/report/conference proceedingChapter in a published conference proceeding


All-day augmented reality (AR) requires contextually-aware artificial intelligence (AI) models that excel across diverse daily contexts. Eye tracking could be a key source of information about user context and intention. However, such models using gaze sometimes struggle to outperform egocentric video-based baseline models. We propose that learning representations of scanpath history in a perceptually-relevant state space may solve this problem. However, scanpaths are often assumed to obey a Markovian assumption, i.e., only the current and previous fixation matter. In a user study (30 participants; 26.2 hours total), we analyzed scanpaths during nine everyday tasks and identified long-term temporal dependencies, with an average timescale of four fixations (2 seconds) into the past (i.e., violating the Markovian assumption). We discovered substantial task-specific variations in these dependencies. This confirms that scanpaths contain stereotyped "motifs"with context-dependent lengths/timescales. We discuss the implications for designing contextual AI models for AR applications.

Original languageEnglish
Title of host publicationProceedings - ETRA 2024, ACM Symposium on Eye Tracking Research and Applications
EditorsStephen N. Spencer
PublisherAssociation for Computing Machinery
Number of pages7
ISBN (Electronic)9798400706073
Publication statusPublished - 4 Jun 2024
Event16th Annual ACM Symposium on Eye Tracking Research and Applications, ETRA 2024 - Hybrid, Glasgow, UK United Kingdom
Duration: 4 Jun 20247 Jun 2024

Publication series

NameEye Tracking Research and Applications Symposium (ETRA)


Conference16th Annual ACM Symposium on Eye Tracking Research and Applications, ETRA 2024
Country/TerritoryUK United Kingdom
CityHybrid, Glasgow


  • All-day AR
  • Contextual AI
  • Deep learning
  • Gaze
  • Information theory
  • Pervasive eye tracking
  • Scanpath
  • Temporal dependencies

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction
  • Ophthalmology
  • Sensory Systems

Cite this