Conference paper
Human and Large Language Model Intent Detection in Image-Based Self-Expression of People with Intellectual Disability
Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, pp.199-208
ACM SIGIR Conference on Human Information Interaction and Retrieval, 2024 (Sheffield, United Kingdom, 10-Mar-2024 - 14-Mar-2024)
Association for Computing Machinery
2024
Abstract
Non-verbal communication is essential for the social inclusion of individuals with an intellectual disability, affecting interactions with others as well as technological systems. This study focuses on non-symbolic communication of people with intellectual disability through generic images without specific or detailed subject matter. A key challenge in this medium is discerning the underlying intentions behind images selected as visual prompts for conversation.
Through interviews with people with intellectual disability, we collected a dataset of images and their associated communication intentions, as well as the interpreted intention from potential audiences including humans and systems. Notably, we employed GPT-4, a Large Language Model (LLM), to decipher the images and provide insights from the Conversational Systems’ (CSs) perspective. Adopting a user-centered qualitative approach, we analyzed this data to understand the nuances of image-based self-expression and identify areas of ambiguity. Additionally, we present an analysis of comprehension levels and challenges for intent detection faced by humans and systems.
Our findings suggest that generic images offer a rich medium for individuals to share personal interests and unique experiences, enriching communication. Within this framework, we identified that situational, personal, and historical contexts facilitate understanding intents. In a comparative analysis of human and systems viewpoints, we found that LLMs have encouraging capabilities for detecting various aspects of ambiguity in predicting user intentions. Based on this, we offer design strategies for intent clarification and crafting more inclusive multimodal conversational tools for individuals with intellectual disability. Findings can be extrapolated to enhance image-based information retrieval and recommendation systems.
Details
- Title
- Human and Large Language Model Intent Detection in Image-Based Self-Expression of People with Intellectual Disability
- Authors
- Alieh Hajizadeh Saffar - Queensland University of TechnologyLaurianne Sitbon - Queensland University of TechnologyMaria Hoogstrate - Queensland University of TechnologyAhmed Abbas - Queensland University of TechnologySirinthip Roomkham - University of the Sunshine Coast, Queensland, School of Science, Technology and EngineeringDimity Miller - Queensland University of Technology
- Publication details
- Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, pp.199-208
- Conference details
- ACM SIGIR Conference on Human Information Interaction and Retrieval, 2024 (Sheffield, United Kingdom, 10-Mar-2024 - 14-Mar-2024)
- Publisher
- Association for Computing Machinery
- DOI
- 10.1145/3627508.3638347; 10.1145/3627508
- Organisation Unit
- School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 991013097002621
- Output Type
- Conference paper
Metrics
5 Record Views