Files and links (1)
AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses699.56 kB
Published VersionCC BY V4.0, Open Access
Journal article
AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses
Future Internet, Vol.16(5), pp.1-15
2024
Abstract
Situational awareness (SA) is crucial in disaster response, enhancing the understanding of the environment. Social media, with its extensive user base, offers valuable real-time information for such scenarios. Although SA systems excel in extracting disaster-related details from user-generated content, a common limitation in prior approaches is their emphasis on single-modal extraction rather than embracing multi-modalities. This paper proposed a multimodal hierarchical graph-based situational awareness (MHGSA) system for comprehensive disaster event classification. Specifically, the proposed multimodal hierarchical graph contains nodes representing different disaster events and the features of the event nodes are extracted from the corresponding images and acoustic features. The proposed feature extraction modules with multi-branches for vision and audio features provide hierarchical node features for disaster events of different granularities, aiming to build a coarse-granularity classification task to constrain the model and enhance fine-granularity classification. The relationships between different disaster events in multi-modalities are learned by graph convolutional neural networks to enhance the system’s ability to recognize disaster events, thus enabling the system to fuse complex features of vision and audio. Experimental results illustrate the effectiveness of the proposed visual and audio feature extraction modules in single-modal scenarios. Furthermore, the MHGSA successfully fuses visual and audio features, yielding promising results in disaster event classification tasks.
Details
- Title
- AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses
- Authors
- Jieli Chen - Xi’an Jiaotong-Liverpool UniversityKah Phooi Seng (Corresponding Author) - Xi’an Jiaotong-Liverpool UniversityLi Minn Ang - University of the Sunshine Coast, Queensland, School of Science, Technology and EngineeringJeremy Smith - University of LiverpoolHanyue Xu - Xi’an Jiaotong-Liverpool University
- Publication details
- Future Internet, Vol.16(5), pp.1-15
- Publisher
- MDPI AG
- Date published
- 2024
- DOI
- 10.3390/fi16050161
- ISSN
- 1999-5903
- Copyright note
- © 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
- Data Availability
- The data used in this study is publicly available in Chen, H.; Xie, W.; Vedaldi, A.; Zisserman, A. Vggsound: A Large-Scale Audio-Visual Dataset. In Proceedings of the ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 4–8 May 2020; pp. 721–725.
- Organisation Unit
- School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 991043596402621
- Output Type
- Journal article
Metrics
1 File views/ downloads
39 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Web Of Science research areas
- Computer Science, Information Systems
UN Sustainable Development Goals (SDGs)
This output has contributed to the advancement of the following goals:
Source: InCites