Image to Label to Answer: An Efficient Framework for Enhanced Clinical Applications in Medical Visual Question Answering

Jianfeng Wang; Kah Phooi Seng; Yi Shen; Li-Minn Ang; Difeng Huang

doi:10.3390/electronics13122273

Back

Image to Label to Answer: An Efficient Framework for Enhanced Clinical Applications in Medical Visual Question Answering

Journal article

Open access

Peer reviewed

Image to Label to Answer: An Efficient Framework for Enhanced Clinical Applications in Medical Visual Question Answering

Jianfeng Wang, Kah Phooi Seng, Yi Shen, Li-Minn Ang and Difeng Huang

Electronics, Vol.13(12), pp.1-12

2024

DOI: https://doi.org/10.3390/electronics13122273

Files and links (1)

pdf

electronics-13-02273-v21.33 MBDownload View

Published VersionCC BY V4.0, Open Access

Abstract

medical visual question answering (Med-VQA)

large language models (LLMs)

multi-label learning

attention mechanisms

zero-shot learning

Medical Visual Question Answering (Med-VQA) faces significant limitations in application development due to sparse and challenging data acquisition. Existing approaches focus on multi-modal learning to equip models with medical image inference and natural language understanding, but this worsens data scarcity in Med-VQA, hindering clinical application and advancement. This paper proposes the ITLTA framework for Med-VQA, designed based on field requirements. ITLTA combines multi-label learning of medical images with the language understanding and reasoning capabilities of large language models (LLMs) to achieve zero-shot learning, meeting natural language module needs without end-to-end training. This approach reduces deployment costs and training data requirements, allowing LLMs to function as flexible, plug-and-play modules. To enhance multi-label classification accuracy, the framework uses external medical image data for pretraining, integrated with a joint feature and label attention mechanism. This configuration ensures robust performance and applicability, even with limited data. Additionally, the framework clarifies the decision-making process for visual labels and question prompts, enhancing the interpretability of Med-VQA. Validated on the VQA-Med 2019 dataset, our method demonstrates superior effectiveness compared to existing methods, confirming its outstanding performance for enhanced clinical applications.

Details

Title: Image to Label to Answer: An Efficient Framework for Enhanced Clinical Applications in Medical Visual Question Answering
Authors: Jianfeng Wang - Xi’an Jiaotong-Liverpool University
Kah Phooi Seng (Corresponding Author) - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
Yi Shen - Suzhou University of Science and Technology
Li-Minn Ang - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
Difeng Huang - Hubei University of Education
Publication details: Electronics, Vol.13(12), pp.1-12
Publisher: MDPI AG
Date published: 2024
DOI: 10.3390/electronics13122273
ISSN: 2079-9292
Copyright note: © 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Data Availability: The data presented in this study are available in this article.
Organisation Unit: School of Science, Technology and Engineering
Language: English
Record Identifier: 991043598402621
Output Type: Journal article

Metrics

1 File views/ downloads

29 Record Views