Cross-modal hash retrieval of medical images based on Transformer semantic alignment_Journal of Biomedical Engineering

Authors：

WU Qianlin ¹ , TANG Lun ¹ , LIU Qinghai ¹ ,  XU Liming ^2,3 , CHEN Qianbin ¹

1. School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China;
2. Sichuan Artificial Intelligence Research Institute, Yibin, Sichuan 644005, P. R. China;
3. School of Computer Science, China West Normal University, Nanchong, Sichuan 637009, P. R. China;

Corresponding author：

XU Liming, Email: xulm@cwnu.edu.cn

Keywords：

Cross-modal Hash; Transformer; Semantic alignment; Segmented training

DOI：

10.7507/1001-5515.202407034

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Medical cross-modal retrieval aims to achieve semantic similarity search between different modalities of medical cases, such as quickly locating relevant ultrasound images through ultrasound reports, or using ultrasound images to retrieve matching reports. However, existing medical cross-modal hash retrieval methods face significant challenges, including semantic and visual differences between modalities and the scalability issues of hash algorithms in handling large-scale data. To address these challenges, this paper proposes a Medical image Semantic Alignment Cross-modal Hashing based on Transformer (MSACH). The algorithm employed a segmented training strategy, combining modality feature extraction and hash function learning, effectively extracting low-dimensional features containing important semantic information. A Transformer encoder was used for cross-modal semantic learning. By introducing manifold similarity constraints, balance constraints, and a linear classification network constraint, the algorithm enhances the discriminability of the hash codes. Experimental results demonstrate that the MSACH algorithm improves the mean average precision (MAP) by 11.8% and 12.8% on two datasets compared to traditional methods. The algorithm exhibits outstanding performance in enhancing retrieval accuracy and handling large-scale medical data, showing promising potential for practical applications.

1.	Huang S C, Pareek A, Seyyedi S, et al. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med, 2020, 3(1): 136.
2.	Das P, Neelima A. An overview of approaches for content-based medical image retrieval. Int J Multim Inf Retr, 2017, 6(4): 271-280.
3.	Peng Y, Qi J, Huang X, et al. CCL: Cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans on Mul, 2017, 20(2): 405-420.
4.	苏海, 钟雨辰. 基于偏差抑制对比学习的无监督深度哈希图像检索. 计算机系统应用, 2024: 1-9.
5.	刘华咏, 徐明慧. 基于混合注意力与偏振非对称损失的哈希图像检索. 计算机科学, 2024: 1-12.
6.	Jiang Q Y, Li W J. Deep cross-modal hashing// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 3232-3240.
7.	Xu L, Zeng X, Zheng B, et al. Multi-manifold deep discriminative cross-modal hashing for medical image retrieval. IEEE Trans Image Process, 2022, 31: 3371-3385.
8.	Guan A, Liu L, Fu X, et al. Precision medical image hash retrieval by interpretability and feature fusion. Comput Methods Programs Biomed, 2022, 222: 106945.
9.	Fang J, Fu H, Liu J. Deep triplet hashing network for case-based medical image retrieval. Med Image Anal, 2021, 69: 101981.
10.	Han J, Men A, Liu Y, et al. IoT-V2E: an uncertainty-aware cross-modal hashing retrieval between infrared-videos and EEGs for automated sleep state analysis. IEEE Internet Things J, 2024, 11(3): 4551-4569.
11.	Huang S C, Pareek A, Zamanian R, et al. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Sci Rep, 2020, 10(1): 22147.
12.	Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation// International Conference on Machine Learning. PMLR, 2022: 12888-12900.
13.	Bao H, Wang W, Dong L, et al. VLMo: Unified vision-language pre-training with mixture-of-modality-experts. Adv Neural Inf Process Syst, 2022, 35: 32897-32912.
14.	Li J, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation. Adv Neural Inf Process Syst, 2021, 34: 9694-9705.
15.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS2017). Long Beach: NIPS, 2017: 6000-6010.
16.	Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention// International Conference on Machine Learning. PMLR, 2021: 10347-10357.
17.	Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv, 2018: 1810.04805.
18.	Loshchilov I, Hutter F. Decoupled weight decay regularization. arXiv preprint arXiv, 2017: 1711.05101.
19.	Nie X, Wang B, Li J, et al. Deep multiscale fusion hashing for cross-modal retrieval. IEEE Trans Circuits Syst Video Technol, 2020, 31(1): 401-410.
20.	Tu R C, Mao X L, Ma B, et al. Deep cross-modal hashing with hashing functions and unified hash codes jointly learning. IEEE Trans Knowl Data Eng, 2020, 34(2): 560-572.
21.	Li T, Yang X, Wang B, et al. Bi-CMR: Bidirectional reinforcement guided hashing for effective cross-modal retrieval// Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2022, 36(9): 10275-10282.
22.	Li C, Deng C, Li N, et al. Self-supervised adversarial hashing networks for cross-modal retrieval// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4242-4251.
23.	Xie D, Deng C, Li C, et al. Multi-task consistency-preserving adversarial hashing for cross-modal retrieval. IEEE Trans Image Process, 2020, 29: 3626-3637.
24.	Xu R, Li C, Yan J, et al. Graph convolutional network hashing for cross-modal retrieval// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: IJCAI, 2019, 2019: 982-988.
25.	Tu J, Liu X, Lin Z, et al. Differentiable cross-modal hashing via multimodal transformers// Proceedings of the 30th ACM International Conference on Multimedia. Lisboa: ACM, 2022: 453-461.

1. Huang S C, Pareek A, Seyyedi S, et al. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med, 2020, 3(1): 136.
2. Das P, Neelima A. An overview of approaches for content-based medical image retrieval. Int J Multim Inf Retr, 2017, 6(4): 271-280.
3. Peng Y, Qi J, Huang X, et al. CCL: Cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans on Mul, 2017, 20(2): 405-420.
4. 苏海, 钟雨辰. 基于偏差抑制对比学习的无监督深度哈希图像检索. 计算机系统应用, 2024: 1-9.
5. 刘华咏, 徐明慧. 基于混合注意力与偏振非对称损失的哈希图像检索. 计算机科学, 2024: 1-12.
6. Jiang Q Y, Li W J. Deep cross-modal hashing// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 3232-3240.
7. Xu L, Zeng X, Zheng B, et al. Multi-manifold deep discriminative cross-modal hashing for medical image retrieval. IEEE Trans Image Process, 2022, 31: 3371-3385.
8. Guan A, Liu L, Fu X, et al. Precision medical image hash retrieval by interpretability and feature fusion. Comput Methods Programs Biomed, 2022, 222: 106945.
9. Fang J, Fu H, Liu J. Deep triplet hashing network for case-based medical image retrieval. Med Image Anal, 2021, 69: 101981.
10. Han J, Men A, Liu Y, et al. IoT-V2E: an uncertainty-aware cross-modal hashing retrieval between infrared-videos and EEGs for automated sleep state analysis. IEEE Internet Things J, 2024, 11(3): 4551-4569.
11. Huang S C, Pareek A, Zamanian R, et al. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Sci Rep, 2020, 10(1): 22147.
12. Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation// International Conference on Machine Learning. PMLR, 2022: 12888-12900.
13. Bao H, Wang W, Dong L, et al. VLMo: Unified vision-language pre-training with mixture-of-modality-experts. Adv Neural Inf Process Syst, 2022, 35: 32897-32912.
14. Li J, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation. Adv Neural Inf Process Syst, 2021, 34: 9694-9705.
15. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS2017). Long Beach: NIPS, 2017: 6000-6010.
16. Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention// International Conference on Machine Learning. PMLR, 2021: 10347-10357.
17. Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv, 2018: 1810.04805.
18. Loshchilov I, Hutter F. Decoupled weight decay regularization. arXiv preprint arXiv, 2017: 1711.05101.
19. Nie X, Wang B, Li J, et al. Deep multiscale fusion hashing for cross-modal retrieval. IEEE Trans Circuits Syst Video Technol, 2020, 31(1): 401-410.
20. Tu R C, Mao X L, Ma B, et al. Deep cross-modal hashing with hashing functions and unified hash codes jointly learning. IEEE Trans Knowl Data Eng, 2020, 34(2): 560-572.
21. Li T, Yang X, Wang B, et al. Bi-CMR: Bidirectional reinforcement guided hashing for effective cross-modal retrieval// Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2022, 36(9): 10275-10282.
22. Li C, Deng C, Li N, et al. Self-supervised adversarial hashing networks for cross-modal retrieval// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4242-4251.
23. Xie D, Deng C, Li C, et al. Multi-task consistency-preserving adversarial hashing for cross-modal retrieval. IEEE Trans Image Process, 2020, 29: 3626-3637.
24. Xu R, Li C, Yan J, et al. Graph convolutional network hashing for cross-modal retrieval// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: IJCAI, 2019, 2019: 982-988.
25. Tu J, Liu X, Lin Z, et al. Differentiable cross-modal hashing via multimodal transformers// Proceedings of the 30th ACM International Conference on Multimedia. Lisboa: ACM, 2022: 453-461.

Journal of Biomedical Engineering

Latest ArticlesCross-modal hash retrieval of medical images based on Transformer semantic alignment

Abstract Full text Figures/Tables Video References Cited by

Format

Content