CAMGF-CLIP: Enhancing Multi-view CLIP for Multimodal Sarcasm Detection with Bidirectional Cross-modal Attention and Adaptive Gated Fusion
Keywords:
Multimodal sarcasm detection, Image-text fusion, Cross-modal attention, Gated fusion, Vision-language modelsAbstract
Multimodal sarcasm detection seeks to identify sarcastic intent in social-media posts that pair an image with text, where sarcasm frequently arises from an incongruity between the two modalities rather than from either modality in isolation. We adopt a reproduce-then-improve methodology based on the corrected MMSD2.0 benchmark and its Multi-view CLIP (MV-CLIP) model. We first re-implement MV-CLIP from scratch and retrain it under controlled conditions, reproducing a faithful baseline of approximately 85% accuracy and 85% macro-F1. We then conduct quantitative and qualitative error analyses and identify a consistent failure mode: the baseline over-predicts sarcasm, achieving high recall but comparatively low precision. We trace this behavior to the fusion stage, where image and text representations are combined through a single scalar-weighted sum that re-weights the modalities but does not allow them to interact. Motivated by this observation, we propose CAMGF-CLIP, which augments MV-CLIP with two complementary, drop-in fusion operators while keeping the remainder of the architecture fixed: (i) an adaptive gated fusion mechanism that learns an example- and feature-specific balance between the modalities, and (ii) bidirectional cross-modal attention, in which the text and image token sequences attend to one another before being combined, providing an explicit mechanism for representing image–text contradiction. Because only the fusion operator changes across models, performance differences can be attributed to the proposed mechanism. On MMSD2.0, our reproduction of MV-CLIP achieves 85.4% accuracy and 85.1% macro-F1. Under an identical, validation-selected protocol, the gated and cross-modal variants achieve 83.5% and 84.7% accuracy, respectively. Although neither exceeds the baseline on the selected checkpoint, both reduce the precision–recall imbalance associated with the baseline's over-prediction, while cross-modal attention matches the baseline's accuracy at its best epoch. We release our implementation and analyses to support reproducibility.
References
[1] L. Qin, S. Huang, Q. Chen, C. Cai, Y. Zhang, B. Liang, W. Che, and R. Xu, “MMSD2.0: Towards a reliable multi-modal sarcasm detection system,” in Findings of the Association for Computational Linguistics (ACL), 2023, pp. 10834–10845.
[2] Y. Cai, H. Cai, and X. Wan, “Multi-modal sarcasm detection in Twitter with hierarchical fusion model,” in Proc. ACL, 2019, pp. 2506–2515.
[3] R. Schifanella, P. de Juan, J. Tetreault, and L. Cao, “Detecting sarcasm in multimodal social platforms,” in Proc. ACM Multimedia, 2016, pp. 1136–1145.
[4] N. Xu, Z. Zeng, and W. Mao, “Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association,” in Proc. ACL, 2020, pp. 3777–3786.
[5] H. Pan, Z. Lin, P. Fu, Y. Qi, and W. Wang, “Modeling intra and inter modality incongruity for multi-modal sarcasm detection,” in Findings of the ACL: EMNLP, 2020, pp. 1383–1392.
[6] B. Liang, C. Lou, X. Li, L. Gui, M. Yang, and R. Xu, “Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs,” in Proc. ACM Multimedia, 2021, pp. 4707–4715.
[7] B. Liang, C. Lou, X. Li, M. Yang, L. Gui, Y. He, W. Pei, and R. Xu, “Multi-modal sarcasm detection via cross-modal graph convolutional network,” in Proc. ACL, 2022, pp. 1767–1777.
[8] H. Liu, W. Wang, and H. Li, “Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement,” in Proc. EMNLP, 2022, pp. 4995–5006.
[9] Y. Tian, N. Xu, R. Zhang, and W. Mao, “Dynamic routing transformer network for multimodal sarcasm detection,” in Proc. ACL, 2023, pp. 2468–2480.
[10] A. Radford et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, 2021, pp. 8748–8763.
[11] A. Dosovitskiy et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021.
[12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770–778.
[13] W. Kim, B. Son, and I. Kim, “ViLT: Vision-and-language transformer without convolution or region supervision,” in Proc. ICML, 2021, pp. 5583–5594.
[14] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” in Proc. NeurIPS, 2021, pp. 9694–9705.
[15] A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017, pp. 5998–6008.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.
[17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[18] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proc. EMNLP, 2014, pp. 1532–1543.
[19] A. Joshi, P. Bhattacharyya, and M. J. Carman, “Automatic sarcasm detection: A survey,” ACM Computing Surveys, vol. 50, no. 5, pp. 1–22, 2017.
[20] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019.
[21] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” in Proc. EMNLP, 2017, pp. 1103–1114.
[22] J. Arevalo, T. Solorio, M. Montes-y-Gómez, and F. A. González, “Gated multimodal units for information fusion,” in Proc. ICLR Workshop, 2017.
[23] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” in Proc. ACL, 2019, pp. 6558–6569.
[24] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” in Proc. NeurIPS, 2021, pp. 14200–14213.
[25] J. Hessel and L. Lee, “Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!,” in Proc. EMNLP, 2020, pp. 861–877.
[26] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019.
[27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
[28] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
[29] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016.
[30] D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv:1606.08415, 2016.
[31] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proc. ACL, 2011, pp. 142–150.
[32] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, 2017.
[33] T. Wolf et al., “Transformers: State-of-the-art natural language processing,” in Proc. EMNLP: System Demonstrations, 2020, pp. 38–45.
[34] A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Proc. NeurIPS, 2019, pp. 8024–8035.
[35] E. Cambria, B. Schuller, Y. Xia, and C. Havasi, “New avenues in opinion mining and sentiment analysis,” IEEE Intelligent Systems, vol. 28, no. 2, pp. 15–21, 2013.
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 AHMED SEIFLEISLAM FAYEZ AYESH ELAYADY
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright in their work. The articles published by the International Journal of Global Engineering and Applications (IJGEA) are licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits others to share and adapt the work for any purpose, provided appropriate credit is given to the original author(s) and the source is acknowledged.
For the full license terms, please visit:
https://creativecommons.org/licenses/by/4.0/
