GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Yanbin Wei1,2*, Shuai Fu1,3*, Weisen Jiang1,2^, Zejian Zhang1,2, Zhixiong Zeng1,2, Qi Wu1,2, James T. Kwok2, Yu Zhang1^
1Southern University of Science and Technology, 2Hong Kong University of Science and Technology,
3Australia Institute of Machine Learning, University of Adelaide
*Equal Contribution
^Correspending Authors
Paper Codebase Datasets Checkpoints

Abstract

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., visual graph) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called Graph to vIsual and Textual IntegrAtion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish Graph-based Vision-Language Question Answering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

GITA: Graph to Visual and Textual Integration Framework

GVLQA (Graph to Vision-Language Question Answering) Datasets

Please check out our vision-language graph reasoning dataset, GVLQA at GVLQA Hugging Face Collection.

Evaluating Graph Reasoning on GVLQA-BASE Dataset

Evaluating Graph Reasoning on Real-World Datasets (Link Prediction and Node Classification)

The Effectiveness of Visual Graph Augmentation (Highlight the significance of Layout Augmentation (AUGLY))

Case Study: Explain the experimental observations with cases in straight-forward intuitions

BibTeX

@misc{wei2024gita,
      title={GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning}, 
      author={Yanbin Wei and Shuai Fu and Weisen Jiang and Zejian Zhang and Zhixiong Zeng and Qi Wu and James T. Kwok and Yu Zhang},
      year={2024},
      eprint={2402.02130},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
      }

@article{wei2024rendering,
  title={Rendering Graphs for Graph Reasoning in Multimodal Large Language Models},
  author={Wei, Yanbin and Fu, Shuai and Jiang, Weisen and Kwok, James T and Zhang, Yu},
  journal={arXiv preprint arXiv:2402.02130},
  year={2024}
}