PeRAG: Multi-Modal Perspective-Oriented Verbalization with RAG for Inclusive Decision Making

Urban policy makers require comprehensive insights into transportation issues and demographic distributions to design equitable and efficient infrastructure. However, analyzing multi-modal data (numeric and visual) while accounting for diverse perspectives remains challenging. To address this, we propose PeRAG, a novel pipeline combining multi-modal perspective-oriented verbalization with Retrieval-Augmented Generation (RAG). Our approach first converts numeric transportation/demographic data and population heatmaps into natural language descriptions using LLaMA, incorporating multiple policy-relevant perspectives. These verbalizations are then fed into the RAG system to generate context-aware, perspective driven responses for urban planners. We demonstrate the effectiveness of PeRAG in generating actionable insights for transportation policy, bridging the gap between raw data and decision-making. Our experiments highlight the pipeline’s ability to handle heterogeneous data modalities while adapting to diverse stakeholder viewpoints, offering a scalable solution for smart city analytics