Video Language Modelling
Video Anomaly Detection using Multimodal Knowledge Graphs
Video Language Models hallucination benchmarking and knowledge-graph-grounded vision-language reasoning.
Overview
Video Language Models hallucination benchmarking and knowledge-graph-grounded vision-language reasoning.
Details
This research cluster investigates the reliability, safety, and structural grounding of Video-Language Models (VLMs). Work spans hallucination taxonomy and benchmarking across text, image, and video modalities; safety alignment for multimodal systems; and knowledge-graph-conditioned reasoning to reduce model drift and factual error.
Project Motive
VLMs inherit the hallucination and alignment vulnerabilities of LLMs while introducing new failure modes through the visual modality — unsafe outputs triggered by images or videos, identity drift in video generation. This project builds the diagnostic tools, benchmarks, and grounding methods needed to make VLMs safer and more factually reliable. The insights and methods developed here continue to inform our work on advanced safety mechanisms and grounding strategies for modern multimodal systems.
Where we are
-
Introduced the first fine-grained taxonomy of hallucination across degree, orientation (factual mirage vs. silver lining), and category for both LLMs and multimodal models.
-
Applied causal knowledge graphs to ground visual scene understanding in video generation and VQA, bridging neuro-symbolic reasoning with LM outputs.
Status
Active and ongoing. Current open threads include hallucination mitigation in video generation, prompt-level interventions for multimodal factuality, and knowledge-graph-conditioned grounding for VLMs.