Analyzing Occupant Behavior in Smart Buildings Using Vision-Language Models and Temporal Segmentation Networks

Understanding occupant behavior is crucial for optimizing energy usage, comfort, and sustainability in buildings. Occupant behaviors within indoor environments are inherently complex, stochastic, and diverse, particularly in scenarios involving interactions among multiple individuals. As a result, accurately detecting, tracking, and interpreting such behaviors poses a challenge. In this paper, we propose a novel pipeline that integrates a Visual-Language Model (VLM) with a Temporal Segmentation Network (TSN). The proposed method employs VLM to generate semantically rich textual descriptors of occupant behaviors through image-text alignment, which are subsequently processed by TSN to capture and segment temporal dynamics effectively. Experimental results in a simulated campus classroom scenario validated the proposed method. It achieved high accuracy occupant behavior recognition and demonstrated its feasibility for practical deployment in smart building applications.