Debaleen Das Spandan

Abstract. Voice-controlled smart assistants have gained widespread popularity, playing a pivotal role in smart homes by providing a natural and convenient interface for interacting with devices However, these assistants are not usable by individuals with physical disabilities and speech impairments. Non-verbal communication methods such as gesture recognition, eye tracking and context awareness can complement these technologies and mitigate some limitations, thereby enhancing the user experience in smart homes. This paper introduces a novel concept of proxemics-enhanced scene graph generation to make human-computer interaction more accessible and intuitive in smart homes. We focus on proxemics, which is the study of the distance between users and surrounding objects, to enable spatial awareness and intuitive automation. Scene graphs offer a structured representation of the positions, relationships, and properties of both static and moving objects in indoor environments. The novelty of this approach lies in the application of proxemics to scene graph generation, extracting spatial information, and understanding scenes to automate smart home actions. We have incorporated a distance attribute into the scene graph predicate to quantify human-object relationships.

This work is a part of my Master’s thesis. This work was published in IEEE MIPR 2024.

Short video showcasing the proof-of-concept