Naslov
A Dynamic Gesture Recognition Interface for Smart Home Control based on Croatian Sign Language

Autori
Kraljević, Luka ; Russo, Mladen ; Pauković, Matija ; Šarić, Matko

Izvornik
Applied Sciences (2076-3417) 10 (2020), 7; 2300, 16

Vrsta, podvrsta i kategorija rada
Radovi u časopisima, članak, znanstveni

Ključne riječi
sign language ; multimodal gesture recognition ; home automatization ; 3D convolution

Sažetak
Deaf and hard-of-hearing people are facing many challenges in everyday life. Their communication is based on the use of a sign language, and the ability of the cultural/social environment to fully understand such a language defines whether or not it will be accessible for them. Technology is a key factor that has the potential to provide solutions to achieve a higher accessibility and therefore improve the quality of life of deaf and hard-of- hearing people. In this paper, we introduce a smart home automatization system specifically designed to provide real- time sign language recognition. The contribution of this paper implies several elements. Novel hierarchical architecture is presented, including resource-and-time- aware modules—a wake-up module and high-performance sign recognition module based on the Conv3D network. To achieve high-performance classification, multi-modal fusion of RGB and depth modality was used with the temporal alignment. Then, a small Croatian sign language database containing 25 different language signs for the use in smart home environment was created in collaboration with the deaf community. The system was deployed on a Nvidia Jetson TX2 embedded system with StereoLabs ZED M stereo camera for online testing. Obtained results demonstrate that the proposed practical solution is a viable approach for real-time smart home control.

Izvorni jezik
Engleski

Znanstvena područja
Računarstvo, Informacijske i komunikacijske znanosti

POVEZANOST RADA