Real-time multimodal fatigue detection using facial vision and alert integration via ESP32 for occupational health applications

Abstract

Early detection of work fatigue is a major challenge in industrial settings due to the lack of non-invasive, accessible, and low-cost systems capable of operating in real time. In this context, this research proposes a multimodal real-time fatigue detection system using facial vision and artificial intelligence, aimed at risk prevention and promoting occupational health. The system integrates geometric and behavioral parameters, such as eye aspect ratio (EAR), head tilt, and mouth opening, processed on a Raspberry Pi 5 using MediaPipe and a hybrid convolutional neural network (CNN) MobileViT model. Visual and audible alerts are managed by an ESP32 microcontroller using the message queuing telemetry transport (MQTT) protocol, while a graphical interface developed in Tkinter allows real-time monitoring of operator status. Experimental results, evaluated in a simulated work environment using AI-generated synthetic videos, show an accuracy greater than 97% and a latency of less than 250 ms, confirming the system's effectiveness in the early detection of signs of drowsiness and attention deficit. In conclusion, the proposal represents a non-invasive, scalable, and efficient solution that combines computer vision, deep learning, and the Internet of Things (IoT) to strengthen workplace safety and well-being.