Indonesian J our nal of Electrical Engineering and Computer Science V ol. 36, No. 2, No v ember 2024, pp. 1199 ∼ 1208 ISSN: 2502-4752, DOI: 10.11591/ijeecs.v36.i2.pp1199-1208 ❒ 1199 An impr o v ed student’ s facial emotions r ecognition method using transfer lear ning Amimi Rajae, Radgui Amina, Ibn El Haj El hassane CEDOC2IT , National Institute of Posts and T elecommunications, Rabat, Morocco Article Inf o Article history: Recei v ed Mar 19, 2024 Re vised Aug 1, 2024 Accepted Aug 5, 2024 K eyw ords: Digital classroom FER system Students af fect states T ransfer learning ABSTRA CT Instructors endea v our to encourage acti v e participation and interaction among learners. Ho we v er , in settings with a lar ge number of students, such as uni v er - sities or online platforms, obtaining real-time feedback and e v aluating teach- ing methodology presents a signicant challenge. In this paper , we intro- duce a student eng agement recognition system based on a h ybrid method us- ing handcrafted features and transfer learning. The research is conducted on tw o databases for emotion detection based on f acial cues (FER13) benchmark ed dataset and our database. W e us e the local binary patterns (LBP) method com- bined with pre-traine d MobileNet model for feature e xtraction and classication. The proposed system adeptly discerns students’ f acial e xpressions and cate go- rizes their eng agement sta tes as either ‘eng aged’ or ‘diseng aged’. W e determine the most ef fecti v e model by e v aluating and comparing se v eral deep learning models, including Inception-V3, V GG16, Ef cientNet, ResNet, and DenseNet. Experimental results underscore the ef cac y of our approach, re v ealing a re- markable accurac y , surpassing benchmarks set by state-of-the-art models. This is an open access article under the CC BY -SA license . Corresponding A uthor: Amimi Rajae CEDOC2IT , National Institute of Posts and T elecommunications Rabat, Morocco Email: amimi.rajae004@gmail.com 1. INTR ODUCTION Recognizing and addressing students’ emoti ons in the learning en vironment is crucial for enhancing their o v erall educational e xperience. Pre vious research, such as that by Leon y et al. [1], has conrmed the potential benets of intelligent systems in pro viding educators with insights int o learners’ emotional states. While the subject of e v aluating student eng agement through f acial e xpres sion analysis has a rich history , the de v elopment of ef cient and straightforw ard methods has been limited. Current student f acial e xpression de- tection methods f ace signicant challenges: machine learning ( ML) algorithm s pro vide limited results, while deep learning (DL) approaches demand e xtensi v e datasets [2]. Therefore, there is a need for methodologies that can ef fecti v ely handle smaller datasets while maintaining high accurac y . Whitehill et al. [3] analyzed eng agement le v els in students using f acial information by emplo ying support v ector machines (SVM) along with Gabor lters for feature e xtraction and classication. The dataset utilized in their research represents a single person captured in a single frame via a webcam. The accurac y of their method is limited, and their study is suited only for online learning settings, thus lacking applicability in v arious learning en vironments. Similarly , T ang et al. [4] introduced an inno v ati v e and ef cient prototype sys- tem. The y emplo yed uniform local g abor binary pattern histogram sequence (ULGBPHS) for feature e xtraction and K-nearest neighbors (KNN) for classication. J ournal homepage: http://ijeecs.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
1200 ❒ ISSN: 2502-4752 The y found similar results to those obtained by Whitehill et al. [3]. Consistent with these ndings, Monkaresi et al. [5], the y ha v e collected a spontaneous database of students’ f acial e xpressions while inter - acting with a g ame-based ph ysics education tool. using the LBP-T OP algorithm, the y classied the de gree of students’ attention based on their f acial e xpressions b ut their study yielded modest accurac y . These early studies established the groundw ork by emplo ying ML algorithms for feature e xtraction and classication. Despite their ef forts, results rema ined modest, primarily due to limitations in dataset size and the use of traditional ML approaches. The breakthrough came with Kim et al. [6] comprehensi v e study , which introduced DL into the eld of FER for education, signalling a paradigm shift. DL methods consistently e xhibited higher accurac y rates, b ut the y necessitated e xtensi v e datasets, a challenge often sidestepped by researchers who resorted to generalized f acial emotion databases. T o bridge this g ap, a fe w recent studies ha v e focused on creating tted datasets for student eng agement recognition. F or instance, in 2019, Ashwin a n d Guddeti [7] proposed a methodology emplo ying a con v olutional neural netw ork (C NN) based on the GoogleNet architecture. The model, trained on both posed and spontaneous databases, achie v ed competiti v e results, though it w as not v eried on small and medium-sized datasets, b ut it does re v eal the ef cienc y of DL methods. Another study by Summer et al. [8] adopted af fect-Net as the foundational architecture. Their model w as trained on a lar ge spontaneous dataset featuring three eng agement le v els (lo w , medium, and high). These ndings underscore the ef cac y of deep learning approaches, specically using CNN architectures lik e GoogleNet and ResNet-50, across di v erse scenarios and datasets, enhancing the accurac y of FER systems. In this paper , we propose a h ybrid approach t hat combines traditional handcrafted features with pre- trained DL algorithms, tted to the task of student eng agement recognition. By e xploring the strengths of both methodologies, we aim to o v ercome the challenges posed by dataset scarcity while enhancing the precision and accurac y of FER systems in educational settings. Our k e y contrib utions include the de v elopment of an optimized FER model based on transfer learn- ing and the creation of a specialized database carefully annotated for reliable model training and assessment. These af rmati v e adv ancements contrib ute to the comprehensi v e goal of enhanci ng real-time student eng age- ment recognition systems within educational settings. W e or g anize this paper as follo ws: section 2 outlines the research methodology , and section 3 presents the e xperimental results and discussion, in addition to a comprehensi v e conclusion of the article and suggestions of potential w ork for future research. 2. PR OPOSED METHOD Our approach combines MobileNet, a lightweight con v olutional neural netw ork, with LBP feature e xtraction. MobileNet ef ciently processes image data, while LBP captures local te xture information. This fusion allo ws for rob ust and ef cient classication, particularly in tasks lik e f acial emotion recognition, where subtle te xture v ariations are crucial. T o train and v alidate the proposed eng agement detection model using LBP-MobileNet, we collected a customised dataset in a real classroom setting. This dataset is specically designed to capture the nuanced f acial e xpressions and emotions e xhibited by students during educational acti vities, ensuring its rele v ance and applicability to our research conte xt. Subsequently , we rigorously tested and e v aluated our approach on the FER13 dataset, a widely recognized benchmark dataset used by numerous researchers in the eld of emotion detection. By conducting e xperim ents on this established dataset, we aim to assess the generalizability and performance of our model across di v erse scenarios and datasets. 2.1. FER13 database The FER2013 [9] dataset comprises 28,709 images annotated with se v en distinct emotions: anger , disgust, fear , happiness, sadness, surprise, and neutral. These images were sourced from the internet, pro viding a di v erse collection. Notably , this dataset serv es as a pre v alent benchmark in the eld of f acial e xpression recognition, frequently referenced in research papers for training and e v aluating models. 2.2. Own database Student f acial e xpression databases remain limited, and most authors train their models using general FER databases. Therefore, ha ving an adequate database is crucial to enhance the models’ accurac y and pro vide impro v ed outcomes [10]. Indonesian J Elec Eng & Comp Sci, V ol. 36, No. 2, No v ember 2024: 1199–1208 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1201 The f acial e xpre ssion database we’ v e de v eloped focuses on capturing spontaneous e xpressions rel ated to eng agement, classifying them into tw o cate gories: “eng aged” and “diseng aged. ” The dataset is substantial, comprising 2 hours of video recordings, 718 f acial images, and in v olving 11 participants, ensuring a balanced demographic with 5 male s and 6 females. The f acial e xpressions were captured using a high-resolution 50 MP camera from a frontal vie w , pro viding detailed and consistent imagery . T o enrich the dataset, a combination of manual and e xpert labelling methods w as emplo yed. This comprehensi v e database serv es as a v aluable resource for the de v elopment and e v aluation of eng agement recognition models. In Figure 1, we present se v eral samples from our dataset. Notably , a unique feature of this dataset i s its inclusion of f aces with v arious occlusions, such as beards, scarv es, dark er skin tones, and instances where indi viduals ha v e their hands on their f aces. These distincti v e attrib utes contrib ute to impro ving our suggested model’ s performance. W e will mak e our database a v ailabl e online to be used in further researches hence we will refer to i t as “StuEmo24: Students’ Emotions - Student F acial Expression Dataset - 2024”. This dataset may signicantly adv ance research in areas lik e emotion recognition systems and human-computer interaction by pro viding a useful resource for training and assessing machine learning models in af fect recognition tasks. (a) (b) Figure 1. Samples of images from our dataset including tw o classes (a) eng aged and (b) diseng aged 2.3. F ace detection F ace tracking and detection in computer vision are typically challenging problems that need adv anced methods for accurate f ace recognition. Numerous approaches, including the HAAR cascade classier [11], histogram of oriented gradient ( HOG) [12], LBP [13], and its v ariants, ha v e demonstrated remarkable f ace detection performance. Ho we v er , when f aces are ti lted, occluded, or multiple f aces are present in an image, con v entional methods lik e HAAR Cascade and pre viously cited approaches, e xhibit limitations. In the con- te xt of our study , which focuses on analyzing students’ eng agement in a classroom setting based on f acial e xpressions, a rob ust method is imperati v e for recognizing f aces under challenging conditions. T o address this, we ha v e adopted the single-shot multi-le v el f ace locali sation in the wild (RetinaF ace) method proposed by Deng et al. [14]. This method has sho wn ef fecti v eness i n detecting f aces under v arious conditions, of fering high accurac y e v en in comple x scenarios. 2.4. Data pr ocessing After the f ace detection and annotation phase, StuEmo24 comprises 457 images cate gorized as “eng aged” and 261 images as “diseng aged. ” T o f acilitate training and testing, we di vided the images into a training set (90%) and a test set (10%) for each cate gory . Gi v en the modest size of our dataset, which might not be optimal for training a neural netw ork from sc ratch, we opted for image augmentation using the K eras library . This strate gy ef fecti v ely enlar ges the dataset before inte grating it into the netw ork to a v oid model o v ertting and non-con v er gence. The augmentation process we used included rotation within a 15-de gree range, zooming with a 0.15 range, shearing with a scale of 0.15, horizontal ipping, and width and height shifting with a v alue of 0.15. These operations collecti v ely contrib ute to a more di v erse and rob ust training dataset. Ho we v er , recognizing the limitations of data augmentation in achie ving a suf ciently e xtensi v e dataset, we recommend considering An impr o ved student’ s facial emotions r eco gnition method ... (Amimi Rajae) Evaluation Warning : The document was created with Spire.PDF for Python.
1202 ❒ ISSN: 2502-4752 the transfer learning technique outlined in the follo wing section. T ransfer learning, as demonstrated in v arious classication tasks [15], has pro v en ef fecti v e with medium size datasets. 2.5. F eatur e extraction Utilizing f acial images for g auging emoti on s in v olv es the cautious detection of minor changes in f acial landmarks. F or instance, transit ions in the edges, corners, and contours of the e yes, shifting from closed to wide open, can indicate a shift from a dull e xpression to one of focused concentration. These al terations in k e y f acial re gions of interest (R OIs), co v ering the left and right e yebro ws, e yes, nose, and mouth, are ltered through the lens of the LBP . LBP serv e as a method for e xtracting features in image processing, particularly adept at capturing local te xture patterns by assessing pix el intensities within a specied neighbourhood. LBP’ s resilience to illumination changes enhances model rob ustness, making it a f a v ourable choice. In this process, we compare each pix el’ s intensity with its neighbours and assign binary v alues based on the comparison, resulting in an LBP code for the central pix el, as in (1). This code represents the te xture pattern within the neighbourhood. By repeati ng this process for e v er y pix el in the image, we generate a comprehensi v e LBP representation, f acilitating the e xamination of the image’ s te xture properties. Then we feed this representat ion into our neural netw ork. In our e xperimental setup, images are captured under v arying lighting conditions, posing challenges to model performance. Ho we v er , the utilization of LBP resolv e this issue by making images insensiti v e to illumination changes, thereby enhancing the rob ustness of the model. Moreo v er , LBP’ s reliance on simple comparison and thresholding operations ensures computational ef cienc y , rendering it suitable for real-time application. Furthermore, LBP’ s ability to ef fecti v ely reduce the dimensi on a lity of the feature space simplies classier operations and mitig ates the risk of o v ertting, thus contrib uting to the o v erall ef fecti v eness of the model. Figure 2 sho ws the application of LBP on StuEmo24. LB P ( p ) = P − 1 X i =0 s ( i ) × 2 i (1) Where: - LBP(p) is the local binary pattern for the centre pix el p. - P is the number of sampling points in the neighbourhood N. - s(i) represents the result of the intensity comparison between the centre pix el p and its i-th neighbour . (a) (b) Figure 2. LBP applied on StuEmo24 (a) procedure of LBP method and (b) samples from StuEmo24 with the application of LBP on f ace images Indonesian J Elec Eng & Comp Sci, V ol. 36, No. 2, No v ember 2024: 1199–1208 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1203 2.6. Students’ engagement r ecognition model Deep neural netw orks (DNNs) ha v e played a transformati v e role in the eld of image classicati on ; ho we v er , the y still f ace se v eral challenges that researchers and practitioners continuously w ork to address. Some of the k e y challenges include data quality and quantity , computational resources, and training time. Therefore, transfer learning is a po werful tool that o v ercomes these challenges, enabling the ef fecti v e reuse of kno wledge from pre-trained models and adapting to ne w tasks. In our study , we tested dif ferent CNNs architectures based on pre-trained models on ImageNet [16] database that comprises more than 14 million images manually annotated. W e ha v e e v aluated the follo wing state-of-the-art algorithms: Ef cientNet, DenseNet, InceptionV3, ResNet, V GG16, and MobileNet. Our proposed model initializes a neural netw ork using the MobileNet [17] architecture as sho wn in Figure 3. MobileNet is a lightweight deep neural netw ork based on depthwise con v olution that applies a separate con v olutional lter to each input channel, leading to a reduction in computation while still capturing spatial information across channels. Its mathematical formulation is as follo ws: Gi v en an input feature map X with depth D i , a depthwise con v olution lter W with depth D f , and spatial dimensions K × K , the depthwise con v olution operation at spatial location ( i, j ) can be represented as (2): D C onv ( i, j , d f ) = K − 1 X k =0 K − 1 X l =0 X ( i + k , j + l , d i ) × W ( k , l , d f ) (2) where: - X(i+k, j+l, d i ) represents the input feature map v alue at spatial location ( i + k , j + l ) and depth d i . - W(k, l, d f ) represents the lter v alue at spatial location ( k , l ) and depth d f . - DCon v(i, j, d f ) is the output feature map v alue at spatial location ( i, j ) and depth d f . The model is congured with a 48x48 input shape, e xcluding fully connected layers, and le v erages pre-trained weights. T o capture intermediate features, we e xtracted the output of the 14th layer from the end of the base model. Subsequently , we applied global max pooling to this chosen layer , reducing spatial dimensions and retaining essential features. W e emplo yed a batch size of 25 images and applied SoftMax as the acti v ation function to the ou t put of the fully connected layer (3), along with the Adam optimizer . Gi v en an input v ector z , SoftMax computes the output v ector y as (3): y i = e z i P n j =1 e z j (3) where: - e is the base of the natural log arithm. - z i is the i -th element of the input v ector z . - P n j =1 e z j is the sum of the e xponential v alues of all elements in the input v ector . Our choice of loss function w as cate gorical cross-entrop y . The cate gorical cross-entrop y loss funct ion is commonly used in multi-class classication tasks. It measures the dissimilarity between the true distrib ution of the data and the predicted distrib ution outputted by the model. Here we used it for binary classication by treating it as a special case where there are only tw o class es. In this case, we represent the true labels as one-hot encoded v ectors with tw o elements. Mathematically , the cate gorical cross-entrop y l oss function for binary classication can be as (4): CCE = − N X i =1 y i · log ( p i ) (4) where: • N is the number of samples • y i is the true label (either 0 or 1) • p i is the predicted probability that the sample belongs to class 1 The model w as trained o v er 100 epochs. Ho we v er , we noted that the accurac y plateaued starting from the 25th epoch. An impr o ved student’ s facial emotions r eco gnition method ... (Amimi Rajae) Evaluation Warning : The document was created with Spire.PDF for Python.
1204 ❒ ISSN: 2502-4752 Figure 3. Proposed model architecture 3. RESUL TS AND DISCUSSION 3.1. Experimental set-up W e em plo y a h ybrid ML-DL approach utilizing the T ensorFlo w and K eras libraries for model training. T o enhance ef cienc y , we adopt a transfer learning model based on a pre-trained neural netw ork, ensuring f aster and more ef fecti v e results. The model training is conducted on gradient [18], a platform designed for constructing and scaling real-w orld machine learning applications. The training en vironment includes a P5000 machine equipped with T ensorFlo w v ersion 2.9.11, 30 GB of RAM, an 8 GB CPU, and a 16 GB GPU. T o ne-tune the learning process, impro v e ef cienc y , and pre v ent o v ertting, we used tw o callbacks from the K eras library: the EarlyStopping callback monitors v alidation accurac y , and stops the training process once the model achie v es the highest accurac y , sa ving time and pre v enting o v ertting. The ReduceLR OnPlateau callback focuses on adjusting the learning rate during training, monitoring v alidation accurac y , and reducing t he learning rate if accurac y stops impro ving. T able 1 describes the model features details. T able 1. Model features details Feature Batch size Epochs Acti v ation function Optimizer LR Loss function Details 25 20 SoftMax Adam 0.001 Cate gorical cross entrop y 3.2. Analysis of the r esults Our study in v estig ated the performance of a DL model combined with an ML algorithm trained on a medium-sized dataset. Pre vious studies ha v e typicall y applied either ML approaches to medium-sized datasets, yielding modest results, or DL approaches to v ery lar ge b ut not conte xtually adapted datasets, resulting in outcomes that are not applicable to real classroom settings. None of the e xisting literature e xplicitly addresses the application of performant DL algorithms on small or medium-s ized datasets. By e xploring the strengths of both methodologies, we aim to o v ercome the challenges posed by dataset scarcity while enhancing the precision and accurac y of SFER systems. Figure 4, demonstrates that by combining the LBP method along with the pre-trained mobileNet, the model e xhibits uctuations in performance during the rst 10 epochs of training. Ho we v er , o v er subsequent epochs, the accurac y steadily increases until reaching a remarkable accurac y of 98%. This achie v ement rep- resents a signicant impro v ement o v er pre vious benchmarks, as e videnced by the best achie v able accurac y of only 92.3% using ResNet-50 as reported by [19]. This enhanced accurac y holds practical implications, partic- ularly in real-time applications where the ability to accurately detect students’ emotional states through f acial e xpressions is of prime importance for pro viding real-time feedback. Furthermore, we e v aluated the model on the FER13 dataset, where it demonstrated strong perfor - mance, achie ving an accurac y of 73.6%, as illust rated in Figure 5. This underscores the rob ustness and gener - alization capabilities of our approach across dif ferent datasets and tasks. Indonesian J Elec Eng & Comp Sci, V ol. 36, No. 2, No v ember 2024: 1199–1208 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1205 (a) (b) Figure 4. Optimized model accurac y and loss rate curv e on our dataset (a) model accurac y curv e and (b) model loss curv e (a) (b) Figure 5. Optimized model accurac y and loss rate curv e on FER13 dataset (a) model accurac y curv e and (b) model loss curv e Our e xperimentation in v olv ed testing v arious models, including Ef cientNet and InceptionNet, all of which pro vided only moderate results, f ailing to surpass 64% accurac y e v en with the inte gration of LBP feature e xtraction. The lo wer accurac y of Ef cientNet (57%) and InceptionNet (64%) might be due to their trade-of f between ef cienc y and comple xity . While the y are designed to be computationally ef cient, the y may struggle to capture nuanced features necessary for accurate f acial emotion recognition. In contrast , emplo ying the V GG16 and DenseNet architectures resulted in a substantial increase in accurac y . Building upon this success, we continued to iterate by incorporating LBP treatment and e xploring dif ferent neural netw ork congurations. Figure 6 sho ws the results. Ultimately , our ef forts come to an end with the adoption of MobileNet coupled with LBP , achie ving an e xceptional accurac y of 98.2% on StuEmo24 dataset. This achie v ement represents a signicant adv ancement, surpassing state-of-the-art results in the eld. Figure 6. Accurac y of v arious models applied on our dataset An impr o ved student’ s facial emotions r eco gnition method ... (Amimi Rajae) Evaluation Warning : The document was created with Spire.PDF for Python.
1206 ❒ ISSN: 2502-4752 3.3. Comparison with state-of-the-art r esults The l iterature demonstrates that handcrafted features e xhibit limited ef cac y in detecting student en- g agement through f acial e xpressions in real-w orld settings [3], [5] particularly when compared to deep learning methods [19]-[21]. Our h ybrid approach outperforms e xisting methods, achie ving superior results. Ho we v er , direct comparison with prior methodologies is challenging due to v ariations in datasets and modalities. Despite these challenges, our results demonstrate comparable and enhanced accurac y and rob ustness. Pre vious research by Ashwin and Guddeti [7] emplo yed a h ybrid approach utilizing solely deep learn- ing models as backbones, pro viding promising results ranging from 70% to 81% accurac y . Additionally , [19] utilized ResNet50 as the base model, achie ving an impressi v e 92.3% accurac y . These ndings encouraged us to e xplore di v erse deep-learning models as backbone architectures. Although ML models ha v e sho wn promis- ing b ut insuf cient results, as demonstrated by Whitehill et al. [3], who a chie v ed an A UC of 0.729 usi ng Gabor features in conjunction with SVM. While Gabor features are well-suited for tasks necessitating te x- ture and shape analysis, such as object detection, ngerprint recognition, and biomedical image analysis, we opted for LBP due to their ef fecti v eness in tasks emphasizing te xture patt erns, including te xture classication, f ace recognition, and emotion recognition. By training MobileNet on the pertinent features e xtracted from LBP , we ef fecti v ely impro v e the nuances of underlying signal uctuations crucial for eng agement recognition. This methodology pro vides notable impro v ements in classication accurac y , distinguishing our approach as a promising adv ancement in the eld. T able 1 illustra tes the results of our method in comparison with recent methodologies. T able 2. Comparison of accurac y of state-of-the-art FER models Author Method Accurac y Ashwin and Guddeti [7] T w o CNN architectures: CNN1 + CNN2 CNN1: 86%, CNN2: 70% Ma et al. [21] CNN 76.9% Gupta et al. [19] ResNet-50 92.3% P abba and K umar [20] Impro v ed v ersion of mini-Xception 72.68% Shen et al. [22] Light weight attentional con v olutional netw ork (SE-D AN) 56% Hu et al. [23] Optimized Shuf eNet v2 63.9% Our method LBP + MobileNet 98.2% Ho we v er , it is important to ackno wledge certain limitations in our approach. Whi le our method demonstrates procienc y in handling occ lusions and v ariations in f acial e xpressions, we ha v e yet to test its performance on a dataset comprising dif ferent lighting condit ions. Lighting v ariations can signicantly impact the accurac y of f acial e xpression recognition models. Although the inte gration of LBP mitig ates the model’ s dependenc y on lighting changes to some e xtent, further e v aluation under di v erse lighting conditions is neces- sary to assess its rob ustness comprehensi v ely . 3.4. A pplication and generalisability of the students’ engagement r ecognition system Our study presents a frame w ork for identifying student eng agement. This system pro vides insightful feedback, which is especially helpful for educators who are just starting out in the eld and w ant to impro v e their instructional strate gies. Furthermore, our method of fers instructors and students personalized content feedback, which can further personalize the teaching and learning process. It also allo ws research into possible relationships between students’ emotional states and their academic achie v ement. Our system can assist instructors in situations lik e webinars and lar ge class sizes by taking care of the dif culty of tracking each student’ s participation in real-time. Our approach is more rob ust than e xisting auto- tutors, and its inte gration into intelligent tutoring systems could impro v e personalized learning e xperiences by replacing traditional methods of predicting emotional states. Due to the spread of CO VID-19, there is a gro wing need to adapt our system to recognize student eng agement e v en when f aces are mask ed. In future w ork, we aim to customize the system to ef fecti v ely consider occluded f aces, particularly those co v ered by masks [24], [25]. Be yond educational settings, our emotional state prediction model may ha v e applications in health- care, where it can assist in patient care by assessing emotional responses to t reatments, therapies, and healthcare en vironments. This could potentially enhance patient satisf action and o v erall healthcare outcomes. Addition- ally , it may of fer insights into w orkplace producti vity and well-being, allo wing for the assessment of emplo yee Indonesian J Elec Eng & Comp Sci, V ol. 36, No. 2, No v ember 2024: 1199–1208 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1207 eng agement, stress le v els, and o v erall well-being in w orkplace en vironments. Utilizing this information, or g a- nizations can optimize w ork en vironments and policies to enhance producti vity and emplo yee satisf action. In each domain, the adaptation of this system necessitates the re-engineering of conte xtual features customized to the specic requirements and conditions of that domain. 4. CONCLUSION The primary objecti v e of this study w as to de v elop a frame w ork for identifying student eng agement through f acial e xpressions, utilizing transfer learning. The personalized insights g ained from monitoring stu- dent eng agement ha v e the potential to enhance instructional strate gies and of fer more eng aging educational e xperiences. In addressing the challenges posed by lo w accurac y in machine learning methods and the need for lar ger datasets in deep learning approaches, our research demonstrated that the fusion of handcrafted features, particularly LBP , with the pre-trained MobileNet model leads to signicant success on medium-sized FER datasets. Through rigorous training with data augmentation and e v aluation ag ai nst the FER13 benchmark ed dataset, this h ybrid method e xhibited rob ustness in predicting learners’ emotional states across di v erse learn- ing en vironments. Mo ving forw ard, it is recom mended that future research w orks on e xpanding the dataset to i nclude images of participants from di v erse cultural backgrounds and wider age ranges. Additionally , in- te grating cogniti v e eng agement assessments, such as cognit i v e quizzes, could clarify the relationship between students’ eng agement states and their academic outcomes. By addressing these recommendations and continu- ing to rene our methodologies, we can further adv ance the understanding and application of f acial e xpression recognition systems in educational conte xts and be yond where analyzing user emotions is critical, such in adv ertisement domain, online shopping and video consumption. REFERENCES [1] D. Leon y , P . J. Mu ˜ noz-Merino, A. P ardo, and C. D. Kloos , “Pro vision of a w areness of learners’ emotions through visual iza- tions in a computer interaction-based en vironment, ” Expert Systems with Applications , v ol. 40, no. 13, pp. 5093–5100, 2013, doi: 10.1016/j.esw a.2013.03.030. [2] A. De v arapalli and J. M. Gonda, “In v estig ation into f acial e xpression recognition methods: a re vie w , ” Indonesian Journal of Elec- trical Engineering and Computer Science (IJEECS) , v ol. 31, no. 3, pp. 1754–1762, 2023, doi: 10.11591/ijeecs.v31.i3.pp1754-1762. [3] J. Whitehill, Z. Serpell, Y . C. Lin, A. F oster , and J. R. Mo v ellan, “The f aces of eng agement: automatic recognition of stu- dent eng agement from f acial e xpressions, ” IEEE T ransactions on Af fecti v e Computing , v ol. 5, no. 1, pp. 86–98, Jan. 2014, doi: 10.1109/T AFFC.2014.2316163. [4] C. T ang, P . Xu, Z. Luo, G. Zhao, and T . Zou, “ Automatic f acial e xpression analysis of students in teaching en vironments, ” in Lecture Notes in Computer Science (including subseries Lecture Notes in Articial Intelligence and Lecture Notes in Bioinformatics) , v ol. 9428, Springer International Publishing, 2015, pp. 439–447. [5] H. Monkaresi, N. Bosch, R. A. Calv o, and S. K. D’Mello, “ Automated detection of eng agement using video-based estima- tion of f acial e xpressions and heart rate, ” IEEE T ransactions on Af fecti v e Computing , v ol. 8, no. 1, pp. 15–28, Jan. 2017, doi: 10.1109/T AFFC.2016.2515084. [6] Y . Kim, T . So yata, and R. F . Behnagh, “T o w ards emotional ly a w are AI smart classroom: current issues and directions for engineering and education, ” IEEE Access, v ol. 6, pp. 5308–5331, 2018, doi: 10.1109/A CCESS.2018.2791861. [7] T . S. Ashwin and R. M. R. Guddeti, “ Automatic detection of students’ af fecti v e states in classroom en vironment using h y- brid con v olutional neural netw orks, ” Education and Information T echnologies , v ol. 25, no. 2, pp. 1387–1415, Oct. 2020, doi: 10.1007/s10639-019-10004-6. [8] O. Sumer , P . Goldber g, S. Dmello, P . Ge rjets, U. T rautwein, and E. Kasneci, “Multimodal eng agement analysis from f acial videos in the classroom, ” IEEE T ransactions on Af fecti v e Computing , v ol. 14, no. 2, pp. 1012–1027, 2023, doi: 10.1109/T AFFC.2021.3127692. [9] I. J. Goodfello w et al., “Challenges in representation learning: a report on three machine learni ng contests, ” in Neural Netw orks , 2015, v ol. 64, pp. 59–63, doi: 10.1016/j.neunet.2014.09.005. [10] R. Amimi, A. Radgui, and H. Ibn El Haj El, “ A surv e y of smart cl assroom: concept, technologies and f acial emotions recognition application, ” in Lecture Notes in Netw orks and Systems , 2023, v ol. 544 LNNS, pp. 326–338, doi: 10.1007/978-3-031-16075-2 23. [11] P . V iola and M. Jones, “Rapid object detection using a boosted cascade of simple features, ” in Proceedings of the IEEE Computer Society Conference on Computer V ision and P attern Recognition , 2001, v ol. 1, pp. I-511-I–518, doi: 10.1109/cvpr .2001.990517. [12] N. Dalal and B. T riggs, “Histograms of oriented gradients for human detection, ” in 2005 IEEE Computer Society Conference on Computer V ision and P attern Recognition (CVPR’05) , v ol. 1, pp. 886–893, doi: 10.1109/CVPR.2005.177. [13] T . Ojala, M. Pietik ¨ ainen, and T . M ¨ aenp ¨ a ¨ a, “Gray scale and rotation in v ariant te xture classication with local binary patterns, ” in Lec- ture Notes in Computer Science (including subseries Lecture Notes in Articial Intelligence and Lecture Notes in Bioinformatics) , 2000, v ol. 1842, pp. 404–420, doi: 10.1007/3-540-45054-8 27. [14] J. Deng, J. Guo, Y . Zhou, J. Y u, I. K otsia, and S. Zafeiriou, “RetinaF ace: single-s tage dense f ace localisation in the wi ld, ” arXi v preprint arXi v:1905.00641 , 2019, [Online]. A v ailable: http://arxi v .or g/abs/1905.00641. [15] M. Zanotti, “T ransfer learning in image classication: ho w much training data do we really need?, ” T o w ards data sci ence . 2020, Accessed: Aug. 29, 2022. [Online]. A v ailable: https://to w ardsdatascience.com/transfer -learning-in-image-classication-ho w-much- training-data-do-we-really-need-7fb570abe774. An impr o ved student’ s facial emotions r eco gnition method ... (Amimi Rajae) Evaluation Warning : The document was created with Spire.PDF for Python.
1208 ❒ ISSN: 2502-4752 [16] A. Krizhe vsk y , I. Sutsk e v er , and G. E. Hinton, “ImageNet classication with deep con v olutional neural netw orks, ” Adv ances in Neural Information Processing Systems , v ol. 2, pp. 1097–1105, 2012. [17] A. G. Ho w ard et al. , “MobileNets: ef cient con v olutional neural netw orks for mobile visi on applications. ” arXi v preprint , 2017, doi: 10.48550/arXi v .1704.04861. [18] J . Caron and J. R. Markusen, “P aperspace. ” pp. 1–23, 2016, [Online]. A v ailable: https://docs.paperspace.com/. [19] S . Gupta, P . K umar , and R. K. T ekchandani, “F acial emotion recognition based real-time learner eng agement detection system in online learning conte xt using deep learning models, ” Multimedia T ools and Applications , v ol. 82, no. 8, pp. 11365–11394, 2023, doi: 10.1007/s11042-022-13558-9. [20] C . P abba and P . K umar , “ An intelligent system for monitoring students’ eng agement in lar ge classroom teaching through f acial e xpression recognition, ” Expert Systems , v ol. 39, no. 1, p. e12839, 2022, doi: 10.1111/e xsy .12839. [21] S . Ma et al., “ A f ace sequence recognition method based on deep con v olutiona l neural netw ork, ” in Proceedings - 2019 18th International Symposium on Distrib uted Computing and Applications for Business Engineering and Science , DCABES 2019, 2019, pp. 104–107, doi: 10.1109/DCABES48411.2019.00033. [22] J. Shen, H. Y ang, J. Li, and Z. Cheng, “ Assessing learni ng eng agement based on f acial e xpression recognition in MOOC’ s scenario, ” Multimedia Systems , v ol. 28, no. 2, pp. 469–478, 2022, doi: 10.1007/s00530-021-00854-x. [23] Y . Hu, Z. Jiang, and K. Zhu, “ An optimized CNN model for eng agement recognition in an e-le arning en vironment, ” Applied Sciences (Switzerland) , v ol. 12, no. 16, p. 8007, 2022, doi: 10.3390/app12168007. [24] B. Hdioud and M. E. H. T irari, “F acial e xpression recognition of mask ed f aces using deep learning, ” IAES International Journal of Articial Intelligence IJ-AI , v ol. 12, no. 2, pp. 921–930, Jun. 2023, doi: 10.11591/ijai.v12.i2.pp921-930. [25] A. Rajae, R. Amina, and I. El Haj El Hassane, “ T o w ards f ace-to-f ace smart classroom that adheres to co vid’19 restrictions, ” in Lecture Notes in Netw orks and Systems , 2023, v ol. 625 LNNS, pp. 47–58, doi: 10.1007/978-3-031-28387-1 5. BIOGRAPHIES OF A UTHORS Amimi Rajae is a researcher at the National Institute of Posts and T elecommunications in Morocco. She holds an engineering diploma in Netw orks and Information T echnologies with a specialization in Multimedia. Amimi’ s research interests span across image processing, machine learning, image analysis, and pattern recognition. W ith a passion for adv ancing kno wledge and inno- v ation, she is dedicated to e xploring the intersection of technology and education to enhance learning e xperiences. Amimi acti v ely contrib utes to research in he r eld and seeks to mak e meaningful con- trib utions to the academic community . F or inquiries or colla borations. She can be contacted at email: amimi.rajae004@gmail.com. Radgui Amina is an associate professor in computer vision at INPT Morocco. She obtained the M.Sc. and the Ph.D. de grees in computer sciences and telecommunications in 2004 and 2010, respecti v ely , at the Mohammed V Uni v ersity . In 2011, she joined the Insti tute of Posts and T elecommunications (INPT) as assistant professor . She is also a member of the multimedia, signal, and communication systems (MUSICS) research team, department at INPT since 2011. Her research interests include computer vision, i mage processing, and the related applications for medical research. She can be contacted at email: radgui@inpt.ac.ma. Ibn El Haj El Hassane is a professor of multimedia signal processing and communica- tions at the INPT , Rabat, Morocco, from 2000 until no w . He is also member of the MUSICS research group at the INPT . He is senior member of IEEE. His research interests include digital signal, speech processing, image and video processing, digital w atermarking and data hiding, deep learning, cogni- ti v e radio, and multimedia communications. He can be contacted at email: ibnelhaj@inpt.ac.ma. Indonesian J Elec Eng & Comp Sci, V ol. 36, No. 2, No v ember 2024: 1199–1208 Evaluation Warning : The document was created with Spire.PDF for Python.