Top

Published in:

2024 | OriginalPaper | Chapter

Multimodal Attention CNN for Human Emotion Recognition

Authors : Gyanendra Tiwary, Shivani Chauhan, Krishan Kumar Goyal

Published in: Cryptology and Network Security with Machine Learning

Publisher: Springer Nature Singapore

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The human face is the mirror of the mind. The face generally tells all that is going on in one’s heart and mind. Just by looking at the faces of our known ones, we may easily guess their mood. But many times, when we meet some unfamiliar person, it’s hard to get his or her mood just by looking at their faces. This is just because the person may have a certain facial structure that makes them by default look angry, happy, or sad. So, we need to spend some time with that person to analyse other parameters before concluding their state of mood. The current work proposed a novel approach that integrates facial images with electroencephalography (EEG) signals for facial expression recognition tasks. When attention-based deep CNN analyses the facial traits of the subject, a parallel Long Short-Term Memory (LSTM) network analyses the EEG signals. A late fusion network combines the features extracted from both networks, and finally, a classification network tells about what is the current mood of the subject. Combining multiple modalities for emotion recognition has shown promising results when compared with other state-of-the-art models. There are multiple real-life applications of emotion recognition models, such as Advertisement Industry, Human–Robot Interaction, Automatic Depression Detection, Mood Audio/Video Players, etc.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter A Comprehensive Review on Transforming Security and Privacy with NLP

next chapter Supervised Learning Approaches for Deceit Identification: Exploring EEG as a Non-invasive Technique

Wang S, Qu J, Zhang Y, Zhang Y (2023) Multimodal emotion recognition from EEG signals and facial expressions. IEEE Access 11:33061–33068. https://doi.org/10.1109/ACCESS.2023.3263670CrossRef

Yang Y, Gao Q, Song Y, Song X, Mao Z, Liu J (2022) Investigating of deaf emotion cognition pattern by EEG and facial expression combination. IEEE J Biomed Health Inform 26(2):589–599. https://doi.org/10.1109/JBHI.2021.3092412CrossRef

Li D et al (2023) Emotion recognition of subjects with hearing impairment based on fusion of facial expression and EEG topographic map. IEEE Trans Neural Syst Rehabil Eng 31:437–445. https://doi.org/10.1109/TNSRE.2022.3225948CrossRef

Yang J, Qian T, Zhang F, Khan SU (2021) Real-time facial expression recognition based on edge computing. IEEE Access 9:76178–76190. https://doi.org/10.1109/ACCESS.2021.3082641CrossRef

Tang Y, Zhang X, Hu X, Wang S, Wang H (2021) Facial expression recognition using frequency neural network. IEEE Trans Image Process 30:444–457. https://doi.org/10.1109/TIP.2020.3037467CrossRef

Wadhawan R, Gandhi T (2022) Landmark-aware and part-based ensemble transfer learning network for static facial expression recognition from images. In: IEEE transactions on artificial intelligence, pp 1–1. https://doi.org/10.1109/tai.2022.3172272

Lee J, Kim S, Kim S, Sohn K (2020) Multi-modal recurrent attention networks for facial expression recognition. IEEE Trans Image Process 29:6977–6991. https://doi.org/10.1109/TIP.2020.2996086CrossRef

Zheng W, Zong Y, Zhou X, Xin M (2018) Cross-domain color facial expression recognition using transductive transfer subspace learning. IEEE Trans Affect Comput 9(1):21–37. https://doi.org/10.1109/TAFFC.2016.2563432CrossRef

Islam R (2019) Parts of the brain & function. Anatomy Info. https://anatomyinfo.com/parts-of-the-brain/

10.

Jatupaiboon N, Pan-Ngum S, Israsena P (2013) Real-time EEG-based happiness detection system. Sci World J 2013. https://doi.org/10.1155/2013/618649

11.

Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

12.

Tiwary G, Chauhan S, Goyal KK (2022) Video based deep CNN model for depression detection. Int J Recent Innov Trends Comput Commun 10(10):59–64. https://doi.org/10.17762/ijritcc.v10i10.5735CrossRef

13.

Karnati M, Seal A, Bhattacharjee D, Yazidi A, Krejcar O (2023) Understanding deep learning techniques for recognition of human emotions using facial expressions: a comprehensive survey. IEEE Trans Instrum Meas 72. https://doi.org/10.1109/TIM.2023.3243661

14.

Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069. https://doi.org/10.1109/TIP.2019.2956143CrossRef

15.

Farzaneh AH, Qi X (2021) Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision 2021, pp 2402–2411

16.

Li S, Deng W (2022) Deep facial expression recognition: a survey. IEEE Trans Affect Comput 13(3):1195–1215. https://doi.org/10.1109/TAFFC.2020.2981446CrossRef

17.

Zhang F, Zhang T, Mao Q, Xu C (2020) Geometry guided pose-invariant facial expression recognition. IEEE Trans Image Process 29:4445–4460. https://doi.org/10.1109/TIP.2020.2972114CrossRef

18.

Zhang F, Zhang T, Mao Q, Xu C (2020) A unified deep model for joint facial expression recognition, face synthesis, and face alignment. IEEE Trans Image Process 29:6574–6589. https://doi.org/10.1109/TIP.2020.2991549CrossRef

19.

Zhang F, Xu M, Xu C (2022) Weakly-supervised facial expression recognition in the wild with noisy data. IEEE Trans Multimedia 24:1800–1814. https://doi.org/10.1109/TMM.2021.3072786CrossRef

20.

Xia Y, Yu H, Wang X, Jian M, Wang FY (2022) Relation-aware facial expression recognition. IEEE Trans Cogn Dev Syst 14(3):1143–1154. https://doi.org/10.1109/TCDS.2021.3100131CrossRef

21.

Lu Y, Wang S, Zhao W, Zhao Y (2019) WGAN-based robust occluded facial expression recognition. IEEE Access 7:93594–93610. https://doi.org/10.1109/ACCESS.2019.2928125CrossRef

22.

Chen J, Guo C, Xu R, Zhang K, Yang Z, Liu H (2022) Toward children’s empathy ability analysis: joint facial expression recognition and intensity estimation using label distribution learning. IEEE Trans Industr Inform 18(1):16–25. https://doi.org/10.1109/TII.2021.3075989CrossRef

23.

Ali G et al (2020) Artificial neural network based ensemble approach for multicultural facial expressions analysis. IEEE Access 8:134950–134963. https://doi.org/10.1109/ACCESS.2020.3009908CrossRef

24.

Cha HS, Choi SJ, Im CH (2020) Real-time recognition of facial expressions using facial electromyograms recorded around the eyes for social virtual reality applications. IEEE Access 8:62065–62075. https://doi.org/10.1109/ACCESS.2020.2983608CrossRef

25.

Dalvi C, Rathod M, Patil S, Gite S, Kotecha K (2021) A survey of AI-based facial emotion recognition: features, ML DL techniques, age-wise datasets and future directions. IEEE Access 9:165806–165840. https://doi.org/10.1109/ACCESS.2021.3131733CrossRef

26.

Deng J, Pang G, Zhang Z, Pang Z, Yang H, Yang G (2019) CGAN based facial expression recognition for human-robot interaction. IEEE Access 7:9848–9859. https://doi.org/10.1109/ACCESS.2019.2891668CrossRef

27.

García M, Ramírez S (2020) Deep neural network architecture: application for facial expression recognition; deep neural network architecture: application for facial expression recognition

28.

He Y, Chen S (2020) Person-independent facial expression recognition based on improved local binary pattern and higher-order singular value decomposition. IEEE Access 8:190184–190193. https://doi.org/10.1109/ACCESS.2020.3032406CrossRef

29.

Huang W, Zhang S, Zhang P, Zha Y, Fang Y, Zhang Y (2022) Identity-aware facial expression recognition via deep metric learning based on synthesized images. IEEE Trans Multimedia 24:3327–3339. https://doi.org/10.1109/TMM.2021.3096068CrossRef

30.

Jiang P, Liu G, Wang Q, Wu J (2020) Accurate and reliable facial expression recognition using advanced softmax loss with fixed weights. IEEE Signal Process Lett 27:725–729. https://doi.org/10.1109/LSP.2020.2989670CrossRef

31.

Jiang P, Wan B, Wang Q, Wu J (2020) Fast and efficient facial expression recognition using a gabor convolutional network. IEEE Signal Process Lett 27:1954–1958. https://doi.org/10.1109/LSP.2020.3031504CrossRef

32.

Karnati M, Seal A, Yazidi A, Krejcar O (2022) FLEPNet: feature level ensemble parallel network for facial expression recognition. IEEE Trans Affect Comput 13(4):2058–2070. https://doi.org/10.1109/TAFFC.2022.3208309CrossRef

33.

Khan S, Chen L, Yan H (2020) Co-clustering to reveal salient facial features for expression recognition. IEEE Trans Affect Comput 11(2):348–360. https://doi.org/10.1109/TAFFC.2017.2780838CrossRef

34.

Kim DH, Baddar WJ, Jang J, Ro YM (2019) Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput 10(2):223–236. https://doi.org/10.1109/TAFFC.2017.2695999CrossRef

35.

Kim JH, Kim BG, Roy PP, Jeong DM (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285. https://doi.org/10.1109/ACCESS.2019.2907327CrossRef

36.

Kulkarni K et al (2021) Automatic recognition of facial displays of unfelt emotions. IEEE Trans Affect Comput 12(2):377–390. https://doi.org/10.1109/TAFFC.2018.2874996CrossRef

37.

Kuruvayil S, Palaniswamy S (2022) Emotion recognition from facial images with simultaneous occlusion, pose and illumination variations using meta-learning. J King Saud Univ Comput Inf Sci 34(9):7271–7282. https://doi.org/10.1016/j.jksuci.2021.06.012CrossRef

38.

Li B, Lima D (2021) Facial expression recognition via ResNet-50. Int J Cogn Comput Eng 2:57–64. https://doi.org/10.1016/j.ijcce.2021.02.002CrossRef

39.

Li H, Wang N, Ding X, Yang X, Gao X (2021) Adaptively learning facial expression representation via C-F labels and distillation. IEEE Trans Image Process 30:2016–2028. https://doi.org/10.1109/TIP.2021.3049955CrossRef

40.

Li H, Wang N, Yang X, Gao X (2022) CRS-CONT: a well-trained general encoder for facial expression analysis. IEEE Trans Image Process 31:4637–4650. https://doi.org/10.1109/TIP.2022.3186536CrossRef

41.

Li M, Xu H, Huang X, Song Z, Liu X, Li X (2021) Facial expression recognition with identity and emotion joint learning. IEEE Trans Affect Comput 12(2):544–550. https://doi.org/10.1109/TAFFC.2018.2880201CrossRef

42.

Li S, Deng W (2022) A deeper look at facial expression dataset bias. IEEE Trans Affect Comput 13(2):881–893. https://doi.org/10.1109/TAFFC.2020.2973158CrossRef

43.

Li Y, Huang X, Zhao G (2021) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263. https://doi.org/10.1109/TIP.2020.3035042CrossRef

44.

Li Y, Zeng J, Shan S, Chen X (2019) Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans Image Process 28(5):2439–2450. https://doi.org/10.1109/TIP.2018.2886767MathSciNetCrossRef

45.

Meng Z, Han S, Liu P, Tong Y (2019) Improving speech related facial action unit recognition by audiovisual information fusion. IEEE Trans Cybern 49(9):3293–3306. https://doi.org/10.1109/TCYB.2018.2840090CrossRef

46.

Ni R, Yang B, Zhou X, Cangelosi A, Liu X (2022) Facial expression recognition through cross-modality attention fusion. IEEE Trans Cogn Dev Syst. https://doi.org/10.1109/TCDS.2022.3150019CrossRef

47.

Poux D, Allaert B, Ihaddadene N, Bilasco IM, Djeraba C, Bennamoun M (2022) Dynamic facial expression recognition under partial occlusion with optical flow reconstruction. IEEE Trans Image Process 31:446–457. https://doi.org/10.1109/TIP.2021.3129120CrossRef

48.

Qi C et al (2018) Facial expressions recognition based on cognition and mapped binary patterns. IEEE Access 6:18795–18803. https://doi.org/10.1109/ACCESS.2018.2816044CrossRef

49.

Qu X et al (2022) Attend to where and when: cascaded attention network for facial expression recognition. IEEE Trans Emerg Top Comput Intell 6(3):580–592. https://doi.org/10.1109/TETCI.2021.3070713CrossRef

50.

Sepas-Moghaddam A, Etemad A, Pereira F, Correia PL (2021) CapsField: light field-based face and expression recognition in the wild using capsule routing. IEEE Trans Image Process 30:2627–2642. https://doi.org/10.1109/TIP.2021.3054476CrossRef

51.

Wang K, Peng X, Yang J, Lu S, Qiao Y, Suppressing uncertainties for large-scale facial expression recognition (Online). Available: https://github.com/kaiwang960112/Self-Cure-Network

52.

Wang Y, Li Y, Song Y, Rong X (2020) The influence of the activation function in a convolution neural network model of facial expression recognition. Appl Sci (Switzerland) 10(5). https://doi.org/10.3390/app10051897

53.

Xia Y, Zheng W, Wang Y, Yu H, Dong J, Wang FY (2022) Local and global perception generative adversarial network for facial expression synthesis. IEEE Trans Circuits Syst Video Technol 32(3):1443–1452. https://doi.org/10.1109/TCSVT.2021.3074032CrossRef

54.

Xie S, Hu H, Chen Y (2021) Facial expression recognition with two-branch disentangled generative adversarial network. IEEE Trans Circ Syst Video Technol 31(6):2359–2371. https://doi.org/10.1109/TCSVT.2020.3024201CrossRef

55.

Yan Y, Huang Y, Chen S, Shen C, Wang H (2020) Joint deep learning of facial expression synthesis and recognition. IEEE Trans Multimedia 22(11):2792–2807. https://doi.org/10.1109/TMM.2019.2962317CrossRef

56.

Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640. https://doi.org/10.1109/ACCESS.2017.2784096CrossRef

57.

Zhang H, Su W, Yu J, Wang Z (2021) Identity-expression dual branch network for facial expression recognition. IEEE Trans Cogn Dev Syst 13(4):898–911. https://doi.org/10.1109/TCDS.2020.3034807CrossRef

58.

Zhang X, Zhang F, Xu C (2022) Joint expression synthesis and representation learning for facial expression recognition. IEEE Trans Circuits Syst Video Technol 32(3):1681–1695. https://doi.org/10.1109/TCSVT.2021.3056098CrossRef

59.

Zheng K, Yang D, Liu J, Cui J (2020) Recognition of teachers’ facial expression intensity based on convolutional neural network and attention mechanism. IEEE Access. https://doi.org/10.1109/ACCESS.2020.3046225CrossRef

60.

Zhang Y, Hossain MZ, Rahman S (2021) DeepVANet: a deep end-to-end network for multi-modal emotion recognition, pp 227–237. https://doi.org/10.1007/978-3-030-85613-7_16

61.

Kossaifi J, Tzimiropoulos G, Todorovic S, Pantic M (2017) AFEW-VA database for valence and arousal estimation in-the-wild. Image Vis Comput 65:23–36. https://doi.org/10.1016/j.imavis.2017.02.001

Title: Multimodal Attention CNN for Human Emotion Recognition
Authors: Gyanendra Tiwary
Shivani Chauhan
Krishan Kumar Goyal
Publisher: Springer Nature Singapore
Book: Cryptology and Network Security with Machine Learning
Print ISBN: 978-981-9706-40-2

Electronic ISBN: 978-981-9706-41-9

Copyright Year: 2024
DOI: https://doi.org/10.1007/978-981-97-0641-9_11

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"