Electrocardiogram-Based Emotion Recognition: History
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor: , ,

Heart rate variability (HRV) serves as a significant physiological measure that mirrors the regulatory capacity of the cardiac autonomic nervous system. It not only indicates the extent of the autonomic nervous system’s influence on heart function but also unveils the connection between emotions and psychological disorders. 

  • heart rate variability
  • emotion recognition

1. Introduction

Emotions are innate physiological responses designed to maintain or restore homeostasis by altering the environment for more appropriate interactions [1]. Emotions are not only significant in the realm of social understanding and interaction, but they also trigger physiological mechanisms in order to proficiently recognize and address a diverse range of challenges and opportunities, thus contributing to our survival and growth [2]. Emotional experiences are closely linked to bodily perceptions, and they help direct attention to critical events, such as physical needs, potential threats, and social interactions. Emotional experiences play a role in coordinating behavioral and physiological responses during these critical occasions [3].
The autonomic nervous system (ANS) plays a significant role in conveying extensive data on emotional states, is tightly connected to emotional arousal, and at the same time, dominates the function of the lungs, heart, and numerous other organ systems [4]. Various mood-related physiological variables are often easily measured and thus detect altered states of the ANS [5]. Such changes usually occur unknowingly. For some stimuli and unconscious responses, the state of the response changes more rapidly than the conscious response. These physical aspects include facial expression, movement and gait, voice characteristics, electrocardiogram, electroencephalogram, electrooculogram, and galvanic skin response [6,7,8,9]. Of particular interest is the fact that the core area of our focus, the electrocardiogram (ECG), has been shown to have a strong correlation with emotional characteristics as well as with ECG waveforms. Numerous scholars have explored the feasibility and limitations of utilizing ECG signals for emotion detection [10]. Furthermore, recent developments in machine learning and deep learning have shown that emotion recognition systems can reliably derive information from ECG data [11]. Most sentiment recognition tasks involving machine learning or deep learning predominantly employ a fully supervised learning paradigm [12]. There are several limitations to this approach. First, in a typical fully supervised learning scenario, each classification or regression task requires re-training the model from scratch, which requires significant computational resources and time. In addition, features obtained from a fully supervised trained model are usually too task-specific to generalize well to other tasks. Finally, typical fully supervised learning usually requires training with large-scale manually labeled datasets, as small datasets usually lead to the performance degradation of deep networks.
The autonomic nervous system (ANS) consists of the sympathetic and parasympathetic branches, working in harmony to oversee the functioning of target organs and tissues, ensuring the preservation of homeostasis [13]. Emotion is closely connected to these branches as they all provide nerve signals to the heart, with a particular emphasis on the AV node. This node collaborates with neuromodulation to facilitate the coordination of neurotransmitters, ensuring they work in harmony to regulate their functions and generate a distinct response to emotional conditions. Sympathetic overactivity may result in additional cardiac contractions or a rapid heartbeat. In contrast, parasympathetic responses to negative emotions are often influenced by olfactory or visual stimuli, which may result in a slowed heartbeat or even cardiac arrest [14]. The spectral examination of heart rate variability (HRV) is considered a non-intrusive technique to assess the balance between the two principal branches of the autonomic nervous system (ANS), which has been suggested for detecting and characterizing emotional states [15]. A set of guidelines has been formulated for evaluating, physiologically interpreting, and clinically applying resting heart rate variability (HRV), which includes the delineation of three distinct spectral components: an ultra-low-frequency (ULF) spectrum ranging from 0 to 0.04 Hz, a low-frequency (LF) spectrum covering the range from 0.04 to 0.15 Hz, and a high-frequency (HF) spectrum spanning from 0.15 to 0.4 Hz. HF band power is regarded as an indicator of parasympathetic activity, primarily attributed to respiratory sinus arrhythmia. The power within the low-frequency range is regarded as an indicator of both the sympathetic and parasympathetic activity of the heart. HRV serves as a mirror to the fluctuations in the autonomic nervous system, which, in turn, can mirror the accompanying emotional states.

2. Electrocardiogram-Based Emotion Recognition 

Presently, the primary focus in emotion recognition through ECG revolves around the utilization of classification algorithms alongside the selection of pertinent features and the scope of identifiable emotions. Machine learning algorithms have proven to be effective for emotion recognition based on ECG data. Axel and his team employed the wavelet scattering method for the extraction of unique characteristics from ECG data. This method facilitated the acquisition of signal features spanning varying time scales to assess their performance. The outcomes of this study demonstrated that the proposed feature-extraction and signal classification algorithm, within the realm of emotional dimensions, achieved an accuracy rate of 88.8% for arousal, 90.2% for valence, and an impressive 95.3% for two-dimensional classification [16]. Theekshana et al. presented a machine learning approach utilizing an ensemble learning method to identify core emotions, including anger, sadness, happiness, and the combination of happiness with electrocardiogram (ECG) data. They utilized spectral analysis as a novel approach to extract features and assessed the efficacy of a widely recognized collection of ensemble learners for emotion classification through a machine learning process. The ensemble learners exhibited a 10.77% enhancement in accuracy when compared to the top-performing individual biosensor-based model [17]. Yan et al. introduced the X-GWO-SVM approach to analyze sentiments from ECG data. They conducted single-subject cross-validation and achieved an impressive average accuracy rate of 95.93% when utilizing the WESAD dataset. This result demonstrates its superior reliability compared to previous implementations of supervised machine learning techniques [18].
Deep learning algorithms are commonly employed in the realm of emotion recognition as well. Pritam et al. devised a novel approach using a self-supervised deep multitask learning framework to detect emotions based on electrocardiogram (ECG) data. In this process, the convolutional layer remained fixed, while the dense layer was fine-tuned using labeled ECG data. Notably, this innovative solution yielded state-of-the-art results in classifying arousal, mood, emotional state, and stress across four distinct datasets [19]. Xu et al. introduced an emotion detection technique rooted in deep learning tailored for healthcare data analysis. Their approach incorporated a multichannel convolutional neural network to extract distinctive features from ECG data and textual content related to emotions, particularly for detecting emotional fatigue. Ultimately, this method amalgamated the features derived from multiple data sources to ascertain emotional states. The experimental findings demonstrated that the proposed model consistently achieved an accuracy rate exceeding 85% in the prediction of emotional fatigue [20]. Chen et al. introduced a novel approach to emotion recognition, which involved combining multiple sensory modalities using the Dempster–Shafer evidence theory. They utilized an SVM classifier to categorize EEG signal features. The results of the experiment demonstrated that the multimodal fusion model outperformed the unimodal emotion detection method, resulting in a significant increase in accuracy by 7.37% and 8.73% as compared to the emotion recognition model based on ECG signals [21]. Hammad et al. applied the PETSFCNN model in their research on emotion recognition, achieving an impressive maximum classification accuracy of 97.56%. They employed a deep neural network in combination with grid search optimization to enhance accuracy in classification tasks. The results of their study indicated that the suggested method surpasses existing techniques in precisely identifying emotions from ECG signals. This implies the possibility of utilizing it as an intelligent system for emotion recognition [22].
When it comes to selecting features for emotion recognition, HRV-related characteristics have gained widespread usage. Guo et al. conducted an analysis of ECG signals to extract heart rate variability (HRV) features, employing techniques encompassing frequency domain, time domain, statistical methods, and Poincaré. The HRV characteristics were later utilized for categorizing different emotional states, utilizing principal component analysis, and then employing a support vector machine to reduce the feature set. Notably, they achieved classification accuracies of 71.4% for distinguishing two emotional states (positive/negative) and 56.9% for classifying five emotional states [23]. Ferdinando et al. employed K-nearest neighbors (KNN) as a classifier in their study. They combined supervised dimensionality reduction using neighborhood component analysis (NCA) with feature-extraction techniques, which included capturing standard HRV features and statistical distributions of instantaneous frequencies. These methods were applied to address the classification of three distinct emotion and arousal categories. The findings suggested that, in most instances, integrating NCA resulted in a significant performance boost of 74% when compared to the absence of NCA in the implementation [24]. Singson et al. harnessed a ResNet-based CNN to analyze both facial expressions and physiological data, with a particular focus on heart rate variability (HRV) features. Their aim was to discern and validate emotions, achieving an accuracy rate of 68.42% through the analysis of ECG signals [25].

This entry is adapted from the peer-reviewed paper 10.3390/s23208636

This entry is offline, you can click here to edit this entry!
Video Production Service