An Autoencoder based Emotional Stress State Detection Approach by using Electroencephalography Signals
محورهای موضوعی : Signal Processing
1 - AI and Big Data Department, Endicott College, Woosong University, Daejeon, South Korea
کلید واژه: EEG Signals, emotion analysis, stress analysis, autoencoder, machine learning, deep learning.,
چکیده مقاله :
Identifying hazards from human error is critical for industrial safety since dangerous and reckless industrial worker actions, as well as a lack of measures, are directly accountable for human-caused problems. Lack of sleep, poor nutrition, physical deformities, and weariness are some of the key factors that contribute to these risky and reckless behaviors that might put a person in a perilous scenario. This scenario causes discomfort, worry, despair, cardiovascular disease, a rapid heart rate, and a slew of other undesirable outcomes. As a result, it would be advantageous to recognize people's mental states in the future in order to provide better care for them. Researchers have been studying electroencephalogram (EEG) signals to determine a person's stress level at work in recent years. A full feature analysis from domains is necessary to develop a successful machine learning model using electroencephalogram (EEG) inputs. By analyzing EEG data, a time-frequency based hybrid bag of features is designed in this research to determine human stress dependent on their sex. This collection of characteristics includes features from two types of assessments: time-domain statistical analysis and frequency-domain wavelet-based feature assessment. The suggested two layered autoencoder based neural networks (AENN) are then used to identify the stress level using a hybrid bag of features. The experiment uses the DEAP dataset, which is freely available. The proposed method has a male accuracy of 77.09% and a female accuracy of 80.93%.
Identifying hazards from human error is critical for industrial safety since dangerous and reckless industrial worker actions, as well as a lack of measures, are directly accountable for human-caused problems. Lack of sleep, poor nutrition, physical deformities, and weariness are some of the key factors that contribute to these risky and reckless behaviors that might put a person in a perilous scenario. This scenario causes discomfort, worry, despair, cardiovascular disease, a rapid heart rate, and a slew of other undesirable outcomes. As a result, it would be advantageous to recognize people's mental states in the future in order to provide better care for them. Researchers have been studying electroencephalogram (EEG) signals to determine a person's stress level at work in recent years. A full feature analysis from domains is necessary to develop a successful machine learning model using electroencephalogram (EEG) inputs. By analyzing EEG data, a time-frequency based hybrid bag of features is designed in this research to determine human stress dependent on their sex. This collection of characteristics includes features from two types of assessments: time-domain statistical analysis and frequency-domain wavelet-based feature assessment. The suggested two layered autoencoder based neural networks (AENN) are then used to identify the stress level using a hybrid bag of features. The experiment uses the DEAP dataset, which is freely available. The proposed method has a male accuracy of 77.09% and a female accuracy of 80.93%.
[1] R. E. Bender, and L. B. Alloy, “Life stress and kindling in bipolar disorder: review of the evidence and integration with emerging biopsychosocial theories”, Clinical psychology review, Vol. 31, No. 3, 2011, pp. 383–398.
[2] T. G. Pickering, “Mental stress as a causal factor in the development of hypertension and cardiovascular disease”, Current hypertension reports, Vol. 3, No. 3, 2001, pp. 249–254.
[3] T. C. Major, and J. M. Conrad, “A survey of brain computer interfaces and their applications”, in IEEE SOUTHEASTCON 2014, 2014, pp. 1–8.
[4] N. Elsayed, Z. S. Zaghloul, and M. Bayoumi, “Brain computer interface: EEG signal preprocessing issues and solutions”, Int. J. Comput. Appl, Vol. 169, No. 3, 2017, pp. 12–16.
[5] R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, “Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style”, Psychophysiology, Vol. 30, No. 1, 1993, pp. 82–89.
[6] A. C. Atencio et al., “Computing stress-related emotional state via frontal cortex asymmetry to be applied in passive-ssBCI”, in 5th ISSNIP-IEEE Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), 2014, pp. 1–6.
[7] F.P. George et al. “Recognition of emotional states using EEG signals based on time-frequency analysis and SVM classifier”, International Journal of Electrical and Computer Engineering, Vol. 9, No. 2, 2019, pp. 1012-1020.
[8] T. F. Bastos-filho, A. Ferreira, A.C. Atencio, S. Arjunan, and D. Kumar, “Evaluation of Feature Extraction Techniques in Emotional State Recognition”, in 4th International conference on intelligent human computer interaction (IHCI), 2012, pp. 1-6.
[9] N. Jatupaiboon, S. Pan-ngum, and P. Israsena, “Real-time EEG-based happiness detection system”, The Scientific World Journal, 2013, pp. 1-12.
[10] M. J. Hasan, and J. M. Kim, “A hybrid feature pool-based emotional stress state detection algorithm using EEG signals”, Brain sciences, 2019, Vol. 9, No. 12, pp. 1-15.
[11] D. Shon, K. Im, J. Park, D. Lim, B. Jang, and J. Kim, “Emotional Stress State Detection Using Genetic Algorithm-Based Feature Selection on EEG Signals”, Int. J. of environmental research and public health, 2018, Vol. 15, No. 11, pp. 1-11.
[12] S. Koelstra, C. Muhl, M. Soleymani, J.S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals”, IEEE transactions on affective computing, 2011, Vol. 3, No. 1, pp.18-31.
[13] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, “EEG-based emotion recognition in music listening”, IEEE Transactions on Biomedical Engineering, 2010, Vol 57, No. 7, pp. 1798-1806.
[14] R. Jenke, A. Peer, and M. Buss, “Feature extraction and selection for emotion recognition from EEG”, IEEE Transactions on Affective Computing, 2014, Vol. 5, No. 3, pp. 327-39.
[15] M. Sohaib, C.-H. Kim, and J.-M. Kim, “A Hybrid Feature Model and Deep-Learning-Based Bearing Fault Diagnosis”, Sensors, 2017, Vol. 17, No. 12, pp. 1-16.
[16] J. A. Russell, “A circumplex model of affect”, Journal of Personality and Social Psychology, 1980, Vol. 39, No. 6, pp. 1161–1178.
[17] S.H. Oh, Y.R. Lee, and H.N. Kim, “A novel EEG feature extraction method using Hjorth parameter”, Int. Journal of Electronics and Electrical Engineering, 2014, Vol. 2, No. 2, pp. 106-110.
[18] P.C. Petrantonakis, and L.J. Hadjileontiadis, “Emotion Recognition from EEG Using Higher Order Crossings”, IEEE Transactions on Information Technology in Biomedicine, 2010, Vol. 14, No. 2, pp. 186–197.
[19] H. Shao, H. Jiang, H. Zhao, and F. Wang, “A novel deep autoencoder feature learning method for rotating machinery fault diagnosis”, Mechanical Systems and Signal Processing, 2017, Vol. 95, pp. 187-204.
http://jist.acecr.org ISSN 2322-1437 / EISSN:2345-2773 |
Journal of Information Systems and Telecommunication
|
An Autoencoder based Emotional Stress State Detection Approach Using Electroencephalography Signals |
Jia Uddin1*
|
1. AI and Big Data Department, Endicott College, Woosong University, Daejeon, South Korea |
Received: 17 Nov 2021/ Revised: 04 Mar 2022/ Accepted: 27 Apr 2022 |
|
Abstract
Identifying hazards from human error is critical for industrial safety since dangerous and reckless industrial worker actions, as well as a lack of measures, are directly accountable for human-caused problems. Lack of sleep, poor nutrition, physical deformities, and weariness are some of the key factors that contribute to these risky and reckless behaviors that might put a person in a perilous scenario. This scenario causes discomfort, worry, despair, cardiovascular disease, a rapid heart rate, and a slew of other undesirable outcomes. As a result, it would be advantageous to recognize people's mental states in the future in order to provide better care for them. Researchers have been studying electroencephalogram (EEG) signals to determine a person's stress level at work in recent years. A full feature analysis from domains is necessary to develop a successful machine learning model using electroencephalogram (EEG) inputs. By analyzing EEG data, a time-frequency based hybrid bag of features is designed in this research to determine human stress dependent on their sex. This collection of characteristics includes features from two types of assessments: time-domain statistical analysis and frequency-domain wavelet-based feature assessment. The suggested two layered autoencoder based neural networks (AENN) are then used to identify the stress level using a hybrid bag of features. The experiment uses the DEAP dataset, which is freely available. The proposed method has a male accuracy of 77.09% and a female accuracy of 80.93%.
Keywords: EEG Signals; Emotion Analysis; Stress Analysis; Autoencoder; Machine Learning.
1- Introduction
For engineering wellbeing, detecting consequences from human error is essential because unsafe and irresponsible manners of employees involved in manufacturing are clearly accountable for human-caused troubles. Among several key factors of these dangerous and irresponsible activities, lack of proper sleep leads a person to an extreme stressful situation. Stress initiates irritation, fear, sadness, vascular illness and numerous additional injurious effects [1], [2]. Numerous forms of brain signals, i.e., functional magnetic resonance imaging (fMRI), near-infrared spectroscopy (NIRS), Electrocorticography (ECoG), and electroencephalogram (EEG), are utilized for evaluating emotional conditions of individual [3]. Among all of these forms of data, EEG can be assessed non-intrusively [4]. The principal objective of this research is to categorize the emotional situation of an individual based on the sex by evaluating pre-processed freely accessible EEG signals.
Several surveys have exhibited relationships between EEG signals and several emotional situations [5–10]. In [5], an EEG-based assessment on the frontal channel with support vector machine (SVM) is designed. In [9], an in-depth analysis of power spectral density (PSD) is proposed to classify the emotional state by SVM. Among these researches, the common attribute is to consider all the features for classifier.
In this research, an EEG signal-driven emotional state classification method is established to evaluate whether a person is experiencing stress. By evaluating the signal, a hybrid feature bag is designed to create a dynamic and robust feature list. This process is divided into two parts: (1) statistical analysis from the time domain, and (2) wavelet-based feature assessment from the frequency domain. In the EEG signals, for the presence of the artifacts [11], it is hard to find the absolute feature information. This study examined pre-processed signals from the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset [12]. The time domain features are defined in detail in Section 3. As the EEG signal has five indistinguishable bandwidths [13], [14], wavelet decomposition is studied to determine the frequency domain features, which are depicted in Section III. From these designed bag of hybrid features, instead of providing all of them to the classifier, some built-in feature reduction mechanism embedded classifier is used to utilize only the most significant features for final classification. Deep networks can obtain extremely characteristic features via their multi-layered model architectures. Moreover, they keep only the most representative information in each layer to reduce the dimensionality and also to improve the classification performance by selecting only the most intrinsic feature information [15]. In this research, a three layered autoencoder based neural network (AENN) is proposed for different emotional state classification. To prove the strength of the suggested technique, few comparisons are made with the approaches discussed into [5], and [9]. This paper's primary contributions can be summarized as follows:
(1) Statistical analysis in the time domain and wavelet-based feature assessment in the frequency domain combine to create a hybrid bag of features.
(2) A two layered AENN is proposed to learn and utilize only the important features through the embedded feed-forward feature selection architecture to improve the final classification accuracy.
The remainder of this paper will be organized as follows. The DEAP dataset is readily available, and the recommended approach is explained in Section 2. Section 3 contains the data agreement, evaluation of the experimental findings, and discussion, and Section 4 concludes the paper.
2- Proposed Method
In Fig. 1, a block diagram proposed approach is provided. The proposed approach is divided into four sections: (1) pre-processed data gathering [12], (2) data arrangement, (3) creation of hybrid bag of features, and (4) AENN-based classification.
The data is first down sampled to 128 Hz, and then the artifacts are eliminated from the data, as seen in this diagram. The current study's annotation was completed after filtering the data using bandpass frequency and common segmentation. The statistical features from the time domain and wavelet-based frequency domain are then examined and retrieved from each class sample. Finally, an Autoencoder-based Neural Network is presented for classification.
Fig. 1 Complete pipeline of the proposed approach.
2-1- Dataset Details
For this study, pre-processed EEG signals from the DEAP dataset [12]. This dataset incorporates emotional responses stimulated by music videos. The information related to the considered dataset is listed into Table 1. The placement of attached sensors for collecting the EEG data, with the real experimental setup is depicted into Fig. 2.
Table 1: Details of the considered dataset
Area | Information |
Number of participants | 32 |
Age range | 19 – 37 years |
Number of male participants | 17 |
Number of female participants | 15 |
Recorded time for EEG | 1 minute 3 seconds |
Pre-train baseline | 3 seconds |
Final considered signal length | 1 minute |
Electrode placement system | 10 – 20 system of electrode |
Each contributor viewed 40 music videos. The dataset retains pre-processed data that down sample the recorded signals to 128 Hz. In this research, the pre-processed EEG signals are considered. For this experimentation, the dataset has to be annotated. From the analyzed valence and arousal level from the recorded EEG response calm and stress states have been specified by Equation (1) and (2) [8], [16].
| (1) | ||||
| (2) |
| (3) |
Here, is the input with the dimension of , and then the encoded layer maps the input data to encoded latent variable , where dimension reduced to . is weight and is the bias here.
In this work, the same mechanism is deployed [15], [19]. The encoded layer latent feature representation is then passed to the SoftMax classifier for final classification. In the proposed AENN, two layers are used to learn the latent feature space in unsupervised way. The main architecture of the proposed AENN is illustrated into Fig. 3.
Fig. 2 Real experimental testbed with subject for collecting EEG data.
Table 2: Statistical explanation of the measured time-domain features
Feature | Equation | Feature | Equation | Feature | Equation |
F1 |
| F2 |
| F3 |
|
F4 |
| F5 |
| F6 |
|
Here, x is the time-domain raw signal. N is the total number of samples. |
Dataset (Sex-based) | Sub-set | Participant ID | An Experimental ID System That Reflects the Unique State | |
Calm | Stress | |||
Male | 1 | 1 | 9,14 | 17,32,34,35,36,37 |
2 | 5 | 13,29 | 23,30,37 | |
3 | 12 | 16,17,28 | 25,29,32,33,35,36,37,38 | |
4 | 16 | 6,12,16,21,36 | 1,15,17,24,26,27,34 | |
5 | 18 | 22,26,34 | 30 | |
6 | 19 | 15,26,27 | 29,38 | |
7 | 20 | 16,26,27,28,40 | 23,25,29 | |
8 | 21 | 3,21,26,34,35 | 20,22,24 | |
9 | 26 | 30 | 34 | |
10 | 27 | 5,15,19,26,27,28,33,40 | 27 | |
11 | 28 | 15,22,24,25 | 35,38 | |
12 | 29 | 15,17 | 30,31,33,35 | |
Female | 1 | 2 | 5,7,10,22,24,36 | 29,30,32,37,38 |
2 | 4 | 2,6,18 | 24,28,32 | |
3 | 8 | 10,37,39 | 31,36 | |
4 | 10 | 15,17,20,22,26,27,28 | 21,30,35,36,37,38,39 | |
5 | 11 | 2,12,16,19,25,26,28,40 | 27,35,37,38,39 | |
6 | 13 | 12,15,16 | 7,21,23,31,34,35,36,37,38,39 | |
7 | 14 | 22,27 | 10,21,23,24,29,30,32,34,35,36,38 | |
8 | 15 | 7,16,22,26 | 24,25,30,38 | |
9 | 22 | 1,6,12,15,16,28 | 23,24,29,30,32,33,35,36,37,38,39 | |
10 | 24 | 33,40 | 21,23,24,30,31,38,39 | |
11 | 25 | 4,5,26,27,28,34 | 2,10,23,29,31,32,33,37,38,39 | |
12 | 31 | 17,22,24,27,28,29 | 23,32,34,37,38,39 | |
13 | 32 | 2,6,15,26,33 | 24,30,37 |
Fig. 3 AENN model architecture.
Fig. 4 10-fold cross validation [10].
3- Experimental Result Analysis
3-1- Arrangement of Dataset
Among 32 contributors [11], contributor numbers 3, 6, 7, 9, 17, 23, and 30 did not signify the attributes of considered emotional states. So, the left over 25 contributors are counted for the final dataset. 32 EEG channels are considered for considering the data with a 128Hz sampling frequency. The particulars of the considered dataset are given in Table 3. The Table reveals that each sub-dataset has a distinctive experimental ID, and for each ID, different music videos are reliable for calm and stress state. Therefore, for sex-based state analysis according to the identity of sex; two main datasets are considered for final classification, i.e., dataset male (consisting of 12 sub-sets, merged together to form male dataset as a whole), and dataset female (consisting of 13 sub-sets, merged together to form male dataset as a whole).
3-2- Performance Analysis of AENN
Each dataset is split into two parts: training and testing, with a 70/30 split between the two. Equation (4) establishes the class-dependent accuracy.
| (4) |
| (5) |
Dataset | Class-wise Accuracy (%) | Average Accuracy (%) | |
Calm | Stress | ||
Male | 78.25 | 75.93 | 77.09 |
Female | 79.42 | 82.44 | 80.93 |
Average (%) |
|
| 79.01 |
The feature embeddings extracted from the AENN encoded layer (2 features values are extracted from encoded layer) is displayed into Fig. 5 for both of the datasets.
In addition to these, to establish the robustness of this approach, few comparisons are made with [5] , and [9].The details of this comparative analysis are depicted into Table 5.
Table 5: Relative evaluation of various methodologies
Methods | Average Accuracy (%) | Decrement from the Proposed Method (%) |
[5] | 68.23 | 10.78 |
[9] | 72.44 | 6.57 |
Proposed | 79.01 | - |
Fig. 5 AENN feature embedding from encoded layer for (a) male dataset, and (b) female dataset.
3-3- Discussion
From Table 4, it is visible that the female is performing better than the male dataset. From Table 3, it is visible that the female dataset contains 13 sub-sets where the male dataset contains 12 sub-sets. To get a better performance from deep networks, amount of dataset is a vital issue. Moreover, for female dataset case, stress state is giving higher accuracy than the calm state. From Table 3, it is clearly visible that the stress state data is larger than the calm state data for female dataset (61 cases for calm state and 84 cases for stress state).
From Fig. 5, for both of the dataset, the feature embeddings are overlapped. That indicates the separation of identical features are not good. The main underlying reasons behind this is little analytical. If the sub-sets are being analyzed from Table 3, for each sub-set, the experiment IDs of calm and stress state are different. In addition, sometimes it is imbalanced as well. So, in this dataset not any particular experiment ID is responsible for clam or stress state. It makes difficult to find out the most intrinsic class-wise information in a very accurate manner.
For comparative analysis, the proposed model outperformed the approach described in [5] by 10.78%, and the approach described in [9] by 6.57%.
4- Conclusions
This paper proposed a sex-based stress state classification method by analyzing EEG signals from brain. First it created a bag of features from the statistical analysis of time-domain and wavelet-based analysis of frequency domain. Therefore, the hybrid bag of features was forwarded to the proposed AENN to identify the stress state by feed-forward architecture based selective features. The proposed approach achieved an accuracy of 77.09% for male dataset and 80.93% for female dataset. In addition to that, it outperformed several existing methods related to this study by at least 6.57%.
Acknowledgments
This research is funded by Woosong University Academic Research in 2022.
References
[1] R. E. Bender, and L. B. Alloy, “Life stress and kindling in bipolar disorder: review of the evidence and integration with emerging biopsychosocial theories”, Clinical psychology review, Vol. 31, No. 3, 2011, pp. 383–398.
[2] T. G. Pickering, “Mental stress as a causal factor in the development of hypertension and cardiovascular disease”, Current hypertension reports, Vol. 3, No. 3, 2001, pp. 249–254.
[3] T. C. Major, and J. M. Conrad, “A survey of brain computer interfaces and their applications”, in IEEE SOUTHEASTCON 2014, 2014, pp. 1–8.
[4] N. Elsayed, Z. S. Zaghloul, and M. Bayoumi, “Brain computer interface: EEG signal preprocessing issues and solutions”, Int. J. Comput. Appl, Vol. 169, No. 3, 2017, pp. 12–16.
[5] R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, “Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style”, Psychophysiology, Vol. 30, No. 1, 1993, pp. 82–89.
[6] A. C. Atencio et al., “Computing stress-related emotional state via frontal cortex asymmetry to be applied in passive-ssBCI”, in 5th ISSNIP-IEEE Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), 2014, pp. 1–6.
[7] F.P. George et al. “Recognition of emotional states using EEG signals based on time-frequency analysis and SVM classifier”, International Journal of Electrical and Computer Engineering, Vol. 9, No. 2, 2019, pp. 1012-1020.
[8] T. F. Bastos-filho, A. Ferreira, A.C. Atencio, S. Arjunan, and D. Kumar, “Evaluation of Feature Extraction Techniques in Emotional State Recognition”, in 4th International conference on intelligent human computer interaction (IHCI), 2012, pp. 1-6.
[9] N. Jatupaiboon, S. Pan-ngum, and P. Israsena, “Real-time EEG-based happiness detection system”, The Scientific World Journal, 2013, pp. 1-12.
[10] M. J. Hasan, and J. M. Kim, “A hybrid feature pool-based emotional stress state detection algorithm using EEG signals”, Brain sciences, 2019, Vol. 9, No. 12, pp. 1-15.
[11] D. Shon, K. Im, J. Park, D. Lim, B. Jang, and J. Kim, “Emotional Stress State Detection Using Genetic Algorithm-Based Feature Selection on EEG Signals”, Int. J. of environmental research and public health, 2018, Vol. 15, No. 11, pp. 1-11.
[12] S. Koelstra, C. Muhl, M. Soleymani, J.S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals”, IEEE transactions on affective computing, 2011, Vol. 3, No. 1, pp.18-31.
[13] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, “EEG-based emotion recognition in music listening”, IEEE Transactions on Biomedical Engineering, 2010, Vol 57, No. 7, pp. 1798-1806.
[14] R. Jenke, A. Peer, and M. Buss, “Feature extraction and selection for emotion recognition from EEG”, IEEE Transactions on Affective Computing, 2014, Vol. 5, No. 3, pp. 327-39.
[15] M. Sohaib, C.-H. Kim, and J.-M. Kim, “A Hybrid Feature Model and Deep-Learning-Based Bearing Fault Diagnosis”, Sensors, 2017, Vol. 17, No. 12, pp. 1-16.
[16] J. A. Russell, “A circumplex model of affect”, Journal of Personality and Social Psychology, 1980, Vol. 39, No. 6, pp. 1161–1178.
[17] S.H. Oh, Y.R. Lee, and H.N. Kim, “A novel EEG feature extraction method using Hjorth parameter”, Int. Journal of Electronics and Electrical Engineering, 2014, Vol. 2, No. 2, pp. 106-110.
[18] P.C. Petrantonakis, and L.J. Hadjileontiadis, “Emotion Recognition from EEG Using Higher Order Crossings”, IEEE Transactions on Information Technology in Biomedicine, 2010, Vol. 14, No. 2, pp. 186–197.
[19] H. Shao, H. Jiang, H. Zhao, and F. Wang, “A novel deep autoencoder feature learning method for rotating machinery fault diagnosis”, Mechanical Systems and Signal Processing, 2017, Vol. 95, pp. 187-204.
* Jia Uddin
jia.uddin@wsu.ac.uk