An Autoencoder based Emotional Stress State Detection Approach by using Electroencephalography Signals

Uddin, Jia

doi:10.52547/jist.32267.11.41.24

Manuscript ID : 2021111732267 Visit : 4783 Page: 24 - 30

10.52547/jist.32267.11.41.24

20.1001.1.23221437.2023.11.41.6.1

Article Type: Original Research

An Autoencoder based Emotional Stress State Detection Approach by using Electroencephalography Signals

Subject Areas : Signal Processing

Jia Uddin ¹

1 - AI and Big Data Department, Endicott College, Woosong University, Daejeon, South Korea

Received: 2021-11-17 Accepted : 2022-04-27 Published : 2023-01-10

Keywords: EEG Signals, emotion analysis, stress analysis, autoencoder, machine learning, deep learning.,

Abstract :

Identifying hazards from human error is critical for industrial safety since dangerous and reckless industrial worker actions, as well as a lack of measures, are directly accountable for human-caused problems. Lack of sleep, poor nutrition, physical deformities, and weariness are some of the key factors that contribute to these risky and reckless behaviors that might put a person in a perilous scenario. This scenario causes discomfort, worry, despair, cardiovascular disease, a rapid heart rate, and a slew of other undesirable outcomes. As a result, it would be advantageous to recognize people's mental states in the future in order to provide better care for them. Researchers have been studying electroencephalogram (EEG) signals to determine a person's stress level at work in recent years. A full feature analysis from domains is necessary to develop a successful machine learning model using electroencephalogram (EEG) inputs. By analyzing EEG data, a time-frequency based hybrid bag of features is designed in this research to determine human stress dependent on their sex. This collection of characteristics includes features from two types of assessments: time-domain statistical analysis and frequency-domain wavelet-based feature assessment. The suggested two layered autoencoder based neural networks (AENN) are then used to identify the stress level using a hybrid bag of features. The experiment uses the DEAP dataset, which is freely available. The proposed method has a male accuracy of 77.09% and a female accuracy of 80.93%.

References:

[1] R. E. Bender, and L. B. Alloy, “Life stress and kindling in bipolar disorder: review of the evidence and integration with emerging biopsychosocial theories”, Clinical psychology review, Vol. 31, No. 3, 2011, pp. 383–398.
[2] T. G. Pickering, “Mental stress as a causal factor in the development of hypertension and cardiovascular disease”, Current hypertension reports, Vol. 3, No. 3, 2001, pp. 249–254.
[3] T. C. Major, and J. M. Conrad, “A survey of brain computer interfaces and their applications”, in IEEE SOUTHEASTCON 2014, 2014, pp. 1–8.
[4] N. Elsayed, Z. S. Zaghloul, and M. Bayoumi, “Brain computer interface: EEG signal preprocessing issues and solutions”, Int. J. Comput. Appl, Vol. 169, No. 3, 2017, pp. 12–16.
[5] R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, “Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style”, Psychophysiology, Vol. 30, No. 1, 1993, pp. 82–89.
[6] A. C. Atencio et al., “Computing stress-related emotional state via frontal cortex asymmetry to be applied in passive-ssBCI”, in 5th ISSNIP-IEEE Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), 2014, pp. 1–6.
[7] F.P. George et al. “Recognition of emotional states using EEG signals based on time-frequency analysis and SVM classifier”, International Journal of Electrical and Computer Engineering, Vol. 9, No. 2, 2019, pp. 1012-1020.
[8] T. F. Bastos-filho, A. Ferreira, A.C. Atencio, S. Arjunan, and D. Kumar, “Evaluation of Feature Extraction Techniques in Emotional State Recognition”, in 4th International conference on intelligent human computer interaction (IHCI), 2012, pp. 1-6.
[9] N. Jatupaiboon, S. Pan-ngum, and P. Israsena, “Real-time EEG-based happiness detection system”, The Scientific World Journal, 2013, pp. 1-12.
[10] M. J. Hasan, and J. M. Kim, “A hybrid feature pool-based emotional stress state detection algorithm using EEG signals”, Brain sciences, 2019, Vol. 9, No. 12, pp. 1-15.
[11] D. Shon, K. Im, J. Park, D. Lim, B. Jang, and J. Kim, “Emotional Stress State Detection Using Genetic Algorithm-Based Feature Selection on EEG Signals”, Int. J. of environmental research and public health, 2018, Vol. 15, No. 11, pp. 1-11.
[12] S. Koelstra, C. Muhl, M. Soleymani, J.S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals”, IEEE transactions on affective computing, 2011, Vol. 3, No. 1, pp.18-31.
[13] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, “EEG-based emotion recognition in music listening”, IEEE Transactions on Biomedical Engineering, 2010, Vol 57, No. 7, pp. 1798-1806.
[14] R. Jenke, A. Peer, and M. Buss, “Feature extraction and selection for emotion recognition from EEG”, IEEE Transactions on Affective Computing, 2014, Vol. 5, No. 3, pp. 327-39.
[15] M. Sohaib, C.-H. Kim, and J.-M. Kim, “A Hybrid Feature Model and Deep-Learning-Based Bearing Fault Diagnosis”, Sensors, 2017, Vol. 17, No. 12, pp. 1-16.
[16] J. A. Russell, “A circumplex model of affect”, Journal of Personality and Social Psychology, 1980, Vol. 39, No. 6, pp. 1161–1178.
[17] S.H. Oh, Y.R. Lee, and H.N. Kim, “A novel EEG feature extraction method using Hjorth parameter”, Int. Journal of Electronics and Electrical Engineering, 2014, Vol. 2, No. 2, pp. 106-110.
[18] P.C. Petrantonakis, and L.J. Hadjileontiadis, “Emotion Recognition from EEG Using Higher Order Crossings”, IEEE Transactions on Information Technology in Biomedicine, 2010, Vol. 14, No. 2, pp. 186–197.
[19] H. Shao, H. Jiang, H. Zhao, and F. Wang, “A novel deep autoencoder feature learning method for rotating machinery fault diagnosis”, Mechanical Systems and Signal Processing, 2017, Vol. 95, pp. 187-204.

Full-Text:

Title

An Autoencoder based Emotional Stress State Detection Approach Using Electroencephalography Signals

Jia Uddin1*

1. AI and Big Data Department, Endicott College, Woosong University, Daejeon, South Korea

Received: 17 Nov 2021/ Revised: 04 Mar 2022/ Accepted: 27 Apr 2022

Abstract

Keywords: EEG Signals; Emotion Analysis; Stress Analysis; Autoencoder; Machine Learning.

1- Introduction

For engineering wellbeing, detecting consequences from human error is essential because unsafe and irresponsible manners of employees involved in manufacturing are clearly accountable for human-caused troubles. Among several key factors of these dangerous and irresponsible activities, lack of proper sleep leads a person to an extreme stressful situation. Stress initiates irritation, fear, sadness, vascular illness and numerous additional injurious effects [1], [2]. Numerous forms of brain signals, i.e., functional magnetic resonance imaging (fMRI), near-infrared spectroscopy (NIRS), Electrocorticography (ECoG), and electroencephalogram (EEG), are utilized for evaluating emotional conditions of individual [3]. Among all of these forms of data, EEG can be assessed non-intrusively [4]. The principal objective of this research is to categorize the emotional situation of an individual based on the sex by evaluating pre-processed freely accessible EEG signals.

Several surveys have exhibited relationships between EEG signals and several emotional situations [5–10]. In [5], an EEG-based assessment on the frontal channel with support vector machine (SVM) is designed. In [9], an in-depth analysis of power spectral density (PSD) is proposed to classify the emotional state by SVM. Among these researches, the common attribute is to consider all the features for classifier.

In this research, an EEG signal-driven emotional state classification method is established to evaluate whether a person is experiencing stress. By evaluating the signal, a hybrid feature bag is designed to create a dynamic and robust feature list. This process is divided into two parts: (1) statistical analysis from the time domain, and (2) wavelet-based feature assessment from the frequency domain. In the EEG signals, for the presence of the artifacts [11], it is hard to find the absolute feature information. This study examined pre-processed signals from the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset [12]. The time domain features are defined in detail in Section 3. As the EEG signal has five indistinguishable bandwidths [13], [14], wavelet decomposition is studied to determine the frequency domain features, which are depicted in Section III. From these designed bag of hybrid features, instead of providing all of them to the classifier, some built-in feature reduction mechanism embedded classifier is used to utilize only the most significant features for final classification. Deep networks can obtain extremely characteristic features via their multi-layered model architectures. Moreover, they keep only the most representative information in each layer to reduce the dimensionality and also to improve the classification performance by selecting only the most intrinsic feature information [15]. In this research, a three layered autoencoder based neural network (AENN) is proposed for different emotional state classification. To prove the strength of the suggested technique, few comparisons are made with the approaches discussed into [5], and [9]. This paper's primary contributions can be summarized as follows:

(1) Statistical analysis in the time domain and wavelet-based feature assessment in the frequency domain combine to create a hybrid bag of features.

(2) A two layered AENN is proposed to learn and utilize only the important features through the embedded feed-forward feature selection architecture to improve the final classification accuracy.

The remainder of this paper will be organized as follows. The DEAP dataset is readily available, and the recommended approach is explained in Section 2. Section 3 contains the data agreement, evaluation of the experimental findings, and discussion, and Section 4 concludes the paper.

2- Proposed Method

In Fig. 1, a block diagram proposed approach is provided. The proposed approach is divided into four sections: (1) pre-processed data gathering [12], (2) data arrangement, (3) creation of hybrid bag of features, and (4) AENN-based classification.

The data is first down sampled to 128 Hz, and then the artifacts are eliminated from the data, as seen in this diagram. The current study's annotation was completed after filtering the data using bandpass frequency and common segmentation. The statistical features from the time domain and wavelet-based frequency domain are then examined and retrieved from each class sample. Finally, an Autoencoder-based Neural Network is presented for classification.

Fig. 1 Complete pipeline of the proposed approach.

2-1- Dataset Details

For this study, pre-processed EEG signals from the DEAP dataset [12]. This dataset incorporates emotional responses stimulated by music videos. The information related to the considered dataset is listed into Table 1. The placement of attached sensors for collecting the EEG data, with the real experimental setup is depicted into Fig. 2.

Table 1: Details of the considered dataset

Area	Information
Number of participants	32
Age range	19 – 37 years
Number of male participants	17
Number of female participants	15
Recorded time for EEG	1 minute 3 seconds
Pre-train baseline	3 seconds
Final considered signal length	1 minute
Electrode placement system	10 – 20 system of electrode

Each contributor viewed 40 music videos. The dataset retains pre-processed data that down sample the recorded signals to 128 Hz. In this research, the pre-processed EEG signals are considered. For this experimentation, the dataset has to be annotated. From the analyzed valence and arousal level from the recorded EEG response calm and stress states have been specified by Equation (1) and (2) [8], [16].

Seven participants do not indicate any mental state of calm or stress after separating the data into two categories. The dataset is ready for the categorization mission, which is represented in Section 3, with the remaining over 25 contributors.

2-2- Hybrid Bag of Features

From time domain, the obtained numerical features are root mean square (F1), kurtosis (F2), skewness (F3), shape factor (F4), and impulse factor (F5). In addition to these, the mobility (F6) of the signal is also counted, which includes the information of the frequency spectrum [11], [17], [18]. The statistical details of these 6 features are described into Table 2.

EEG signals are distributed into 5 frequency bands, i.e., delta, theta, alpha, beta, and gamma [13], [14]. Here, as the sampling frequency of considered dataset is 128 Hz, level 5 wavelet decomposition facilitates to accumulate these 5 frequency bands. From the analyzed wavelet coefficients, wavelet energy (F7) and standard deviation (F8) is considered.

2-3- Classification by Autoencoder based Neural Network (AENN)

Autoencoder is mainly an unsupervised algorithm which learns the representation of the data by minimizing the reconstruction error from the layered architecture. It takes

an input value and then by using a function in encoded the input value as . Then, that encoded value turns into an output value , which is identical to the input. The main goal of autoencoder is to make the output value very similar to the input value by minimizing the reconstruction error. When it finally can be able to make the best reconstruction output, then the encoded value from the encoded layer, learns the best data representation. In other words, it recreates the input from the encoded output appear in the encoded layer. Encoded layer produces a brand-new bag of features which is a mixture of the initial features. The encoded layer can be expressed by Equation (3):

	(1)
	(2)

(3)

Here, is the input with the dimension of , and then the encoded layer maps the input data to encoded latent variable , where dimension reduced to . is weight and is the bias here.

In this work, the same mechanism is deployed [15], [19]. The encoded layer latent feature representation is then passed to the SoftMax classifier for final classification. In the proposed AENN, two layers are used to learn the latent feature space in unsupervised way. The main architecture of the proposed AENN is illustrated into Fig. 3.

Fig. 2 Real experimental testbed with subject for collecting EEG data.

Table 2: Statistical explanation of the measured time-domain features

Table 3: Particulars of the considered dataset

Feature	Equation	Feature	Equation	Feature	Equation
F1		F2		F3
F4		F5		F6
Here, x is the time-domain raw signal. N is the total number of samples.

Dataset (Sex-based)	Sub-set	Participant ID	An Experimental ID System That Reflects the Unique State
Dataset (Sex-based)	Sub-set	Participant ID	Calm	Stress
Male	1	1	9,14	17,32,34,35,36,37
	2	5	13,29	23,30,37
	3	12	16,17,28	25,29,32,33,35,36,37,38
	4	16	6,12,16,21,36	1,15,17,24,26,27,34
	5	18	22,26,34	30
	6	19	15,26,27	29,38
	7	20	16,26,27,28,40	23,25,29
	8	21	3,21,26,34,35	20,22,24
	9	26	30	34
	10	27	5,15,19,26,27,28,33,40	27
	11	28	15,22,24,25	35,38
	12	29	15,17	30,31,33,35
Female	1	2	5,7,10,22,24,36	29,30,32,37,38
	2	4	2,6,18	24,28,32
	3	8	10,37,39	31,36
	4	10	15,17,20,22,26,27,28	21,30,35,36,37,38,39
	5	11	2,12,16,19,25,26,28,40	27,35,37,38,39
	6	13	12,15,16	7,21,23,31,34,35,36,37,38,39
	7	14	22,27	10,21,23,24,29,30,32,34,35,36,38
	8	15	7,16,22,26	24,25,30,38
	9	22	1,6,12,15,16,28	23,24,29,30,32,33,35,36,37,38,39
	10	24	33,40	21,23,24,30,31,38,39
	11	25	4,5,26,27,28,34	2,10,23,29,31,32,33,37,38,39
	12	31	17,22,24,27,28,29	23,32,34,37,38,39
	13	32	2,6,15,26,33	24,30,37

Fig. 3 AENN model architecture.

Fig. 4 10-fold cross validation [10].

3- Experimental Result Analysis

3-1- Arrangement of Dataset

Among 32 contributors [11], contributor numbers 3, 6, 7, 9, 17, 23, and 30 did not signify the attributes of considered emotional states. So, the left over 25 contributors are counted for the final dataset. 32 EEG channels are considered for considering the data with a 128Hz sampling frequency. The particulars of the considered dataset are given in Table 3. The Table reveals that each sub-dataset has a distinctive experimental ID, and for each ID, different music videos are reliable for calm and stress state. Therefore, for sex-based state analysis according to the identity of sex; two main datasets are considered for final classification, i.e., dataset male (consisting of 12 sub-sets, merged together to form male dataset as a whole), and dataset female (consisting of 13 sub-sets, merged together to form male dataset as a whole).

3-2- Performance Analysis of AENN

Each dataset is split into two parts: training and testing, with a 70/30 split between the two. Equation (4) establishes the class-dependent accuracy.

To ascertain the final average accuracy, Equation (5) is utilized.

(4)

Using 10-fold cross-validation, the ultimate accuracy is calculated. K-fold [10] cross-validation, as illustrated in Figure 4, involves randomly dividing the training set into ten groups, or folds. The ultimate accuracy is calculated using 10-fold cross-validation. The training dataset is then divided into two parts: P for training and Q for validation. Training folds P are used to create the model, while validation fold Q is used to validate it. The AENN's parameters are tuned using the validation fold B. Every iteration (10 times), the validation fold is rotated, and the remaining data is utilized to train the AENN. The specifics of the final accuracy after 10-fold cross-validation are provided in Table 4.

Table 4: Classification accuracy of the proposed method

(5)

Dataset	Class-wise Accuracy (%)		Average Accuracy (%)
Dataset	Calm	Stress	Average Accuracy (%)
Male	78.25	75.93	77.09
Female	79.42	82.44	80.93
Average (%)			79.01

The feature embeddings extracted from the AENN encoded layer (2 features values are extracted from encoded layer) is displayed into Fig. 5 for both of the datasets.

In addition to these, to establish the robustness of this approach, few comparisons are made with [5] , and [9].The details of this comparative analysis are depicted into Table 5.

Table 5: Relative evaluation of various methodologies

Methods	Average Accuracy (%)	Decrement from the Proposed Method (%)
[5]	68.23	10.78
[9]	72.44	6.57
Proposed	79.01	-

Fig. 5 AENN feature embedding from encoded layer for (a) male dataset, and (b) female dataset.

3-3- Discussion

From Table 4, it is visible that the female is performing better than the male dataset. From Table 3, it is visible that the female dataset contains 13 sub-sets where the male dataset contains 12 sub-sets. To get a better performance from deep networks, amount of dataset is a vital issue. Moreover, for female dataset case, stress state is giving higher accuracy than the calm state. From Table 3, it is clearly visible that the stress state data is larger than the calm state data for female dataset (61 cases for calm state and 84 cases for stress state).

From Fig. 5, for both of the dataset, the feature embeddings are overlapped. That indicates the separation of identical features are not good. The main underlying reasons behind this is little analytical. If the sub-sets are being analyzed from Table 3, for each sub-set, the experiment IDs of calm and stress state are different. In addition, sometimes it is imbalanced as well. So, in this dataset not any particular experiment ID is responsible for clam or stress state. It makes difficult to find out the most intrinsic class-wise information in a very accurate manner.

For comparative analysis, the proposed model outperformed the approach described in [5] by 10.78%, and the approach described in [9] by 6.57%.

4- Conclusions

This paper proposed a sex-based stress state classification method by analyzing EEG signals from brain. First it created a bag of features from the statistical analysis of time-domain and wavelet-based analysis of frequency domain. Therefore, the hybrid bag of features was forwarded to the proposed AENN to identify the stress state by feed-forward architecture based selective features. The proposed approach achieved an accuracy of 77.09% for male dataset and 80.93% for female dataset. In addition to that, it outperformed several existing methods related to this study by at least 6.57%.

Acknowledgments

This research is funded by Woosong University Academic Research in 2022.

References

[1] R. E. Bender, and L. B. Alloy, “Life stress and kindling in bipolar disorder: review of the evidence and integration with emerging biopsychosocial theories”, Clinical psychology review, Vol. 31, No. 3, 2011, pp. 383–398.

[2] T. G. Pickering, “Mental stress as a causal factor in the development of hypertension and cardiovascular disease”, Current hypertension reports, Vol. 3, No. 3, 2001, pp. 249–254.

[3] T. C. Major, and J. M. Conrad, “A survey of brain computer interfaces and their applications”, in IEEE SOUTHEASTCON 2014, 2014, pp. 1–8.

[4] N. Elsayed, Z. S. Zaghloul, and M. Bayoumi, “Brain computer interface: EEG signal preprocessing issues and solutions”, Int. J. Comput. Appl, Vol. 169, No. 3, 2017, pp. 12–16.

[5] R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, “Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style”, Psychophysiology, Vol. 30, No. 1, 1993, pp. 82–89.

[6] A. C. Atencio et al., “Computing stress-related emotional state via frontal cortex asymmetry to be applied in passive-ssBCI”, in 5th ISSNIP-IEEE Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), 2014, pp. 1–6.

[7] F.P. George et al. “Recognition of emotional states using EEG signals based on time-frequency analysis and SVM classifier”, International Journal of Electrical and Computer Engineering, Vol. 9, No. 2, 2019, pp. 1012-1020.

[8] T. F. Bastos-filho, A. Ferreira, A.C. Atencio, S. Arjunan, and D. Kumar, “Evaluation of Feature Extraction Techniques in Emotional State Recognition”, in 4th International conference on intelligent human computer interaction (IHCI), 2012, pp. 1-6.

[9] N. Jatupaiboon, S. Pan-ngum, and P. Israsena, “Real-time EEG-based happiness detection system”, The Scientific World Journal, 2013, pp. 1-12.

[10] M. J. Hasan, and J. M. Kim, “A hybrid feature pool-based emotional stress state detection algorithm using EEG signals”, Brain sciences, 2019, Vol. 9, No. 12, pp. 1-15.

[11] D. Shon, K. Im, J. Park, D. Lim, B. Jang, and J. Kim, “Emotional Stress State Detection Using Genetic Algorithm-Based Feature Selection on EEG Signals”, Int. J. of environmental research and public health, 2018, Vol. 15, No. 11, pp. 1-11.

[12] S. Koelstra, C. Muhl, M. Soleymani, J.S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals”, IEEE transactions on affective computing, 2011, Vol. 3, No. 1, pp.18-31.

[13] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, “EEG-based emotion recognition in music listening”, IEEE Transactions on Biomedical Engineering, 2010, Vol 57, No. 7, pp. 1798-1806.

[14] R. Jenke, A. Peer, and M. Buss, “Feature extraction and selection for emotion recognition from EEG”, IEEE Transactions on Affective Computing, 2014, Vol. 5, No. 3, pp. 327-39.

[15] M. Sohaib, C.-H. Kim, and J.-M. Kim, “A Hybrid Feature Model and Deep-Learning-Based Bearing Fault Diagnosis”, Sensors, 2017, Vol. 17, No. 12, pp. 1-16.

[16] J. A. Russell, “A circumplex model of affect”, Journal of Personality and Social Psychology, 1980, Vol. 39, No. 6, pp. 1161–1178.

[17] S.H. Oh, Y.R. Lee, and H.N. Kim, “A novel EEG feature extraction method using Hjorth parameter”, Int. Journal of Electronics and Electrical Engineering, 2014, Vol. 2, No. 2, pp. 106-110.

[18] P.C. Petrantonakis, and L.J. Hadjileontiadis, “Emotion Recognition from EEG Using Higher Order Crossings”, IEEE Transactions on Information Technology in Biomedicine, 2010, Vol. 14, No. 2, pp. 186–197.

[19] H. Shao, H. Jiang, H. Zhao, and F. Wang, “A novel deep autoencoder feature learning method for rotating machinery fault diagnosis”, Mechanical Systems and Signal Processing, 2017, Vol. 95, pp. 187-204.

* Jia Uddin

jia.uddin@wsu.ac.uk

Links

Related Centers

Technical Support

Official pages

Share To

Article Url

An Autoencoder based Emotional Stress State Detection Approach by using Electroencephalography Signals

Rimag