HSERNet: Hybrid Speech Emotion Recognition using Conformer-Convolutional Neural Networks

HSERNet: Hybrid Speech Emotion Recognition using Conformer-Convolutional Neural Networks | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 25 February 2025 V1 Latest version Share on HSERNet: Hybrid Speech Emotion Recognition using Conformer-Convolutional Neural Networks Authors : Manikanta Kadamba [email protected] and Padmaja Madugula 0000-0002-9188-2410 Authors Info & Affiliations https://doi.org/10.22541/au.174048503.32222055/v1 341 views 135 downloads Contents Abstract Introduction Related work PROPOSED METHODOLOGY Results and Discussion CONCLUSION FUTURE ENHANCEMENT References Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract Recent development of speech emotion recognition (SER) causes have shifted more and more to enhancing the computer’s understanding of people’s emotions by using their voice. This paper proposes a new SER algorithm that reduces the difficulty of the process by directly working with the raw speech data rather than depending on the human expert’s choice of the acoustic features. The method proposed is therefore a Conformer-CNN model where one part derives from the Conformer block, which is adept at learning long-term temporal features, and the other from the CNN that is proficient at capturing localized emotional features from audio inputs. The Conformer block encodes speech in a dynamic domain and retain its contextual information within the spectrogram, while the CNN layers are tailored for extracting fine-grained emotional patterns from the spatial domain of spectrograms. Combined, these components provide the ability to capture fine-grained and coarse-grained emotion patterns, which in turn makes the analysis of speech data more profound. This combination of approaches allows the application to capture the rather complex interactions between human emotions and all the non-static features of speech safely. When tested on multilingual dataset, the algorithm demonstrated the improvements in terms of the accuracy and the interpretability as well as the ability for capturing temporal and affective contexts. This research benefits the general and wide field of human-computer studies and interaction to wit, refining reaction of machineries to human emotions more appropriately and accurately. Introduction SER has raised its role as one of the key components of affective computing that tries to close the gap between expressed emotions and their computational recognition. Cross sectionally emotions constitutes a core component of the human communication, interpersonal relations, business and personal decisions and health status of people. Arming machines with capabilities to identify emotion-related stimuli improves their ways of synchronizing with people, potential uses of HMI, mental health assessment, customer service, and so on. In the conventional approach to building SER systems, hand crafted acoustic features like pitch, energy and spectral characteristics of the speech are used for analysis. Although these methods proved moderate effectiveness they have following drawbacks. Most manually designed features are intrinsically unable to encode the complex and temporal characteristics of emotions inherent in raw speech data. In addition to this, these approaches elicit issues of applicability in a variety of languages, as well as dynamism when applied in various settings. This confines their use especially in multilingual and in noisy environments where emotions and their variations are more delicate. Besides, interpretation is a challenge that traditional approaches have failed to meet adequately; a significant consideration, especially when it comes to constructing trustworthy emotion recognition systems. To this end, this research presents an enhanced Speech Emotion Recognition system that feeds directly on the raw speech signals negating the need for these acoustic features. This system’s core is a brand-new architectural concept based on Conformer blocks and Convolutional Neural Networks (CNNs). The Conformer block which is built to capture long-term temporal and dynamic speech patterns defines the temporal and contextual flow of speech well. However, CNN layers have focused more on capturing local and hierarchical emotional information and more convenient for detecting complex pattern in the raw signal. Combining these techniques into the one framework gives the proposed architecture a holistic assessment of both emotional and temporal aspects in speech. Conformer-CNN is presented as an improved THE technology by integrating a Conformer front-end with a CNN acoustic model, using raw waveforms. This exclusion of bias helps overcome problems of feature selection, also deepens the models ability to discern more subtle and significant emotions. Temporal relationships are well managed in the Conformer block, and thus, the system accurately captures the temporal evolution of emotions in speech, the characteristics of which are managed effectively by the CNN layers to capture local spatial dependencies. This makes the model capable of maintaining general and specific representations regarding the emotional content to make better and easy to explain forecasting. In order to assess the merit of the proposed technique, the system was examined on several open source multilingual datasets. That is these datasets originates from different linguistic and cultural environments thus making the architecture both robust and scalable. The results showed a marked gain in accuracy and interpretability relative to conventional techniques where the hybrid structure was again confirmed that can deal with actual issues in SER. Compared to that, by processing the raw audio signals directly, the proposed system is able to outperform the hand-crafted feature extraction method in terms of generalization across languages and datasets. Utilization possibilities of SER system Proposed are presented in various fields. In human-computer interaction, technologically developed objects like adaptive virtual personalities, including virtual assistants and chatbots, can get an emotionally intelligent user experience. In the aspect of mental health surveillance, the system can help in elucidating the emotional status possibly holding the capacity to be used as an early intervention and support tool. In customer service, for instance, emotion recognition can help increase user satisfaction by allowing agencies to modify their replies depending on the emotional context of the communications. The four use cases discussed above capture more complicated Scenarios and show the usefulness of the proposed approach in real-world scenarios. In addition, the framework based on the hybrid architecture opens the way to the further development of the Speech Emotion Recognition. Consequently, its ability to take in raw speech signals makes it possible to create real-time emotion detection systems that are efficient in dynamic contexts. Also, the model’s capability of being extended to other languages and datasets render it a suitable candidate for multilingual SER, a problem not given adequate attention in present research. Therefore, this study contributes to the body of knowledge in Speech Emotion Recognition since it develops a new model, specifically the Conformer-CNN that directly processes speech data in order to classify emotions. The proposed system combines the ability of Conformer blocks and CNNs to model long temporal dependencies as well as capture local emotional patterns, overall giving a richer analysis of the input signals. The architecture reduces dependence on handcrafted features major hurdles with traditional approaches and improves the system’s accuracy out of bound while making it scalable and easy to interpret. When compared with other multilingual datasets, this study clearly shows the effectiveness of the proposed model for a variety of applications in the affective computing domain. Not only does this work improve the performance SER systems but it also opens up many directions for future research in languages, real-time and scalability of emotion recognition systems. Therefore, through this approach the project seeks to advance the implementation of the SER in areas such as: HCI, mental health, and customer service, so that systems may become more emotionally intelligent to better interpret human emotions. Related work SER has become popular in affective computing research because of its uses in HCI and other affect-based smart systems. Several techniques have been suggested to improve the recognition of emotions, using acoustic, textual, and both acoustic-textual features. These findings have established beyond doubt the fact that linguistic features are central to SER. In the previous work, Sakurai and Kosaka [1] showed an increase in accuracy by using both the acoustic and linguistic features for the recognition of emotions even if there are recognition errors. In a similar vein, Kurian and Tripathi [2] suggested a framework for affect recognition using both facial and vocal cues achieving substantial improvements in recognising emotions exhibited by autistic children. They also have great influence on SER performance, where hybrid architectures have been exposed to give better results. Norval and Wang [3] earlier came up with the Dendritic Convolutional LSTM (DCLSTM) architecture which paid attention to segregation of both the spatial and temporal features of an audio clip with more focus to the pre-processing of the audio data. Galić et al. [4] investigated issues related to emotional speech recognition through the use of methods such as pitch shifting and time stretching; the authors herein reported enhanced accuracy [5]. Even more, deep learning-based methods improve SER. A novel hybrid CNN-LSTM framework was presented in [6] for multimodal emotion analysis for speech, text, and video data in order to enhance the accuracy of emotion recognition. Duan et al. [7] employ deep learning and the Transformer models to incorporate emotions into the User Interface. Modification of the conventional feature extraction techniques has also been considered. Meanwhile, Halimah et al [8] examined newer MFCC feature extraction modification for SER in Indonesian language by using single-layer LSTMs and found greater accuracy. Wang et al. [9] presented an IERP Challenge obtaining a DWFormer model with audio-only features fine-tuned to the use of pretrained models in SER. Finally, the integration of personality traits into SER system is a promising avenue further discussed by Cai et al [10] because it encourages system flexibility in responding to individual difference and suggests that a framework for emotion recognition should be personalized. A compilation of these advancements is given below which collectively illustrate how SER is progressing with different modalities and its applications. SER has attracting a lot of interest in the field of affect computing because of the key interest in improving human computer interaction as well as affective’ intelligent systems. A lot of improvement has taken place over the use of different modalities and the enhancement of deep learning in the development of SER systems. In the present day, several developments have been made based on the self-supervised learning (SSL) models for SER. For ECGI and IEMOCAP datasets, Zhang et al. [11] introduced shallow Wav2vec 2.0 structures that involve an attentional mechanism to effectively learn both local and global emotional information. Similarly, Xia et al. [12] compared the results of multimodal emotion recognition based on text and audio features including BERT and data2vec2.0 have shown good improvements of accuracy rates through the use of contextual data. On similar notions, techniques like residual vector quantization (RVQ) have also been utilised in order to enhance emotional representation in SER. Xie et al. [13] proposed ERVQ, an emotion recognition model that uses multi-scale emotional detail extraction; this model was superior to baseline, pre-trained speech EMs in in-domain and OOD cases. Furthermore, Rajapakshe et al. [14] discussed issues involved in domain adaptation processes of real-world SER applications through the use of reinforcement learning techniques that realized enormous improvement in both the cross-corpus and cross-language experiments. In specific, Sharan et al. [15] used Mel-spectrograms and a CNN-RNN model for SER. They identified spectral features through CNNs and temporal behaviors through RNNs; the work yielded high efficacy on data sets including EMODB and RAVDESS. Chyan et al. [16] further generalized this concept by incorporating both linguistic and paralinguistic information in a soft voting ensemble model, thereby increasing the recognition accuracy of signals because of combined contextual and prosodic data. Multi-scale emotion modeling that combines utilizing coarse and fine-grained embeddings has been investigated by Yin et al. in the context of the HuBERT-related research focused on emotional speech synthesis [17]. Their multi-speaker synthesis system proved the novelty of using speech representations of voice source for designing SER programmes that are empathetic and contextual. In their recent work, Hirano et al. [18] pointed out that the number of speech and text features combined in an integrated model for empathetic dialogue systems. Generally, their model was more accurate in contrast to the baselines of single-modality, and these results pointed to the need of data fusion to enhance SER performance. Other advanced approaches to computing feature attributes of a document have improved SER to higher levels. In another study, Rudd et al. [19] proposed Mel-spectrogram plus harmonic-percussive component analysis (HPCA) coupled with variational mode decomposition (VMD) for enhancing acoustic features with a high classification rate. A hybrid CNN-GRU method was proposed by Namey et al. [20] where multi-head attention-based cochleagrams and spectrograms for emotion recognition specifically for Bangla and English datasets. Other research has also looked into issues such as architecture enhancements of the neural network for SER. Sequential and real time emotion recognition using LSTM and CNN-LSTM models were studied by Sharma et al [21] and Khan et al [22] also assessed the impact of various optimizers to boost the performance of RNN, CNN and MLP based SER systems. These efforts combined show the active progress of SER confirming the utility of using multi-modal data, deep and recent models and featuresto enhance the robustness of emotions recognition systems in different domains. Dataset The RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) is well-known dataset used for Speech Emotion Recognition (SER) task. It comprises recordings from 24 professional actors, equally split between 12 males and 12 females, delivering vocal expressions in eight distinct emotions: joy, sorrow, rage, fear, surprise, dislike, content, and nothing. Such balanced representation guarantees comprehensive analysis of the different variations in the emotional expressions in vocals. The dataset in addendum consists of 1440 audio files- 960 speech and 480 songs. The two scenario specific files contain both emotional and neutral set which makes it a rich dataset where models can be trained and tested for emotion classification. The recordings are in WAV format and are of good quality with a sample rate of 48 kHz to capture all the details needed for specific acoustics investigation. Since RAVDESS deals with wooden facial expressions, it is highly appreciated for the diversity and realism of presented emotions and is often used as a reference for new methods in affective computing. Due to increasing trends and structures, its content and its uniform format increase the consistency and efficiency of model training for improving SER systems. This dataset has been essential in contributing to knowledge in the area of vocal emotion recognition and use in HMI and mental health and related areas . PROPOSED METHODOLOGY This proposed SER system consists of Conformer blocks and Convolutional Neural Network (CNNs) to extract features from raw speech inputs. Such approach avoids the need of feature engineering stressed out that the model automatically decided about localized and long-term temporal features important for classification of emotions. The stages include data pre-processing,eration of detail architecture, optimisation and evaluation of the model . Figure 1 Architecture of Hybrid Conformer-CNN Data Preprocessing The input layer binds data in the form of 1D containing raw audio without the need for any structural adjustment. Each waveform shape is of size (nsamples, time_steps, 1) the architecture as used the ‘nsamples’ to represent an audio sample, ‘time_steps’ as the length of the sample. Due to this, the audio signals are normalized to have a standard range because large input values are known to destabilize training. The preprocessed data is then reshaped to (time_steps, 1) thus preserving time series nature of the data after normalization. It enables the input imagery to be ready for the subsequent neural network layers in order inputs taken by the system . Model Architecture To extract raw feature representations of the speech data, the architecture incorporates Conformer blocks and CNNs to study both local and global patterns. A step-by-step breakdown of the architecture is provided below: Initial 1D Convolutional Layer The first layer extracts low-level features from raw audio, such as phonetic characteristics. This 1D convolutional layer uses: • 512 filters with a kernel size of 3, enabling detailed feature extraction from short segments of audio. • MaxPooling1D is applied after convolution to downsample the feature maps, reducing dimensionality while preserving critical information. • Batch Normalization stabilizes the training process by normalizing layer inputs, and the Swish activation function introduces non-linearity, enhancing the model’s capacity to learn complex patterns. Conformer Block As a recurrent block, the Conformer is instrumental to capturing both the long-term dependencies and contextual connections in the function of the sequence of audio. It is composed of: • Convolutional Layer: The first part of the Conformer block, which extracts local features from input sequences. • Multi-Head Attention Mechanism: Bridges temporal dependencies in the same way in which it allows the network to attend to different parts of the input sequence. The parallel processing is essential in tracking development of emotions over time. • Feed-Forward Network (FFN): Exocorts the attention outputs to gather more complex interactions and improve the feature’s dimensions. • Residual Connections: To provide efficient gradient flow, the attention and feed-forward layer’s outputs reinsert to the input through skip connections. CNN Block Following the Conformer block, the CNN block extracts localized emotional cues such as changes in pitch, tone, and rhythm. This block consists of: • A 1D convolutional layer for extracting features from smaller regions of the input. • Batch Normalization to stabilize training, followed by max-pooling to further downsample the data and emphasize key features. • A dropout layer is included to reduce overfitting by deactivating random neurons during training. Hybrid Conformer-CNN Integration The outputs from the Conformer and CNN are then summed together so that the residues from each can be used for features from both global and local levels. Since feature maps from these blocks may differ in size: • Up-sampling or Cropping helps in Factor to match up the feature map dimension sizes. • This is attainable through a following 1D convolutional layer with kernel size of 1 to enhance compatibility for integration. • Conformer CNN skips directly connect the output of the Conformer and CNN blocks, while maintaining dependencies with long distances and local neighborhoods. Global Pooling and Dense Layers The combined feature maps are taken to a GlobalAveragePooling1D layer to reduce the amount of information along the temporal dimension. It makes it possible for the model to capture global dependencies at a sequence level of the input. A dropout layer comes next in order to avoid cases of overfitting of the model. The output of this search space is fed through a dense layer with 1024 Swish activation as the final fully connected layer before classification. Output Layer The last layer, that is, the output layer utilizes softmax activation function from which probability of each of predefined emotion class is derived. Neurons can be directly linked to categories of emotions (happy, sad, angry, etc.) In this case the amount of neurons corresponds to the amount of emotion categories. Loss Function and Optimization The model applies the categorical cross-entropy technique as the loss function, most appropriate for multi-class classification problem. For the purpose of enhancing the training process, the AdamW optimizer is applied, which includes weight decay. In order to enhance the convergence the ReduceLROnPlateau learning rate scheduler adjusts learning rate using the validation requirements. Further, early stopping is employed to prevent training if the validation performance is not enhanced through next epochs to save computation time. Training Procedure The model used has a batch size of 32 for training and a maximum number of 50 epochs. L2 regularization were used in the dense layers to eliminate overfitting and dropout was also used. For rate learning and early termination, cabs are provided to guarantee effective training. Evaluation Metrics As the main criterion for evaluation, accuracy is used to measure performance on the developed model. Further on, and to consider threshold-dependent performance, accuracy, precision, recall, and F1 scores are measured for each class. Confusion matrix gives an understanding of the manner in which the model distinguish emotion classes. Results and Discussion Figure 2 Confusion Matrix Widely geometrical with emotions identified and analysed using the confusion matrix as demonstrated above, the Conformer Transformer model for Speech Emotion Recognition (SER) ascertained high accuracy in most emotions. The concept of each row in the matrix symbolized as the true emotion and each column represented the predicted emotion. Diagonal values highlighted in yellow represent correct classification, where the predicted label is the same as the actual label while other values represent wrong classification. The matrix also shows that according to the model most of the diagonal values are closed to 1 pointing out that the model earns an overall accuracy. For example, the model recognizes “angry” and “surprised” emotions with 97% accuracy, thereby proving that this model able to capture these emotions correctly. Similar to the previous categories, there are also high levels of identification with other feelings: “calm” -95% and “neutral” -87% proving the efficiency of the model to identify a rich set of emotions. However, the off-diagonal values which involves misclassification is very low. For example, “neutral” is sometimes mistaken for “calm” (7%), or close emotions such as “fearful” and “sad” are misclassified. These errors should be attributed to acoustic similarity of certain emotions which remain difficult even for sophisticated architectures. In totality, the confusion matrix shows the stability of the mode and its high flexibility in classifying particular emotional statuses. The overall high accuracy for most classes, and the absolute minimum of misclassification points to the fact that the Conformer Transformer model is well suited for SER tasks, and is able to capture both subtle and prominent feelings from the content of speech information. Figure 3 Classification Report The classification report gives the performance assessment of the Conformer Transformer model for emotion recognition task with a high over-all test accuracy of 91.54%. Authors provide the analysis of such essential performance measurements as precision, recall, and F1-score for each of the emotion classes and provide detailed understanding of the model’s performance in detecting and differentiating between different emotions. Once more, the model shows high accuracy in identifying ‘angry’, ‘calm’, and ‘surprised’ moods because all the F1-scores are high, and fluctuating around 0.95. This means it has a good trade off between precision where positive true predictions are well identified and recall where the actual positives found are well identified. These high F1-scores indicate how strong and reliable the proposed model is in identifying these feelings with minimal levels of mistake. The “sad” emotion, however, yielded the smallest F1-score of 0.84, which indicates that the network has some level of trouble in correctly categorizing the emotion from others. This can be explained by the fact that some concepts might belong to more than one category – for example, “sad” might be confused with “neutral” or “fear” emotions, which is typical for the majority of the experiments devoted to recognizing emotions. This macro average F1 score of 0.91 can also be seen as a reinforcement of the fact that this model gives equal importance to each of the class. At the same time, the globally averaged F1 score of 0.92 shows its availability in the scenarios when some of the emotions being classified are possibly more frequent than others. In sum, with an aid of the proposed Conformer Transformer model, the classification report highlights that the suggested approach performs very well for emotion recognition tasks. That it can perform high accuracy, recall, and F1-scores on most classes evidences its flexibility enough to be recommended as a suitable method that can be used in recognizing emotional signs in spoken material. The fact that there are only a number of difficulties involved in accurately identifying such contextual emotions as ‘ ‘sad’ ’ indicates that there is room for improvement and that the use of additional information or tuning of some pieces of architecture could help do so. CONCLUSION The introduced SER system evidences the prototypes that advance the arena of emotionally incompetent human-computer interaction is a reality of the near future. Being a Conformer-CNN based system, the proposed framework captures local emotion-related features as well as longer temporally dependent patterns from raw speech signals. This reduced the dependence on manually crafted features which enabled the model to self-learn relevant patterns which makes it portable and shift from one setting to the next, or one language to another. This is evidenced from confusion matrix and classification report which shows high classification accuracy-of the system and the ability to use robust performance metrics that classify and recognize the various emotions among humans. This capability is crucial for the development of conversational systems like virtual agents, self-help.counseling and other client-server applications where ability to recognize and adapt to mood may improve user engagement and satisfaction. Nonetheless, challenges only in the classification of similar emotions like “sad” highlight minor issues of the efficient operability of the program. These gaps may be resolved further enhancing the architecture, adding more contextual information or indeed utilizing the multimodal inputs like facial recognition or textual inputs. Altogether, the system is significant in the roadmap to humanizing HCI’s by creating an interface that comprehends users’ feelings and even their context, thus reaching out a goal to enhance the general user interfaces and make them smarter and more perceptive towards users’ states, and as a result, developing better and more profound and responsive relationship between people and technologies. FUTURE ENHANCEMENT As for future work, the UASDS based SER system could consider fusing the multimodal feature inputs including facial and body movement, and textual context for better Emotion recognition results. The integration of these additional modalities with the speech data could improve the understanding of human emotions because it can easily differentiate between ‘sad’ and ‘neutral’ emotions that are usually ambivalent. Real-time emotion detection is another improvement on the current one that allows dynamic use in virtual assistants, mental health, and adaptive learning applications. Extending the architecture to suit deployment on devices at the edge of the network or in the mobile framework can also enhance deployment and utilization. Moreover, some finer tuning could be applied by employing unsupervised or self-supervised learning in order to improve the model’s performance as to diversify or more rare types of emotions expressed in different cultures and languages. The training of the system may also need to incorporate additional samples which are naturalistic or translated in multiple languages to complement the system’s performance in real world or international environment. References 1. [1] M. Sakurai and T. Kosaka, “Emotion Recognition Combining Acoustic and Linguistic Features Based on Speech Recognition Results,” 2021 IEEE 10th Global Conference on Consumer Electronics, GCCE 2021 , pp. 824–827, 2021. [2] A. Kurian and S. Tripathi, “m_AutNet – A Framework for Personalized Multimodal Emotion Recognition in Autistic Children,” IEEE Access , 2024. [3] M. Norval and Z. Wang, “Hybrid Architecture and pre-processing for Speech Emotion Recognition,” 2024 International Conference on Advanced Mechatronic Systems (ICAMechS) , pp. 31–36, Nov. 2024. [4] J. Galić, S. Šajić, and B. Marković, “Exploring the Impact of Data Augmentation Techniques on Emotional Speech Recognition,” 2024 32nd Telecommunications Forum (TELFOR) , pp. 1–4, Nov. 2024. [5] C. Vaidya, D. N. Haldar, P. Y. Lohi, A. M. Rahman, P. Harshita, and D. K. Siriya, “SpeakSmart: Empowering Public Speakers, Elevating Every Speech,” 2024 8th International Conference on Computational System and Information Technology for Sustainable Solutions (CSITSS) , pp. 1–6, Nov. 2024. [6] A. N, C. M. Mohana, A. Sharma, A. Sharma, G. Mittal, and Navadhesh. M, “SER Analysis -An Approach to enhance Human Health,” 2024 5th International Conference on Data Intelligence and Cognitive Informatics (ICDICI) , pp. 513–518, Nov. 2024. [7] S. Duan, Z. Wang, S. Wang, M. Chen, and R. Zhang, “Emotion-Aware Interaction Design in Intelligent User Interface Using Multi-Modal Deep Learning,” 2024 5th International Symposium on Computer Engineering and Intelligent Communications (ISCEIC) , pp. 110–114, Nov. 2024. [8] N. S. Halimah, R. Hidayat, and I. W. Mustika, “Performance Investigation of Customized MFCC Feature Extraction in Recognizing Indonesian Conversational Emotion,” 2024 16th International Conference on Information Technology and Electrical Engineering (ICITEE) , pp. 207–211, Oct. 2024. [9] H. Wang, X. Jia, J. Deng, and R. Zheng, “Speech Emotion Recognition using Fine-Tuned DWFormer: A Study on Track 1 of the IERP Challenge 2024,” 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP) , pp. 403–407, Nov. 2024. [10] C. Cai et al. , “IERP 2024: Induced Emotion Recognition with Personality Characteristics Challenge 2024,” 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP) , pp. 413–416, Nov. 2024. [11] Y. Zhang, M. Jia, X. Cao, and Z. Zhao, “Speech Emotion Recognition Based on Shallow Structure of Wav2vec 2.0 and Attention Mechanism,” 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP) , pp. 398–402, Nov. 2024. [12] Z. Xia, Z. Ma, Z. Zheng, and X. Chen, “Improving Emotion Recognition with Pre-Trained Models, Multimodality, and Contextual Information,” 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP) , pp. 636–640, Nov. 2024. [13] J. Xie, Y. Xiang, H. Wang, X. Wu, Z. Wu, and H. Meng, “ERVQ: Leverage Residual Vector Quantization for Speech Emotion Recognition,” 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP) , pp. 456–460, Nov. 2024. [14] T. Rajapakshe, R. Rana, S. Khalifa, and B. W. Schuller, “Domain Adapting Deep Reinforcement Learning for Real-World Speech Emotion Recognition,” IEEE Access , vol. 12, pp. 193101–193114, 2024. [15] R. V. Sharan, C. Mascolo, and B. W. Schuller, “Emotion Recognition from Speech Signals by Mel-Spectrogram and a CNN-RNN,” 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , pp. 1–4, Jul. 2024. [16] P. Chyan, R. Y. Carolus, and S. Mallu, “Emotion Recognition from Human Speech Using Linguistic and Paralinguistic Features Through a Soft Voting Approach,” 2024 11th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI) , pp. 481–486, Sep. 2024. [17] M. Yin, Z. Huang, K. Lu, and H. Zhou, “Multi-speaker Emotional Speech Synthesis Based on Self-Supervised Speech Representations,” 2024 IEEE 5th International Conference on Pattern Recognition and Machine Learning (PRML) , pp. 469–473, Jul. 2024. [18] “Multimodal Emotion Recognition in Empathetic Dialogue: An Integrated Approach Using Speech and Text | IEEE Conference Publication | IEEE Xplore.” Accessed: Jan. 04, 2025. [Online]. Available: https://ieeexplore.ieee.org/document/10773744 [19] D. H. Rudd, X. Gao, M. R. Islam, H. Huo, and G. Xu, “Speech Emotion Recognition Using Mel Spectrogram HPCA and Variational Mode Decomposition,” 2024 11th International Conference on Behavioural and Social Computing (BESC) , pp. 1–7, Aug. 2024. [20] A. A. Namey, K. Akter, M. Azad Hossain, and M. Ali Akber Dewan, “CochleaSpecNet: An Attention Based Dual Branch Hybrid CNN-GRU Network for Speech Emotion Recognition Using Cochleagram and Spectrogram,” IEEE Access , 2024. [21] “Speech Emotion Recognition Using Deep Learning | IEEE Conference Publication | IEEE Xplore.” Accessed: Jan. 04, 2025. [Online]. Available: https://ieeexplore.ieee.org/document/10774956 [22] S. Khan, B. Almas, N. Tariq, F. Ul Haq, A. Faisal, and P. Kumar, “Emotion Recognition of Human Speech Using Different Optimizer Techniques,” 2024 International Conference on Emerging Trends in Networks and Computer Communications (ETNCC) , pp. 1–7, Jul. 2024. Crossref Google Scholar Information & Authors Information Version history V1 Version 1 25 February 2025 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords affective computing conformer-cnn hybrid deep learning human-computer interaction speech emotion recognition Authors Affiliations Manikanta Kadamba [email protected] Gandhi Institute of Technology and Management View all articles by this author Padmaja Madugula 0000-0002-9188-2410 Gandhi Institute of Technology and Management View all articles by this author Metrics & Citations Metrics Article Usage 341 views 135 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Manikanta Kadamba, Padmaja Madugula. HSERNet: Hybrid Speech Emotion Recognition using Conformer-Convolutional Neural Networks. Authorea . 25 February 2025. DOI: https://doi.org/10.22541/au.174048503.32222055/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.174048503.32222055/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'a0008959f8abf047',t:'MTc3OTUwMjM1NQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();