HSMTL-Net: A Multi-Task Learning Framework for Human Segmentation with Integrated Regression and Classification Modules

HSMTL-Net: A Multi-Task Learning Framework for Human Segmentation with Integrated Regression and Classification Modules | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article HSMTL-Net: A Multi-Task Learning Framework for Human Segmentation with Integrated Regression and Classification Modules Faezeh Rohani, Soroush Babaee Khobdeh This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5919455/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Human segmentation is a critical task in computer vision, with many applications in different fields. In this paper, we propose a novel multi-task learning approach, called HSMTL-Net, for human segmentation that incorporates classification and regression tasks along with segmentation. The proposed model is inspired with the U-Net architecture but with less layers and same depth features, which consists of three modules: Seg-Module, Gender-Module, and Seg-Tuner-Module. The Seg-Module performs the segmentation task, while the Gender-Module performs the gender classification task, and the Seg-Tuner-Module performs the tasks that relates to age and other parameters estimation. The proposed approach achieved an exceptional performance for human segmentation, and in addition to that, it includes gender classification and age estimation. Multi-task learning enables the model to learn multiple related tasks simultaneously using shared features and parameters, thereby improving the generalization and efficiency of the model. As a result, the proposed model achieved a Dice score of 97.83% and a mean IoU of 96.21%, which outperformed all previous approaches. Human Segmentation Multi-task learning Gender classification Age estimation Supervise.ly Segmentation Person Dataset Figures Figure 1 Figure 2 Figure 3 1. Introduction In recent years, machine learning has focused on improving the performance of computer vision applications and its related tasks. As a result, many approaches have been proposed for image classification, segmentation, and registration in various fields such as facial recognition [ 1 ], pattern detection [ 2 ], transportation autonomous driving [ 3 ], healthcare, and medical image processing [ 4 , 5 ]. Nowadays, segmentation is considered as one of the time-consuming and challenging components of computer vision; due to this fact, machine learning-based approaches are being used to increase its accuracy and reduce the computational effort needed. Human detection and segmentation is one of the important and active fields of computer vision, which has applications in many cases such as individuals detection in various images [ 6 ]،, counting people, driving assistance systems [ 7 ], monitoring employees in industries [ 8 ], analysing people's behaviour [ 9 ], and more. Another important application of segmentation that has recently received a lot of attention is sport analysis. In this field, there are various tasks that can be performed, such as tracking players during a game [ 10 ], detecting sports equipment (such as hockey sticks[ 11 ] or the ball in football match videos [ 12 ]), estimating human pose in sports [ 13 ], and detecting different actions during sporting events [ 14 ]. The rest of this paper is structured as follows: Section 2 presents the related works, Section 3 describes the methodology and proposed approach, Section 4 presents the results, and Section 5 concludes the paper. 2. Literature Review Despite the importance of human segmentation in different fields and its various applications, only a limited number of research studies have focused on providing computer vision-based solutions. Therefore, in this study, in addition to human segmentation, some papers that briefly explain different machine learning models proposed for object segmentation, specifically human segmentation, are discussed. Firstly, there is a brief mention of traditional machine learning-based methods, and then the focus is on explaining deep learning-based approaches. Furthermore, some explanations about multi-task learning approaches and their advantages are also presented. Various methods have been used to solve the segmentation task, such as thresholding, region growing, k-means clustering, graph cuts, and histogram-based clustering [ 15 ]. These methods were common a decade ago and mainly focused on low-level features. Additionally, Zhao et al.[ 16 ] used the Markov Chain Monte Carlo method to estimate the probability of the presence of individual humans in crowded situations. With the emergence of deep learning and the introduction of various deep networks, noticeable improvements have been achieved in both performance and accuracy for segmentation tasks [ 17 , 18 ]. These approaches are hierarchical feature extractors that are trained on large-scale datasets [ 19 ]. Some models proposed for segmentation and object detection work based on pixel classification to generate pixel-wise labels. These models provide comprehensive information about image pixels such as the desired object class, location, and shape [ 20 ]. Numerous models inspired by the FCN network have been proposed for human segmentation [ 21 , 22 ]. To improve the FCN performance, various approaches have been introduced, such as ENet (efficient neural network) [ 23 ], which is specifically proposed for tasks that require low-latency operations. After FCN, networks such as U-Net[ 24 ] were introduced for segmentation, which performs segmentation with higher accuracy by using skip connections between corresponding encoder and decoder layers and preventing feature loss. For this reason, models inspired by U-Net are often used in medical applications [ 25 ]. Deep Lab[ 26 ] is another model that uses + to increase the field-of-view in the feature extraction stage. In [ 27 ], a CNN network was used specifically for person segmentation to sharpen the foreground and blur the background. In [ 28 ], the Boundary-Aware Network (BANet) was proposed for human segmentation, which is a lightweight network that selectively extracts regional information from images and trains the network with an emphasis on the target region boundaries. In another article [ 29 ], the HVIS (Human Video Instance Segmentation) network was introduced, which performs human instance segmentation using an inner center sampling mechanism that effectively reduces the problem of positive sample ambiguity. All of the mentioned works so far are based on single-task learning approaches, while some articles have proposed models based on multi-task learning that have provided high performance for human segmentation. In general, multi-task learning is a type of machine learning in which multiple tasks are trained simultaneously. In this learning, due to the learning of common features related to multiple tasks, the ability to generalize is increased, and the network's performance in various tasks is improved [ 30 ]. Moreover, multi-task learning implicitly increases the amount of data, which has shown significant performance in various applications according to [ 31 ], such as medical image processing, object detection, face detection, pedestrian detection, and NLP applications [ 32 ]. Some previous works have attempted to improve the accuracy and performance of segmentation in various applications by employing multi-task learning [ 33 , 34 , 35 , 36 ]. Liang et al.[ 35 ] presented a new model called Look into Person (LIP), which works with higher accuracy in human-centric analyses. This approach implements human parsing and pose jointly and can ultimately estimate parsing and pose simultaneously. Jug et al.[ 34 ] also trained a network called SPD (Segmentation-Pose-DensePose) in a framework with multiple tasks of skeleton estimation, dense pose prediction, and segmentation to improve body person segmentation. In this paper, a new framework abbreviated as HSMTL-Net has been proposed for human segmentation, based on multi-task learning. This approach not only learns human segmentation but also other tasks, which can improve the performance of segmentation and allow for other tasks to be executed simultaneously. Since this article focuses on human segmentation, if other information can be extracted from images along with segmentation maps, more optimal uses can be achieved. Therefore, the proposed model has attempted to achieve this goal. In summary, the innovations of this article can be summarized as follows: Introducing a novel method for extracting information from images simultaneously for human segmentation and other related tasks. Based on the research conducted, the architecture of the proposed approach is the first multi-task learning approach that not only performs segmentation but also includes classification and regression tasks. By employing this approach, not only can the problem of data scarcity, which is a significant issue in the field of deep learning networks, be overcome, but also three different tasks can be processed simultaneously. The achieved accuracy for segmentation on the Supervise.ly Person Dataset is better compared to state-of-the-art methods. 3. Proposed Method In this paper, HSMTL-Net network is proposed for human segmentation, which is a multi-task deep learning network that not only performs the segmentation task, but also predicts the gender and other features of the detected human. As it shown in Fig. 1 , base of the HSMTL-Net network is inspired by the U-Net structure and by utilizing multi-task learning architectures and training classification and regression tasks, in addition to providing a novel network, significant results have been achieved. Subjective and objective results show that using multi-task learning not only leads to a considerable improvement in segmentation results, but also predicts the gender and other learned features of the person accurately. Furthermore, this method reduces the training load for each task and improves the quality and speed of model training. One of the most important challenges in using deep neural networks is the size of the dataset on which the model is trained. More data leads to better network performance, but large datasets are not always available for every application. Moreover, providing and labelling data is a difficult and time-consuming task. Therefore, providing an approach that can train the network with a not-so-large dataset and achieve good accuracy has always been of interest. In general, multi-task learning implicitly increases the data and compensates for the need for large datasets. According to the Implicit Data Reinforcement Principle in multi-task learning, when a deep network is trained simultaneously for different tasks, the network tries to learn domain-independent patterns and use them to solve the assigned tasks. In this procedure, any possible change in data or errors that cause the data to be unbalanced or non-uniform are called noises. If tasks A and B are assigned to the network, there is some noise for each one that the network tries to ignore during training and learn domain-independent patterns. If only one task is assigned to the network, these patterns are learned only for that task. However, with multi-task learning, the average of domain-independent patterns is learned for different tasks. In other words, the network learns a more effective set of features, which leads to implicit reinforcement of the data [ 31 ]. In the proposed approach, unlike most deep learning-based approaches that train two tasks together, three tasks are learned together. In order to implement a multi-task network, labels related to different tasks must be available or obtainable for each of the tasks. As it observed in Fig. 1 , the proposed HSMTL-Net network consists of three separate modules, including the Seg-Module for human area detection, Seg-Tuner-Module for predicting human segmentation properties and also age estimation, and Gender-Module for gender detection. In the following, first, explanations are provided for the pre-processing stage, which involves extracting different labels from the image, and then separately for the proposed model modules: 3.1. Pre-processing In the pre-processing stage, the images are initially resized to 256x256 pixels, and ground truth is provided in files with the .json format. In this stage, the images are also converted from this format to .png and resized accordingly. In order to train the proposed multi-task network, labels corresponding to each task are required, so that by defining the corresponding loss function, the network can be trained to make predictions by minimizing the loss. The dataset collectors have provided the labels (ground truth) for the segmentation task. Regarding the other labels for classification and regression tasks, they were extracted using the information related to the images and their ground truth. Some of these labels were provided by technicians, while others were prepared using image processing techniques. In order to find those labels that can be subjectively recorded by technicians’ view, a survey was conducted with the help of three individuals. The images were shown to them, and their opinions were recorded. In the next step, the optimal label was selected using the voting operator for gender and mean operator for age determination. The labels for gender and age, which are related to the classification task and one of the parameters in regression tasks, were obtained using this method. Other parameters obtained for regression include area, perimeter, and centroid in the x and y directions, which were extracted using image processing techniques. Figure 2 shows the process of determining the labels. As an example, Fig. 2 illustrates the challenge of estimating the age of a human in an image, where different individuals yield varying predictions. However, by utilizing a mean operator, the predictions can be consolidated to arrive at a more accurate estimate. This process is repeated for all images in different sets, including the train and test sets, with consistent actions and decisions taken to label each image accordingly. 3.2. Seg-Module: As it shown in Fig. 1 , the module that is related to the segmentation task, is inspired by the U-Net network. The U-Net network consists of two parts: an encoder and a decoder. The encoder part consists of blocks that include convolution and pooling layers and are used to extract features. In other words, the input image is fed into these layers, and its feature maps are extracted. By passing through the convolution layers, the depth of the features increases, and by passing through the pooling layers, the dimensions of these feature maps decrease. The decoder part also includes Up-Convolution layer, which is responsible for increasing the dimensions of the extracted features. Finally, by applying a convolution with a soft-max activation function, the output map, which includes the segmented region, is obtained. Moreover, as shown in the Fig. 1 , there are skip connections between the corresponding layers of the encoder and decoder, which reduce information loss and preserve the extracted features better. Additionally, in the proposed approach, in order to reduce the number of learnable parameters and make optimal use of the extracted features, the number of convolution layers in encoder part has decreased and depth of the network has been the same of U-Net network. For this reason, it is called a modified U-Net. 3.3. Seg-Tuner-Module As the name of this module suggests, its goal is to make the segmentation area more precise. Human segmentation has its own specific challenges. For example, the hair regions should be segmented with higher precision than other parts of the body. In this regard, by incorporating a regression task and training parameters such as age and geometric features extracted from the ground truth region, better segmentation can be achieved. Additionally, since the network is trained based on the multi-task approach, it learns features under the framework of a regression task that enhances the accuracy of the segmentation. Moreover, by learning parameters such as age, the usefulness of the proposed approach can be increased, and in addition to segmentation, the age of human in the segmented region can be accurately identified. To implement this module, we utilized bottleneck features and merged them together with a feature fusion layer to provide robust features for the regression task. Following this stage, the features were inputted into three dense layers with dimensions of 512, 256, and 5. 3.4. Gender-Module As can be seen in Fig. 1 , another module has been provided in the proposed approach that determines the gender of the segmented human. This module was actually designed to improve the segmentation task. However, by receiving feature maps of bottleneck that obtained from the encoder layers and considering the classification task and its related loss function, it not only improved the network's training but also accurately performs gender classification. To implement this task, the extracted features from the encoder layers are fed into several convolution layer with different kernel size and after that by average pooling layer, and classification is performed through this process. 3.5. Loss Functions In order to train the HSMTL-Net network, the Eq. (1) has been used, which is written as: \(\:{L}_{T}={\alpha\:}_{1}{L}_{SegM}+{{\alpha\:}_{2}L}_{SegTuner}+{{\alpha\:}_{3}L}_{GM}\) (1) Each of the terms in the equation represents a cost function that is learned for different tasks in the network. Initially, an explanation is provided about the segmentation task's loss function, which is used in this approach. This loss function itself consists of several parts, which are described below. $$\:{L}_{SegM}=w(E+Dice\:)$$ 2 Eq. ( 2 ) defines the segmentation loss function, that here is called: \(\:{L}_{SegM}\) . For HSMTL-Net it is a combination of the cross-entropy loss (E) and the Dice similarity coefficient, with a weight map (w) used to magnify the Dice metric on the boundaries of the human. The weight map assigns higher weight to the boundary regions, which are typically more difficult to segment. $$\:E=-\sum\:_{x,y\in\:{Z}^{2}}(\text{l}\text{o}\text{g}{p}_{c}(x,y\left)\right)$$ 3 $$\:Dice=\:1-\:\frac{2*(G\cap\:S)}{\left|G\right|+\left|S\right|}$$ 4 In these equations, G is ground truth; S is the network segmentation mask and \(\:(x,y)\) are coordinates of the considered pixel. \(\:w\) , which is the weighting map in (2), was introduced in [ 36 ] to enhance the training process and improve performance of segmentation. Its magnification of loss on the boundaries during the training stage is done by emphasizing on edges, which is defined as: $$\:w\left(x\right)=1+{\omega\:}_{0}.\text{exp}\frac{d\left(x\right)}{2{\sigma\:}^{2}}$$ 5 Here, (𝑥) is distance between the pixel 𝑥 and ground truth boundaries, \(\:\sigma\:\) represents the variance of Gaussian kernel and \(\:{\omega\:}_{0}\) is a predefined constant. In Eq. 1, \(\:{{\alpha\:}_{2}L}_{SegTuner}\:\) refers to the cost function of the Seg-Tuner-Module. Since the module works based on a regression task, the most commonly used regression cost function, mean squared error (MSE), and is used in this task, which is written as: $$\:{L}_{ET}=\frac{1}{n}\sum\:_{i}^{n}{\left({P}_{Pred}\right(i)-{P}_{gt}(i\left)\right)}^{2}$$ 6 This equation deals with the parameters related to the segmented region of the human. \(\:{P}_{Pred}\) represents the predicted parameters vector by the network, and \(\:{P}_{gt}\:\) is the parameters vector from the ground truth. The parameters used for this purpose are the central coordinates ( \(\:{c}_{x},\:{c}_{y}\) ), perimeter, area, size of the largest diameter of the ground truth region, and the angle between the diameter and the x-axis, along with the person's age. In order to train the classification task ( \(\:{L}_{GM}\) ), since the number of classes is two and the images of both classes are available in equal numbers (data is balanced) the binary cross entropy function has been used. As seen in Eq. 1, the loss function used for the proposed approach is the weighted average of three loss functions related to segmentation, regression, and classification tasks. The values of these loss functions are shown in the Table 1 . These values are hyper parameters and are selected based on the implementation and results analysis. However, generally in multi-task learning, the task with higher priority will have a higher weight. Additionally, since errors are back propagated through the network based on the loss functions, propagating the loss function from different parts of the network can reduce the vanishing gradient, which in turn can lead to better network training and improved performance. Table 1 The numerical values of the parameters in the loss functions of HSMTL-Net \(\:{\varvec{\alpha\:}}_{1}\) \(\:{\varvec{\alpha\:}}_{2}\) \(\:{\varvec{\alpha\:}}_{3}\) \(\:{\:\varvec{w}}_{0}\) \(\:\varvec{\sigma\:}\) 3 2 1 30 15 4. Discussion and results This section provides partial explanations about the proposed model, including details about the dataset used for evaluating the model and some implementation details. Evaluation metrics are presented, and in the results section, subjective and objective results are reported. Finally, the proposed model's results are compared with other state-of-the-art methods that have addressed human segmentation. The proposed model was implemented using Python 3.7 and TensorFlow and trained and tested on an NVIDIA GPU GeForce RTX 2080 Ti. The model was trained using stochastic gradient descent with momentum (learning rate = 0.001) for 200 epochs, and the training time was about 39 hours. 4.1 Dataset The "supervise.ly-filtered-segmentation-person-dataset"[ 37 ] dataset contains images with different resolutions and quality. Each image has a complete ground truth that is available as a mask for each image and shows which parts of the image should belong to a person or people. This dataset can be used for training and evaluating algorithms for person detection, predicting people's movements, and analysing videos. This dataset includes images of people in various topics, including: Street images containing people in cities and villages Indoor images containing people inside buildings and rooms Outdoor space images containing people in parks, airports, and squares Images of athletes performing various sports activities, including football, basketball, volleyball, tennis, etc. Images of people in different situations such as accidents, emergencies, and demonstrations This dataset is available as one of the public datasets for human detection in images and contains a total of 5711 images, of which 5110 images are randomly selected as the training set, and the rest are considered as the test set. Additionally, to reduce the risk of overfitting and improve the proposed model's learning, 20% of the training set is randomly selected as the validation set. 4.2 Metrics This paper, to quantitatively evaluate the proposed HSMTL-Net model, a separate evaluation metric has been considered for each task. Intersection-over-Union (IoU) and Dice score have been calculated for the segmentation task. Accuracy (Acc) has been used for the classification task, and Mean Square Error (MSE) has been used to evaluate the accuracy and error for the regression task. IoU, also known as Jaccard index, is a standard evaluation metric for segmentation tasks. The equation to calculate IoU is as follows: $$\:IoU=\frac{TP}{FP+TP+FN}$$ 7 here \(\:TP\) represents true positives; \(\:FP\) is false positives, and \(\:FN\) false negatives. When IoU is calculated for each class, the mean IoU can be calculated for the classification procedure, as shown in Eq. 7 . \(\:Mean\:IoU=\left(\frac{1}{{n}_{cl}}\right)\sum\:_{i=1}^{{n}_{cl}}\frac{{n}_{ii}}{{t}_{i}+{\sum\:}_{j}{n}_{ji}+{n}_{ii}}\) i ≠ j (8) In this equation, \(\:{n}_{ij}\) is the number of pixels of class i predicted to belong to class \(\:\text{j}\) , and \(\:{n}_{ii}\) represents the number of correctly classified pixels for class \(\:\text{i}\) , (true positives). In fact, \(\:{n}_{ij}\) is the number of pixels that has been wrongly classified (false positives). Similarly, \(\:{n}_{ji}\) is the number of pixels that has been wrongly not classified, or false negatives. In addition, \(\:{n}_{cl}\) represents the total number of classes and \(\:{t}_{i}\:\) is the total number of pixels of class i. For the segmentation task, the Dice score parameter is also a popular metric for measuring the similarity between two images. This parameter is calculated based on the value between the pixels of the two images. This parameter is calculated using Eq. 4 . In image classification tasks, the accuracy parameter is calculated as the ratio of the number of images that have been correctly classified to the total number of images in the dataset. In other words, accuracy indicates the percentage of images that have been correctly identified. It is calculated as follows: $$\:Accuracy\:=\frac{TP+TN}{TP+TN+FP+FN}$$ 9 MSE (Mean Squared Error) is also an evaluation metric for regression models that measures the performance of the model by using the squared errors between the model's predictions and the actual values of the labels. In other words, MSE is the average of the squares of the prediction errors of the model. To calculate MSE, we first need to calculate the prediction errors of the model, which are equal to the difference between the model's prediction and the actual label values. Then, to calculate MSE, we need to divide the sum of the squares of these errors by the number of data samples used for evaluation, which is calculated as follows: $$\:Mean\:IoU=\left(\frac{1}{n}\right)\sum\:({{Y}_{i}-{Y}_{i}^{\prime })}^{2}$$ 10 In this equation, n is the number of data samples, y i is the actual label value for the i th sample, and y` i is the model's prediction for the i th sample. 4.3. Objective and subjective results The quantitative results of the HSMTL-Net model are analysed in several stages. In the first stage, the results of human segmentation are reported as a single-task, indicating that the model was trained with only one task and the model used is based on the U-net model but with less depth. In the next stage, the accuracy of segmentation is evaluated using the multi-task learning approach, which is improved by adding other tasks such as classification and regression. Additionally, the accuracy of gender classification and regression is also investigated. Finally, the performance of the proposed model is compared with other approaches that have been proposed for human segmentation. As shown in Table 2 , the performance of the HSMTL-Net model is evaluated at different stages. In the single-task learning mode, where segmentation is the only task learned by the network, the Dice Score and mean IoU are 90.31% and 88.76%, respectively. By adding the regression task, not only the accuracy of the model significantly improved, but also the challenging and intricate regions in the segmentation results were better segmented, as shown in the subjective results section. The third row of Table 2shows that if the classification task is added to the single-task network that performs human segmentation, an improvement can be observed. Although this improvement is not as significant as the improvement obtained by adding the regression task, it is observable that by adding the Gender-Module to the Seg-Module and using multi-task learning, not only the segmentation task accuracy improved but also a high accuracy of 92.83% is reported for gender classification. This indicates that the features learned during training for segmentation and classification are effective features. As seen in the fourth row of Table 2 , by adding the Seg-Tuner-Module as the regression task, the segmentation accuracy for Dice Score and mean IoU reached 95.03% and 93.97%, respectively. Additionally, the MSE, which indicates the level of prediction error of the extracted features from the ground truth and age, is 0.0184, indicating that the features extracted simultaneously for segmentation and regression were effective and contributed to high performance in both tasks. In the next stage (the fifth row of Table 2 ), by adding both regression and classification tasks and training them simultaneously as the HSMTL-Net model, the effect of these tasks on the segmentation task is investigated, and the impact of multi-task learning on the classification and regression tasks is also observed. As seen in the table, by learning these tasks simultaneously, the segmentation task achieved 97.83% and 96.21% for the Dice score and mean IoU metrics, respectively. Additionally, the HSMTL-Net model had better accuracy for both the regression and classification tasks, reaching an accuracy of 97.96% for gender classification, and the prediction error of the regression task based on the MSE metric was 0.00895, indicating an improvement in the performance of this task. Table 2 The quantitative results of HSMTL-Net MSE Accuracy Mean IoU Dice Score Tasks Models - - 88.76 90.31 Segmentation Seg-Module - 92.83 89.69 91.89 Segmentation and classification Seg-Module and Gender-Module 0.0184 - 93.97 95.03 Segmentation and regression Seg-Module and Seg- Tuner Module 0.00995 97.96 96.21 97.83 Segmentation, classification and regression HSMTL-Net After investigating the various stages of improving the objective results of HSMTL-Net, the impact of multi-task learning on subjective results was also examined. Samples were selected to highlight the impact of all stages, and Fig. 3shows four examples of these images from the test set and reports the performance of the proposed approach. As can be seen, the first column shows the original image and the last column shows the ground truth image. To further explore the impact of adding classification and regression tasks, these images were fed to the proposed model and the results were displayed. As can be seen, the second column shows the result of the modified U-Net network, which is essentially the Seg-Module. In the third column, the network's performance for adding the Gender-Module as a classification task is displayed. In the next column, the addition of the Seg-Tuner-Module as a regression task is shown. As can be seen, the addition of each of these tasks has had a positive effect and has improved the segmentation. The last column shows the result obtained from the proposed HSMTL-Net, which has not only segmented the overall shape of the human but has also done a good job of segmenting finer areas such as hair strands, fingers, and other details in the images. In addition to the segmentation output, the network's other outputs for gender and age detection (one of the parameters of the regression task) are also presented in this example. In this case, the genders of the humans have been correctly identified, and their ages have been estimated as 15, 69, 13, and 26 from top to bottom, respectively, which is not far-fetched. Overall, this example demonstrates that not only is the performance of a multi-task network better than a single-task network, but considering other tasks can also improve the performance of the proposed model and obtain other outputs from it. As mentioned in the pre-processing section, all images were resized to 256x256 pixels, and after entering the network, the outputs were resized back to their original size to make improvements in segmentation more tangible. In this section, the performance of the proposed method is compared with state-of-the-art methods that have been evaluated on the supervise.ly-filtered-segmentation-person-dataset [ 37 ]. In [ 38 ], this dataset is used for research on virtual backgrounds in video conferences, and the reported accuracy is the result of online learning. Although this work focuses on processing images online, its performance in the segmentation task can be compared with the method proposed here. The segmentation results of this research are reported in Table 3 , in the second row. Chen et al. [ 39 ] proposed a network architecture called Boundary-Aware Network (BANet) for portrait segmentation that can achieve both high precision, their result is shown in the Table 3 . It is true that data augmentation is a common technique used in some approaches to increase the number of data samples and improve the learning process for human segmentation, as mentioned in [ 40 , 39 ]. However, in this work, we did not use data augmentation, and instead, we achieved notable results by using the multi-task learning approach without increasing the number of data samples in segmentation, classification, and regression tasks. This indicates the effectiveness of the proposed HSMTL-Net model, which is capable of learning from multiple tasks simultaneously and achieving high performance without relying on data augmentation. The results presented in the paper demonstrate that the proposed approach not only has a significant performance for different tasks but also has outperformed previous methods evaluated on this dataset in the segmentation task, providing a higher accuracy. Table 3 Comparison of the proposed approach with previous state-of-the-art approaches. Models Dice Score % Mean IoU % Chuang et al. [ 38 ] - 89.50 Marin et al. [ 41 ] 90.0 Jiang et al. [ 40 ] - 92.2 Chen et al. [ 39 ] 95.2 Proposed model : HSMTL-Net 97.83 96.21 5. Conclusion In this paper, we presented HSMTL-Net, a novel approach based on multi-task learning for human segmentation in images. The results demonstrated that the proposed model effectively segmented human objects in the images with high accuracy and was capable of accurately segmenting fine details, such as hair strands and fingers. It also estimates the gender and age of humans in the images with high precision. The proposed model achieved state-of-the-art performance compared to other methods that have addressed human segmentation on the "supervise.ly-filtered-segmentation-person-dataset". This approach extracts relevant features by training different tasks simultaneously, which not only increases generalization but also reduces the need for data augmentation and achieves high accuracy without using data augmentation. Here, human segmentation was first performed using a modified U-Net, and then the performance of the base network was evaluated by adding various tasks such as classification and regression. Finally, a novel network is proposed that improves not only the segmentation but also the accuracy of classification and regression. This research demonstrates that the use of multi-task learning in deep learning, which always suffers from a lack of data, has a significant impact and can lead to high accuracy and performance. In future works, the authors plan to further investigate this approach on other datasets, especially people segmentation and object segmentation datasets, to increase the accuracy of challenging segmentations in various applications by using and expanding it. Declarations Author Contribution A. wrote the main manuscript text and implementation B. prepared figures and tables and implementationAll authors reviewed the manuscript References Saito S, Li T, Li H (2016) Real-time facial segmentation and performance capture from rgb input, Computer Vision–ECCV : 14th European Conference, 2016 Rohani F et al (2024) Extracting gait and balance pattern features from skeleton data to diagnose attention deficit/hyperactivity disorder in children. J Supercomputing 80(6):8330–8356 Kaymak Çağrı, Ayşegül, Uçar (2019) A brief survey and an application of seman tic image segmentation for autonomous driving, Handbook of deep learning applications , pp. 161–200 Rohani F et al (2022) Exploring effective features in ADHD diagnosis among children through EEG/evoked potentials using machine learning techniques. Comput Knowl Eng 5(2):1–10 Rohani F, Far MAM, Fatemeh Fazayeli Bavojdan (2015) From Business Process Management to Flexible Image Analysis Applications: A Case Study. Comput Biology Bioinf 3(3):40–44 Imran Ahmed M, Ahmad et al (2019) Person detector for diferent overhead views using machine learning. Int J Mach Learn Cybernet 10:2657–2668 Geronimo D et al (2009) Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans Pattern Anal Mach Intell 32(7):1239–1258 Ahmed I, Ahmad A, Piccialli F et al (2017) A robust features-based person tracker for overhead views in industrial environment. IEEE Internet Things J 28(3):598–605 Cristani M, Raghavendra R, Del Bue A, Murino V (2013) Human behavior analysis in video surveillance: A social signal processing perspective, Neurocomputing , vol. 100, pp. 86–97 Mohib Ullah FA, Cheikh (2018) A directed sparse graphical model for multi-target tracking, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1816–1823 Cai Z, Neher H, Vats K, Clausi DA, Zelek J (2019) Temporal Hockey Action Recognition via Pose and Optical Flows, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , Kamble PR, Keskar AG, Bhurchandi KM (2019) A deep learning ball tracking system in soccer videos. Opto-Electron Rev 27(1):58–69 Lewis Bridgeman M, Volino et al (2019) Multi-person 3d pose estimation and tracking in sports Shyamal Buch V, Escorcia C, Shen et al (2017) SST: Single-Stream Temporal Action Proposals, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2911–2920 Zaitoun NM, Musbah J, Aqel (2015) Survey on image segmentation techniques. Procedia Comput Sci 65:797–806 Tao Zhao R, Nevatia (2003) Bayesian human segmentation in crowded situations, IEEE Computer Society Conference on Computer Vision and Pattern Recognition , vol. 2 Bharath Hariharan P, Arbelaez R, Girshick J, Malik (2015) Hypercolumns for Object Segmentation and Fine-Grained Localization, the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 447–456 Zahra Sobhaninia N et al (2022) Karimi,., Medial Residual Encoder Layers for Classification of Brain Tumors in Magnetic Resonance Images, in 30th International Conference on Electrical Engineering (ICEE) , Tehran, Iran, Islamic Republic of Olga Russakovsky J, Deng H, Su et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis volume 115:pages211–252 Jonathan L, Shelhamer E (2015) Trevor Darrell;, Fully convolutional networks for semantic segmentation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 3431–3440 Song C, Huang Y et al (2015) 1000fps human segmentation with deep convolutional neural networks, 3rd IAPR Asian Conference on Pattern Recognition (ACPR) , pp. 474–478 Imran A, Misbah, Ahmad et al (2020) Comparison of Deep-Learning-Based Segmentation Models: Using Top View Person Images. IEEE Access, 8 Paszke A, Chaurasia A et al (2016) Enet: A deep neural network architecture for real-time semantic segmentation, arXiv:1606.02147 , Olaf Ronneberger P, Fischer, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. Med Image Comput Computer-Assisted Intervention–MICCAI Zahra S, Ali, Emami et al (2020) Localization of Fetal Head in Ultrasound Images by Multiscale View and Deep Neural Networks, in 25th International Computer Conference, Computer Society of Iran (CSICC) , Tehran, Iran Chen L-C, Papandreou G et al (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848 Xiaoyong Shen A, Hertzmann et al (2016) Automatic Portrait Segmentation for Image Stylization. Comput Graphics Forum, 35, 2 Chen X, Qi D, Shen J Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation, arXiv preprint arXiv:1901.03814. , 2019 Jan 12 Yu R, Tian C et al (2022) Real-time human-centric segmentation for complex video scenes. Image Vis Comput, p. 104552 Caruana R (1998) Multitask Learning. Autonomous Agents and Multi-Agent Systems Ruder S An Overview of Multi-Task Learning in deep neural networks, 2017. [Online]. Available: arXiv preprint arXiv:1706.05098. Zhang Y,Qiang, Yang (2021) A Survey on Multi-Task Learning, IEEE Transactions on Knowledge and Data Engineering , vol. 34, no. 12, pp. 5586–5609 Zahra Sobhaninia N (2023) Brain tumor segmentation by cascaded multiscale multitask learning framework based on feature aggregation. Biomed Signal Process Control 85:104834 Julijan Jug A, Lampe et al (2022) Body Segmentation Using Multi-task Learning, in International Conference on Artificial Intelligence in Information and Communication (ICAIIC) Liang X, Gong K, Shen X, Lin L (2018) Look into person: Joint body parsing & pose estimation network and a new benchmark. IEEE Trans Pattern Anal Mach Intell 41(4):871–885 Zahra Sobhaninia S, Rafiei et al (2019) Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning, in 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC) , Berlin, Germany Supervisely person dataset Jo Chuang Q, Dong JIT-M (2020) Efficient Online Distillation for Background Matting, Chen X, Qi D, Shen J (2019) Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation Ziyu, Jiang et al (2021) CE-PeopleSeg: Real-time people segmentation with 10% CPU usage for video conference, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 914–922 Dmitrii Marin Z, He et al (2019) Efficient Segmentation: Learning Downsampling Near Semantic Boundaries, in Proceedings of the IEEE/CVF International Conference on Computer Vision Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5919455","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":409208371,"identity":"1dbe334e-bc44-4517-b70d-b1555d63f79b","order_by":0,"name":"Faezeh Rohani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9UlEQVRIiWNgGAWjYBAC9gYgwQgiJIA4oQJIMDM34NXCcwBFyxmQFkZStDC2McC4eLSwHz784eMOhnx+6eZnDx7Oq43mbwdq+VGxDbcWnrQEw5lnGCxnzjlmbpC47XjujMOMDYw9Z27j1GLPkGOQzNvGYGBwI8FMInHbsdwGoBZmxjbcWnj43384/BesJf2bROKcY7nzCWqRyGFsZgRryQHa0lCTu4GwlmfGjL1tEgaSM3LKJBKOHcjdCNRyEJ9fePiTH3/42WZjwC+Rvk3yR01d7rzzhw8++FGBWwsUSMAYh8HkAULqkUEdKYpHwSgYBaNghAAA5Y9Z/7bY290AAAAASUVORK5CYII=","orcid":"","institution":"Sanabad Golbahar Institute of Higher Education","correspondingAuthor":true,"prefix":"","firstName":"Faezeh","middleName":"","lastName":"Rohani","suffix":""},{"id":409208372,"identity":"2db8b50b-ae36-4ee8-8ab4-be4060d65640","order_by":1,"name":"Soroush Babaee Khobdeh","email":"","orcid":"","institution":"Postbank Iran","correspondingAuthor":false,"prefix":"","firstName":"Soroush","middleName":"Babaee","lastName":"Khobdeh","suffix":""}],"badges":[],"createdAt":"2025-01-28 15:23:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5919455/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5919455/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75307932,"identity":"0c95bc20-b55a-4a75-a19f-48b4fa45276c","added_by":"auto","created_at":"2025-02-03 08:41:16","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":568282,"visible":true,"origin":"","legend":"\u003cp\u003eAn overview of the proposed model called HSMTL-Net network, which consists of three modules, Seg-Module, Seg-Tuner-Module, and Gender-Module, shows that three tasks are trained simultaneously, and it has three outputs.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5919455/v1/66e12aa62ed07181c903455b.png"},{"id":75307933,"identity":"cd252e02-b73a-4002-8d76-a862a91f6db9","added_by":"auto","created_at":"2025-02-03 08:41:16","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":136749,"visible":true,"origin":"","legend":"\u003cp\u003eAn overview of the process of determining gender and age labels.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-5919455/v1/2358ddda1da26bfc0003f64b.png"},{"id":75308590,"identity":"24a02e45-9d41-4aff-8161-86820db44c8d","added_by":"auto","created_at":"2025-02-03 08:49:16","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":967142,"visible":true,"origin":"","legend":"\u003cp\u003eThe subjective results related to the HSMTL-Net are presented step by step. (a):input image, \u0026nbsp;(b): represents the results of the modified U-Net (c): the results of an multi-task approach that contains Seg-Module and Gender-Module (d): the results of an multi-task approach that contains Seg-Module and Seg-Tuner-Module (e): the results of approach: HSMTL-Net (f): Ground truth of input images\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-5919455/v1/3d061155c11bae4f8693c2cc.png"},{"id":78611874,"identity":"780e0cdc-0294-4512-8928-f1bc7fe0b7da","added_by":"auto","created_at":"2025-03-16 14:31:34","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2205436,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5919455/v1/35c2856d-e0bc-418e-865b-da5e7d1bf771.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"HSMTL-Net: A Multi-Task Learning Framework for Human Segmentation with Integrated Regression and Classification Modules","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eIn recent years, machine learning has focused on improving the performance of computer vision applications and its related tasks. As a result, many approaches have been proposed for image classification, segmentation, and registration in various fields such as facial recognition [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], pattern detection [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], transportation autonomous driving [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], healthcare, and medical image processing [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Nowadays, segmentation is considered as one of the time-consuming and challenging components of computer vision; due to this fact, machine learning-based approaches are being used to increase its accuracy and reduce the computational effort needed. Human detection and segmentation is one of the important and active fields of computer vision, which has applications in many cases such as individuals detection in various images [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]،, counting people, driving assistance systems [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], monitoring employees in industries [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], analysing people's behaviour [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], and more. Another important application of segmentation that has recently received a lot of attention is sport analysis. In this field, there are various tasks that can be performed, such as tracking players during a game [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], detecting sports equipment (such as hockey sticks[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] or the ball in football match videos [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]), estimating human pose in sports [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], and detecting different actions during sporting events [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe rest of this paper is structured as follows: Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents the related works, Section \u003cspan refid=\"Sec3\" class=\"InternalRef\"\u003e3\u003c/span\u003e describes the methodology and proposed approach, Section \u003cspan refid=\"Sec9\" class=\"InternalRef\"\u003e4\u003c/span\u003e presents the results, and Section \u003cspan refid=\"Sec13\" class=\"InternalRef\"\u003e5\u003c/span\u003e concludes the paper.\u003c/p\u003e"},{"header":"2. Literature Review","content":"\u003cp\u003eDespite the importance of human segmentation in different fields and its various applications, only a limited number of research studies have focused on providing computer vision-based solutions. Therefore, in this study, in addition to human segmentation, some papers that briefly explain different machine learning models proposed for object segmentation, specifically human segmentation, are discussed. Firstly, there is a brief mention of traditional machine learning-based methods, and then the focus is on explaining deep learning-based approaches. Furthermore, some explanations about multi-task learning approaches and their advantages are also presented.\u003c/p\u003e \u003cp\u003eVarious methods have been used to solve the segmentation task, such as thresholding, region growing, k-means clustering, graph cuts, and histogram-based clustering [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. These methods were common a decade ago and mainly focused on low-level features. Additionally, Zhao et al.[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] used the Markov Chain Monte Carlo method to estimate the probability of the presence of individual humans in crowded situations. With the emergence of deep learning and the introduction of various deep networks, noticeable improvements have been achieved in both performance and accuracy for segmentation tasks [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. These approaches are hierarchical feature extractors that are trained on large-scale datasets [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Some models proposed for segmentation and object detection work based on pixel classification to generate pixel-wise labels. These models provide comprehensive information about image pixels such as the desired object class, location, and shape [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Numerous models inspired by the FCN network have been proposed for human segmentation [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. To improve the FCN performance, various approaches have been introduced, such as ENet (efficient neural network) [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], which is specifically proposed for tasks that require low-latency operations. After FCN, networks such as U-Net[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] were introduced for segmentation, which performs segmentation with higher accuracy by using skip connections between corresponding encoder and decoder layers and preventing feature loss. For this reason, models inspired by U-Net are often used in medical applications [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Deep Lab[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e] is another model that uses\u0026thinsp;+\u0026thinsp;to increase the field-of-view in the feature extraction stage. In [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], a CNN network was used specifically for person segmentation to sharpen the foreground and blur the background. In [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], the Boundary-Aware Network (BANet) was proposed for human segmentation, which is a lightweight network that selectively extracts regional information from images and trains the network with an emphasis on the target region boundaries. In another article [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e], the HVIS (Human Video Instance Segmentation) network was introduced, which performs human instance segmentation using an inner center sampling mechanism that effectively reduces the problem of positive sample ambiguity. All of the mentioned works so far are based on single-task learning approaches, while some articles have proposed models based on multi-task learning that have provided high performance for human segmentation.\u003c/p\u003e \u003cp\u003eIn general, multi-task learning is a type of machine learning in which multiple tasks are trained simultaneously. In this learning, due to the learning of common features related to multiple tasks, the ability to generalize is increased, and the network's performance in various tasks is improved [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. Moreover, multi-task learning implicitly increases the amount of data, which has shown significant performance in various applications according to [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], such as medical image processing, object detection, face detection, pedestrian detection, and NLP applications [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. Some previous works have attempted to improve the accuracy and performance of segmentation in various applications by employing multi-task learning [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. Liang et al.[\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e] presented a new model called Look into Person (LIP), which works with higher accuracy in human-centric analyses. This approach implements human parsing and pose jointly and can ultimately estimate parsing and pose simultaneously. Jug et al.[\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e] also trained a network called SPD (Segmentation-Pose-DensePose) in a framework with multiple tasks of skeleton estimation, dense pose prediction, and segmentation to improve body person segmentation.\u003c/p\u003e \u003cp\u003eIn this paper, a new framework abbreviated as HSMTL-Net has been proposed for human segmentation, based on multi-task learning. This approach not only learns human segmentation but also other tasks, which can improve the performance of segmentation and allow for other tasks to be executed simultaneously. Since this article focuses on human segmentation, if other information can be extracted from images along with segmentation maps, more optimal uses can be achieved. Therefore, the proposed model has attempted to achieve this goal. In summary, the innovations of this article can be summarized as follows:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eIntroducing a novel method for extracting information from images simultaneously for human segmentation and other related tasks.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eBased on the research conducted, the architecture of the proposed approach is the first multi-task learning approach that not only performs segmentation but also includes classification and regression tasks. By employing this approach, not only can the problem of data scarcity, which is a significant issue in the field of deep learning networks, be overcome, but also three different tasks can be processed simultaneously.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe achieved accuracy for segmentation on the Supervise.ly Person Dataset is better compared to state-of-the-art methods.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e"},{"header":"3. Proposed Method","content":"\u003cp\u003eIn this paper, HSMTL-Net network is proposed for human segmentation, which is a multi-task deep learning network that not only performs the segmentation task, but also predicts the gender and other features of the detected human. As it shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, base of the HSMTL-Net network is inspired by the U-Net structure and by utilizing multi-task learning architectures and training classification and regression tasks, in addition to providing a novel network, significant results have been achieved. Subjective and objective results show that using multi-task learning not only leads to a considerable improvement in segmentation results, but also predicts the gender and other learned features of the person accurately. Furthermore, this method reduces the training load for each task and improves the quality and speed of model training.\u003c/p\u003e \u003cp\u003eOne of the most important challenges in using deep neural networks is the size of the dataset on which the model is trained. More data leads to better network performance, but large datasets are not always available for every application. Moreover, providing and labelling data is a difficult and time-consuming task. Therefore, providing an approach that can train the network with a not-so-large dataset and achieve good accuracy has always been of interest. In general, multi-task learning implicitly increases the data and compensates for the need for large datasets. According to the Implicit Data Reinforcement Principle in multi-task learning, when a deep network is trained simultaneously for different tasks, the network tries to learn domain-independent patterns and use them to solve the assigned tasks. In this procedure, any possible change in data or errors that cause the data to be unbalanced or non-uniform are called noises. If tasks A and B are assigned to the network, there is some noise for each one that the network tries to ignore during training and learn domain-independent patterns. If only one task is assigned to the network, these patterns are learned only for that task. However, with multi-task learning, the average of domain-independent patterns is learned for different tasks. In other words, the network learns a more effective set of features, which leads to implicit reinforcement of the data [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. In the proposed approach, unlike most deep learning-based approaches that train two tasks together, three tasks are learned together.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIn order to implement a multi-task network, labels related to different tasks must be available or obtainable for each of the tasks. As it observed in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, the proposed HSMTL-Net network consists of three separate modules, including the Seg-Module for human area detection, Seg-Tuner-Module for predicting human segmentation properties and also age estimation, and Gender-Module for gender detection. In the following, first, explanations are provided for the pre-processing stage, which involves extracting different labels from the image, and then separately for the proposed model modules:\u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1. Pre-processing\u003c/h2\u003e \u003cp\u003eIn the pre-processing stage, the images are initially resized to 256x256 pixels, and ground truth is provided in files with the .json format. In this stage, the images are also converted from this format to .png and resized accordingly.\u003c/p\u003e \u003cp\u003eIn order to train the proposed multi-task network, labels corresponding to each task are required, so that by defining the corresponding loss function, the network can be trained to make predictions by minimizing the loss. The dataset collectors have provided the labels (ground truth) for the segmentation task. Regarding the other labels for classification and regression tasks, they were extracted using the information related to the images and their ground truth. Some of these labels were provided by technicians, while others were prepared using image processing techniques. In order to find those labels that can be subjectively recorded by technicians\u0026rsquo; view, a survey was conducted with the help of three individuals. The images were shown to them, and their opinions were recorded. In the next step, the optimal label was selected using the voting operator for gender and mean operator for age determination. The labels for gender and age, which are related to the classification task and one of the parameters in regression tasks, were obtained using this method. Other parameters obtained for regression include area, perimeter, and centroid in the x and y directions, which were extracted using image processing techniques. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the process of determining the labels. As an example, Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e illustrates the challenge of estimating the age of a human in an image, where different individuals yield varying predictions. However, by utilizing a mean operator, the predictions can be consolidated to arrive at a more accurate estimate. This process is repeated for all images in different sets, including the train and test sets, with consistent actions and decisions taken to label each image accordingly.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Seg-Module:\u003c/h2\u003e \u003cp\u003eAs it shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, the module that is related to the segmentation task, is inspired by the U-Net network. The U-Net network consists of two parts: an encoder and a decoder. The encoder part consists of blocks that include convolution and pooling layers and are used to extract features. In other words, the input image is fed into these layers, and its feature maps are extracted. By passing through the convolution layers, the depth of the features increases, and by passing through the pooling layers, the dimensions of these feature maps decrease. The decoder part also includes Up-Convolution layer, which is responsible for increasing the dimensions of the extracted features. Finally, by applying a convolution with a soft-max activation function, the output map, which includes the segmented region, is obtained. Moreover, as shown in the Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, there are skip connections between the corresponding layers of the encoder and decoder, which reduce information loss and preserve the extracted features better. Additionally, in the proposed approach, in order to reduce the number of learnable parameters and make optimal use of the extracted features, the number of convolution layers in encoder part has decreased and depth of the network has been the same of U-Net network. For this reason, it is called a modified U-Net.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.3. Seg-Tuner-Module\u003c/h2\u003e \u003cp\u003eAs the name of this module suggests, its goal is to make the segmentation area more precise. Human segmentation has its own specific challenges. For example, the hair regions should be segmented with higher precision than other parts of the body. In this regard, by incorporating a regression task and training parameters such as age and geometric features extracted from the ground truth region, better segmentation can be achieved. Additionally, since the network is trained based on the multi-task approach, it learns features under the framework of a regression task that enhances the accuracy of the segmentation. Moreover, by learning parameters such as age, the usefulness of the proposed approach can be increased, and in addition to segmentation, the age of human in the segmented region can be accurately identified. To implement this module, we utilized bottleneck features and merged them together with a feature fusion layer to provide robust features for the regression task. Following this stage, the features were inputted into three dense layers with dimensions of 512, 256, and 5.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.4. Gender-Module\u003c/h2\u003e \u003cp\u003eAs can be seen in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, another module has been provided in the proposed approach that determines the gender of the segmented human. This module was actually designed to improve the segmentation task. However, by receiving feature maps of bottleneck that obtained from the encoder layers and considering the classification task and its related loss function, it not only improved the network's training but also accurately performs gender classification. To implement this task, the extracted features from the encoder layers are fed into several convolution layer with different kernel size and after that by average pooling layer, and classification is performed through this process.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.5. Loss Functions\u003c/h2\u003e \u003cp\u003eIn order to train the HSMTL-Net network, the Eq.\u0026nbsp;(1) has been used, which is written as:\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{L}_{T}={\\alpha\\:}_{1}{L}_{SegM}+{{\\alpha\\:}_{2}L}_{SegTuner}+{{\\alpha\\:}_{3}L}_{GM}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(1)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eEach of the terms in the equation represents a cost function that is learned for different tasks in the network. Initially, an explanation is provided about the segmentation task's loss function, which is used in this approach. This loss function itself consists of several parts, which are described below.\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\:{L}_{SegM}=w(E+Dice\\:)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eEq.\u0026nbsp;(\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e2\u003c/span\u003e) defines the segmentation loss function, that here is called: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{L}_{SegM}\\)\u003c/span\u003e\u003c/span\u003e. For HSMTL-Net it is a combination of the cross-entropy loss (E) and the Dice similarity coefficient, with a weight map (w) used to magnify the Dice metric on the boundaries of the human. The weight map assigns higher weight to the boundary regions, which are typically more difficult to segment.\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$\\:E=-\\sum\\:_{x,y\\in\\:{Z}^{2}}(\\text{l}\\text{o}\\text{g}{p}_{c}(x,y\\left)\\right)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e\n$$\\:Dice=\\:1-\\:\\frac{2*(G\\cap\\:S)}{\\left|G\\right|+\\left|S\\right|}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIn these equations, G is ground truth; S is the network segmentation mask and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:(x,y)\\)\u003c/span\u003e\u003c/span\u003e are coordinates of the considered pixel. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:w\\)\u003c/span\u003e\u003c/span\u003e, which is the weighting map in (2), was introduced in [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e] to enhance the training process and improve performance of segmentation. Its magnification of loss on the boundaries during the training stage is done by emphasizing on edges, which is defined as:\u003cdiv id=\"Equ4\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ4\" name=\"EquationSource\"\u003e\n$$\\:w\\left(x\\right)=1+{\\omega\\:}_{0}.\\text{exp}\\frac{d\\left(x\\right)}{2{\\sigma\\:}^{2}}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e5\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eHere, (\u0026#119909;) is distance between the pixel \u0026#119909; and ground truth boundaries, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\sigma\\:\\)\u003c/span\u003e\u003c/span\u003e represents the variance of Gaussian kernel and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\omega\\:}_{0}\\)\u003c/span\u003e\u003c/span\u003e is a predefined constant.\u003c/p\u003e \u003cp\u003eIn Eq.\u0026nbsp;1, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{{\\alpha\\:}_{2}L}_{SegTuner}\\:\\)\u003c/span\u003e\u003c/span\u003e refers to the cost function of the Seg-Tuner-Module. Since the module works based on a regression task, the most commonly used regression cost function, mean squared error (MSE), and is used in this task, which is written as:\u003cdiv id=\"Equ5\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ5\" name=\"EquationSource\"\u003e\n$$\\:{L}_{ET}=\\frac{1}{n}\\sum\\:_{i}^{n}{\\left({P}_{Pred}\\right(i)-{P}_{gt}(i\\left)\\right)}^{2}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e6\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThis equation deals with the parameters related to the segmented region of the human. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{P}_{Pred}\\)\u003c/span\u003e\u003c/span\u003e represents the predicted parameters vector by the network, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{P}_{gt}\\:\\)\u003c/span\u003e\u003c/span\u003eis the parameters vector from the ground truth. The parameters used for this purpose are the central coordinates (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{c}_{x},\\:{c}_{y}\\)\u003c/span\u003e\u003c/span\u003e), perimeter, area, size of the largest diameter of the ground truth region, and the angle between the diameter and the x-axis, along with the person's age. In order to train the classification task (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{L}_{GM}\\)\u003c/span\u003e\u003c/span\u003e), since the number of classes is two and the images of both classes are available in equal numbers (data is balanced) the binary cross entropy function has been used.\u003c/p\u003e \u003cp\u003eAs seen in Eq.\u0026nbsp;1, the loss function used for the proposed approach is the weighted average of three loss functions related to segmentation, regression, and classification tasks. The values of these loss functions are shown in the Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. These values are hyper parameters and are selected based on the implementation and results analysis. However, generally in multi-task learning, the task with higher priority will have a higher weight. Additionally, since errors are back propagated through the network based on the loss functions, propagating the loss function from different parts of the network can reduce the vanishing gradient, which in turn can lead to better network training and improved performance.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe numerical values of the parameters in the loss functions of HSMTL-Net\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\varvec{\\alpha\\:}}_{1}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\varvec{\\alpha\\:}}_{2}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\varvec{\\alpha\\:}}_{3}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\:\\varvec{w}}_{0}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\varvec{\\sigma\\:}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003e3\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e30\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e15\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. Discussion and results","content":"\u003cp\u003eThis section provides partial explanations about the proposed model, including details about the dataset used for evaluating the model and some implementation details. Evaluation metrics are presented, and in the results section, subjective and objective results are reported. Finally, the proposed model\u0026apos;s results are compared with other state-of-the-art methods that have addressed human segmentation.\u003c/p\u003e\n\u003cp\u003eThe proposed model was implemented using Python 3.7 and TensorFlow and trained and tested on an NVIDIA GPU GeForce RTX 2080 Ti. The model was trained using stochastic gradient descent with momentum (learning rate\u0026thinsp;=\u0026thinsp;0.001) for 200 epochs, and the training time was about 39 hours.\u003c/p\u003e\n\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e\n \u003ch2\u003e4.1 Dataset\u003c/h2\u003e\n \u003cp\u003eThe \u0026quot;supervise.ly-filtered-segmentation-person-dataset\u0026quot;[\u003cspan class=\"CitationRef\"\u003e37\u003c/span\u003e] dataset contains images with different resolutions and quality. Each image has a complete ground truth that is available as a mask for each image and shows which parts of the image should belong to a person or people. This dataset can be used for training and evaluating algorithms for person detection, predicting people\u0026apos;s movements, and analysing videos. This dataset includes images of people in various topics, including:\u003c/p\u003e\n \u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eStreet images containing people in cities and villages\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eIndoor images containing people inside buildings and rooms\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eOutdoor space images containing people in parks, airports, and squares\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eImages of athletes performing various sports activities, including football, basketball, volleyball, tennis, etc.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eImages of people in different situations such as accidents, emergencies, and demonstrations\u003c/p\u003e\n \u003c/li\u003e\n \u003c/ul\u003e\n \u003cp\u003eThis dataset is available as one of the public datasets for human detection in images and contains a total of 5711 images, of which 5110 images are randomly selected as the training set, and the rest are considered as the test set. Additionally, to reduce the risk of overfitting and improve the proposed model\u0026apos;s learning, 20% of the training set is randomly selected as the validation set.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n \u003ch2\u003e4.2 Metrics\u003c/h2\u003e\n \u003cp\u003eThis paper, to quantitatively evaluate the proposed HSMTL-Net model, a separate evaluation metric has been considered for each task. Intersection-over-Union (IoU) and Dice score have been calculated for the segmentation task. Accuracy (Acc) has been used for the classification task, and Mean Square Error (MSE) has been used to evaluate the accuracy and error for the regression task. IoU, also known as Jaccard index, is a standard evaluation metric for segmentation tasks. The equation to calculate IoU is as follows:\u003c/p\u003e\n \u003cdiv id=\"Equ6\" class=\"Equation\"\u003e\n \u003cdiv id=\"FileID_Equ6\" class=\"mathdisplay\"\u003e$$\\:IoU=\\frac{TP}{FP+TP+FN}$$\u003c/div\u003e\n \u003cdiv class=\"EquationNumber\"\u003e7\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003ehere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:TP\\)\u003c/span\u003e\u003c/span\u003e represents true positives; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:FP\\)\u003c/span\u003e\u003c/span\u003e is false positives, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:FN\\)\u003c/span\u003e\u003c/span\u003e false negatives. When IoU is calculated for each class, the mean IoU can be calculated for the classification procedure, as shown in Eq. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\n \u003ctable id=\"Tabb\" border=\"1\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:Mean\\:IoU=\\left(\\frac{1}{{n}_{cl}}\\right)\\sum\\:_{i=1}^{{n}_{cl}}\\frac{{n}_{ii}}{{t}_{i}+{\\sum\\:}_{j}{n}_{ji}+{n}_{ii}}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003ei\u0026thinsp;\u0026ne;\u0026thinsp;j\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e(8)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cp\u003eIn this equation, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{ij}\\)\u003c/span\u003e\u003c/span\u003e is the number of pixels of class i predicted to belong to class \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{j}\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{ii}\\)\u003c/span\u003e\u003c/span\u003e represents the number of correctly classified pixels for class \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{i}\\)\u003c/span\u003e\u003c/span\u003e, (true positives). In fact, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{ij}\\)\u003c/span\u003e\u003c/span\u003e is the number of pixels that has been wrongly classified (false positives). Similarly, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{ji}\\)\u003c/span\u003e\u003c/span\u003e is the number of pixels that has been wrongly not classified, or false negatives. In addition, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{cl}\\)\u003c/span\u003e\u003c/span\u003e represents the total number of classes and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{t}_{i}\\:\\)\u003c/span\u003e\u003c/span\u003eis the total number of pixels of class i.\u003c/p\u003e\n \u003cp\u003eFor the segmentation task, the Dice score parameter is also a popular metric for measuring the similarity between two images. This parameter is calculated based on the value between the pixels of the two images. This parameter is calculated using Eq.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eIn image classification tasks, the accuracy parameter is calculated as the ratio of the number of images that have been correctly classified to the total number of images in the dataset. In other words, accuracy indicates the percentage of images that have been correctly identified. It is calculated as follows:\u003c/p\u003e\n \u003cdiv id=\"Equ7\" class=\"Equation\"\u003e\n \u003cdiv id=\"FileID_Equ7\" class=\"mathdisplay\"\u003e$$\\:Accuracy\\:=\\frac{TP+TN}{TP+TN+FP+FN}$$\u003c/div\u003e\n \u003cdiv class=\"EquationNumber\"\u003e9\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eMSE (Mean Squared Error) is also an evaluation metric for regression models that measures the performance of the model by using the squared errors between the model\u0026apos;s predictions and the actual values of the labels. In other words, MSE is the average of the squares of the prediction errors of the model. To calculate MSE, we first need to calculate the prediction errors of the model, which are equal to the difference between the model\u0026apos;s prediction and the actual label values. Then, to calculate MSE, we need to divide the sum of the squares of these errors by the number of data samples used for evaluation, which is calculated as follows:\u003c/p\u003e\n \u003cdiv id=\"Equ8\" class=\"Equation\"\u003e\n \u003cdiv id=\"FileID_Equ8\" class=\"mathdisplay\"\u003e$$\\:Mean\\:IoU=\\left(\\frac{1}{n}\\right)\\sum\\:({{Y}_{i}-{Y}_{i}^{\\prime })}^{2}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e10\u003c/div\u003e\u003c/div\u003e\u003cp\u003eIn this equation, n is the number of data samples, y\u003csub\u003ei\u003c/sub\u003e is the actual label value for the i\u003csub\u003eth\u003c/sub\u003e sample, and y`\u003csub\u003ei\u003c/sub\u003e is the model\u0026apos;s prediction for the i\u003csub\u003eth\u003c/sub\u003e sample.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\u003ch2\u003e4.3. Objective and subjective results\u003c/h2\u003e\u003cp\u003eThe quantitative results of the HSMTL-Net model are analysed in several stages. In the first stage, the results of human segmentation are reported as a single-task, indicating that the model was trained with only one task and the model used is based on the U-net model but with less depth. In the next stage, the accuracy of segmentation is evaluated using the multi-task learning approach, which is improved by adding other tasks such as classification and regression. Additionally, the accuracy of gender classification and regression is also investigated. Finally, the performance of the proposed model is compared with other approaches that have been proposed for human segmentation.\u003c/p\u003e\u003cp\u003eAs shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, the performance of the HSMTL-Net model is evaluated at different stages. In the single-task learning mode, where segmentation is the only task learned by the network, the Dice Score and mean IoU are 90.31% and 88.76%, respectively. By adding the regression task, not only the accuracy of the model significantly improved, but also the challenging and intricate regions in the segmentation results were better segmented, as shown in the subjective results section.\u003c/p\u003e\u003cp\u003eThe third row of Table\u0026nbsp;2shows that if the classification task is added to the single-task network that performs human segmentation, an improvement can be observed. Although this improvement is not as significant as the improvement obtained by adding the regression task, it is observable that by adding the Gender-Module to the Seg-Module and using multi-task learning, not only the segmentation task accuracy improved but also a high accuracy of 92.83% is reported for gender classification. This indicates that the features learned during training for segmentation and classification are effective features.\u003c/p\u003e\u003cp\u003eAs seen in the fourth row of Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, by adding the Seg-Tuner-Module as the regression task, the segmentation accuracy for Dice Score and mean IoU reached 95.03% and 93.97%, respectively. Additionally, the MSE, which indicates the level of prediction error of the extracted features from the ground truth and age, is 0.0184, indicating that the features extracted simultaneously for segmentation and regression were effective and contributed to high performance in both tasks.\u003c/p\u003e\u003cp\u003eIn the next stage (the fifth row of Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e), by adding both regression and classification tasks and training them simultaneously as the HSMTL-Net model, the effect of these tasks on the segmentation task is investigated, and the impact of multi-task learning on the classification and regression tasks is also observed. As seen in the table, by learning these tasks simultaneously, the segmentation task achieved 97.83% and 96.21% for the Dice score and mean IoU metrics, respectively. Additionally, the HSMTL-Net model had better accuracy for both the regression and classification tasks, reaching an accuracy of 97.96% for gender classification, and the prediction error of the regression task based on the MSE metric was 0.00895, indicating an improvement in the performance of this task.\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable id=\"Tab2\" border=\"1\"\u003e\u003ccaption\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eThe quantitative results of HSMTL-Net\u0026nbsp;\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd style=\"width: 58px;\"\u003e\u003cp\u003eMSE\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 82px;\"\u003e\u003cp\u003eAccuracy\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 74px;\"\u003e\u003cp\u003eMean IoU\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 84px;\"\u003e\u003cp\u003eDice Score\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 174px;\"\u003e\u003cp\u003eTasks\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 156px;\"\u003e\u003cp\u003eModels\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd style=\"width: 58px;\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 82px;\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 74px;\"\u003e\u003cp\u003e88.76\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 84px;\"\u003e\u003cp\u003e90.31\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 174px;\"\u003e\u003cp\u003eSegmentation\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 156px;\"\u003e\u003cp\u003eSeg-Module\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd style=\"width: 58px;\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 82px;\"\u003e\u003cp\u003e92.83\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 74px;\"\u003e\u003cp\u003e89.69\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 84px;\"\u003e\u003cp\u003e91.89\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 174px;\"\u003e\u003cp\u003eSegmentation and classification\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 156px;\"\u003e\u003cp\u003eSeg-Module and Gender-Module\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd style=\"width: 58px;\"\u003e\u003cp\u003e0.0184\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 82px;\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 74px;\"\u003e\u003cp\u003e93.97\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 84px;\"\u003e\u003cp\u003e95.03\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 174px;\"\u003e\u003cp\u003eSegmentation and regression\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 156px;\"\u003e\u003cp\u003eSeg-Module and Seg- Tuner Module\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd style=\"width: 58px;\"\u003e\u003cp\u003e0.00995\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 82px;\"\u003e\u003cp\u003e97.96\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 74px;\"\u003e\u003cp\u003e96.21\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 84px;\"\u003e\u003cp\u003e97.83\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 174px;\"\u003e\u003cp\u003eSegmentation, classification and regression\u003c/p\u003e\u003c/td\u003e\u003ctd style=\"width: 156px;\"\u003e\u003cp\u003e\u0026nbsp;\u003c/p\u003e\u003cp\u003eHSMTL-Net\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003eAfter investigating the various stages of improving the objective results of HSMTL-Net, the impact of multi-task learning on subjective results was also examined. Samples were selected to highlight the impact of all stages, and Fig.\u0026nbsp;3shows four examples of these images from the test set and reports the performance of the proposed approach. As can be seen, the first column shows the original image and the last column shows the ground truth image. To further explore the impact of adding classification and regression tasks, these images were fed to the proposed model and the results were displayed. As can be seen, the second column shows the result of the modified U-Net network, which is essentially the Seg-Module. In the third column, the network\u0026apos;s performance for adding the Gender-Module as a classification task is displayed. In the next column, the addition of the Seg-Tuner-Module as a regression task is shown. As can be seen, the addition of each of these tasks has had a positive effect and has improved the segmentation. The last column shows the result obtained from the proposed HSMTL-Net, which has not only segmented the overall shape of the human but has also done a good job of segmenting finer areas such as hair strands, fingers, and other details in the images. In addition to the segmentation output, the network\u0026apos;s other outputs for gender and age detection (one of the parameters of the regression task) are also presented in this example. In this case, the genders of the humans have been correctly identified, and their ages have been estimated as 15, 69, 13, and 26 from top to bottom, respectively, which is not far-fetched. Overall, this example demonstrates that not only is the performance of a multi-task network better than a single-task network, but considering other tasks can also improve the performance of the proposed model and obtain other outputs from it. As mentioned in the pre-processing section, all images were resized to 256x256 pixels, and after entering the network, the outputs were resized back to their original size to make improvements in segmentation more tangible.\u003c/div\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003eIn this section, the performance of the proposed method is compared with state-of-the-art methods that have been evaluated on the supervise.ly-filtered-segmentation-person-dataset [\u003cspan class=\"CitationRef\"\u003e37\u003c/span\u003e]. In [\u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e], this dataset is used for research on virtual backgrounds in video conferences, and the reported accuracy is the result of online learning. Although this work focuses on processing images online, its performance in the segmentation task can be compared with the method proposed here. The segmentation results of this research are reported in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, in the second row.\u003c/div\u003e\u003c/div\u003e\u003cp\u003eChen et al. [\u003cspan class=\"CitationRef\"\u003e39\u003c/span\u003e] proposed a network architecture called Boundary-Aware Network (BANet) for portrait segmentation that can achieve both high precision, their result is shown in the Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. It is true that data augmentation is a common technique used in some approaches to increase the number of data samples and improve the learning process for human segmentation, as mentioned in [\u003cspan class=\"CitationRef\"\u003e40\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e39\u003c/span\u003e]. However, in this work, we did not use data augmentation, and instead, we achieved notable results by using the multi-task learning approach without increasing the number of data samples in segmentation, classification, and regression tasks. This indicates the effectiveness of the proposed HSMTL-Net model, which is capable of learning from multiple tasks simultaneously and achieving high performance without relying on data augmentation. The results presented in the paper demonstrate that the proposed approach not only has a significant performance for different tasks but also has outperformed previous methods evaluated on this dataset in the segmentation task, providing a higher accuracy.\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv class=\"colspec\" align=\"left\"\u003e\u0026nbsp;\u003c/div\u003e\u003ctable id=\"Tab3\" border=\"1\"\u003e\u003ccaption\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eComparison of the proposed approach with previous state-of-the-art approaches.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\"\u003e\u003cp\u003eModels\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\"\u003e\u003cp\u003eDice Score %\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\"\u003e\u003cp\u003eMean IoU %\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\"\u003e\u003cp\u003eChuang et al. [\u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\"\u003e\u003cp\u003e89.50\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\"\u003e\u003cp\u003eMarin et al. [\u003cspan class=\"CitationRef\"\u003e41\u003c/span\u003e]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"char\"\u003e\u003cp\u003e90.0\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\"\u003e\u003cp\u003eJiang et al. [\u003cspan class=\"CitationRef\"\u003e40\u003c/span\u003e]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\"\u003e\u003cp\u003e92.2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\"\u003e\u003cp\u003eChen et al. [\u003cspan class=\"CitationRef\"\u003e39\u003c/span\u003e]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"char\"\u003e\u003cp\u003e95.2\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\"\u003e\u003cp\u003e\u003cstrong\u003eProposed model : HSMTL-Net\u003c/strong\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\"\u003e\u003cp\u003e\u003cstrong\u003e97.83\u003c/strong\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\"\u003e\u003cp\u003e\u003cstrong\u003e96.21\u003c/strong\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e\u003c/div\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eIn this paper, we presented HSMTL-Net, a novel approach based on multi-task learning for human segmentation in images. The results demonstrated that the proposed model effectively segmented human objects in the images with high accuracy and was capable of accurately segmenting fine details, such as hair strands and fingers. It also estimates the gender and age of humans in the images with high precision. The proposed model achieved state-of-the-art performance compared to other methods that have addressed human segmentation on the \"supervise.ly-filtered-segmentation-person-dataset\". This approach extracts relevant features by training different tasks simultaneously, which not only increases generalization but also reduces the need for data augmentation and achieves high accuracy without using data augmentation. Here, human segmentation was first performed using a modified U-Net, and then the performance of the base network was evaluated by adding various tasks such as classification and regression. Finally, a novel network is proposed that improves not only the segmentation but also the accuracy of classification and regression. This research demonstrates that the use of multi-task learning in deep learning, which always suffers from a lack of data, has a significant impact and can lead to high accuracy and performance. In future works, the authors plan to further investigate this approach on other datasets, especially people segmentation and object segmentation datasets, to increase the accuracy of challenging segmentations in various applications by using and expanding it.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eA. wrote the main manuscript text and implementation B. prepared figures and tables and implementationAll authors reviewed the manuscript\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSaito S, Li T, Li H (2016) Real-time facial segmentation and performance capture from rgb input, \u003cem\u003eComputer Vision\u0026ndash;ECCV\u003c/em\u003e : 14th European Conference, 2016\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRohani F et al (2024) Extracting gait and balance pattern features from skeleton data to diagnose attention deficit/hyperactivity disorder in children. J Supercomputing 80(6):8330\u0026ndash;8356\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaymak \u0026Ccedil;ağrı, Ayşeg\u0026uuml;l, U\u0026ccedil;ar (2019) A brief survey and an application of seman tic image segmentation for autonomous driving, \u003cem\u003eHandbook of deep learning applications\u003c/em\u003e, pp. 161\u0026ndash;200\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRohani F et al (2022) Exploring effective features in ADHD diagnosis among children through EEG/evoked potentials using machine learning techniques. Comput Knowl Eng 5(2):1\u0026ndash;10\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRohani F, Far MAM, Fatemeh Fazayeli Bavojdan (2015) From Business Process Management to Flexible Image Analysis Applications: A Case Study. Comput Biology Bioinf 3(3):40\u0026ndash;44\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eImran Ahmed M, Ahmad et al (2019) Person detector for diferent overhead views using machine learning. Int J Mach Learn Cybernet 10:2657\u0026ndash;2668\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGeronimo D et al (2009) Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans Pattern Anal Mach Intell 32(7):1239\u0026ndash;1258\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAhmed I, Ahmad A, Piccialli F et al (2017) A robust features-based person tracker for overhead views in industrial environment. IEEE Internet Things J 28(3):598\u0026ndash;605\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCristani M, Raghavendra R, Del Bue A, Murino V (2013) Human behavior analysis in video surveillance: A social signal processing perspective, \u003cem\u003eNeurocomputing\u003c/em\u003e, vol. 100, pp. 86\u0026ndash;97\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMohib Ullah FA, Cheikh (2018) A directed sparse graphical model for multi-target tracking, \u003cem\u003eIEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e, pp. 1816\u0026ndash;1823\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCai Z, Neher H, Vats K, Clausi DA, Zelek J (2019) Temporal Hockey Action Recognition via Pose and Optical Flows, \u003cem\u003eIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops\u003c/em\u003e,\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKamble PR, Keskar AG, Bhurchandi KM (2019) A deep learning ball tracking system in soccer videos. Opto-Electron Rev 27(1):58\u0026ndash;69\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLewis Bridgeman M, Volino et al (2019) Multi-person 3d pose estimation and tracking in sports\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShyamal Buch V, Escorcia C, Shen et al (2017) SST: Single-Stream Temporal Action Proposals, \u003cem\u003eProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e, pp. 2911\u0026ndash;2920\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZaitoun NM, Musbah J, Aqel (2015) Survey on image segmentation techniques. Procedia Comput Sci 65:797\u0026ndash;806\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTao Zhao R, Nevatia (2003) Bayesian human segmentation in crowded situations, \u003cem\u003eIEEE Computer Society Conference on Computer Vision and Pattern Recognition\u003c/em\u003e, vol. 2\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBharath Hariharan P, Arbelaez R, Girshick J, Malik (2015) Hypercolumns for Object Segmentation and Fine-Grained Localization, \u003cem\u003ethe IEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e, pp. 447\u0026ndash;456\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZahra Sobhaninia N et al (2022) Karimi,., Medial Residual Encoder Layers for Classification of Brain Tumors in Magnetic Resonance Images, in \u003cem\u003e30th International Conference on Electrical Engineering (ICEE)\u003c/em\u003e, Tehran, Iran, Islamic Republic of\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOlga Russakovsky J, Deng H, Su et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis volume 115:pages211\u0026ndash;252\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJonathan L, Shelhamer E (2015) Trevor Darrell;, Fully convolutional networks for semantic segmentation, \u003cem\u003eIEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e, pp. 3431\u0026ndash;3440\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSong C, Huang Y et al (2015) 1000fps human segmentation with deep convolutional neural networks, \u003cem\u003e3rd IAPR Asian Conference on Pattern Recognition (ACPR)\u003c/em\u003e, pp. 474\u0026ndash;478\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eImran A, Misbah, Ahmad et al (2020) Comparison of Deep-Learning-Based Segmentation Models: Using Top View Person Images. IEEE Access, 8\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePaszke A, Chaurasia A et al (2016) Enet: A deep neural network architecture for real-time semantic segmentation, \u003cem\u003earXiv:1606.02147\u003c/em\u003e,\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOlaf Ronneberger P, Fischer, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. Med Image Comput Computer-Assisted Intervention\u0026ndash;MICCAI\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZahra S, Ali, Emami et al (2020) Localization of Fetal Head in Ultrasound Images by Multiscale View and Deep Neural Networks, in \u003cem\u003e25th International Computer Conference, Computer Society of Iran (CSICC)\u003c/em\u003e, Tehran, Iran\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen L-C, Papandreou G et al (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834\u0026ndash;848\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXiaoyong Shen A, Hertzmann et al (2016) Automatic Portrait Segmentation for Image Stylization. Comput Graphics Forum, 35, 2\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen X, Qi D, Shen J Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation, \u003cem\u003earXiv preprint arXiv:1901.03814.\u003c/em\u003e, 2019 Jan 12\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu R, Tian C et al (2022) Real-time human-centric segmentation for complex video scenes. Image Vis Comput, p. 104552\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCaruana R (1998) Multitask Learning. Autonomous Agents and Multi-Agent Systems\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRuder S An Overview of Multi-Task Learning in deep neural networks, 2017. [Online]. Available: arXiv preprint arXiv:1706.05098.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang Y,Qiang, Yang (2021) A Survey on Multi-Task Learning, \u003cem\u003eIEEE Transactions on Knowledge and Data Engineering\u003c/em\u003e, vol. 34, no. 12, pp. 5586\u0026ndash;5609\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZahra Sobhaninia N (2023) Brain tumor segmentation by cascaded multiscale multitask learning framework based on feature aggregation. Biomed Signal Process Control 85:104834\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJulijan Jug A, Lampe et al (2022) Body Segmentation Using Multi-task Learning, in \u003cem\u003eInternational Conference on Artificial Intelligence in Information and Communication (ICAIIC)\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiang X, Gong K, Shen X, Lin L (2018) Look into person: Joint body parsing \u0026amp; pose estimation network and a new benchmark. IEEE Trans Pattern Anal Mach Intell 41(4):871\u0026ndash;885\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZahra Sobhaninia S, Rafiei et al (2019) Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning, in \u003cem\u003e41st annual international conference of the IEEE engineering in medicine and biology society (EMBC)\u003c/em\u003e, Berlin, Germany\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSupervisely person dataset\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJo Chuang Q, Dong JIT-M (2020) Efficient Online Distillation for Background Matting,\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen X, Qi D, Shen J (2019) Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZiyu, Jiang et al (2021) CE-PeopleSeg: Real-time people segmentation with 10% CPU usage for video conference, \u003cem\u003eProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition\u003c/em\u003e, pp. 914\u0026ndash;922\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDmitrii Marin Z, He et al (2019) Efficient Segmentation: Learning Downsampling Near Semantic Boundaries, in \u003cem\u003eProceedings of the IEEE/CVF International Conference on Computer Vision\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Human Segmentation, Multi-task learning, Gender classification, Age estimation, Supervise.ly Segmentation Person Dataset","lastPublishedDoi":"10.21203/rs.3.rs-5919455/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5919455/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eHuman segmentation is a critical task in computer vision, with many applications in different fields. In this paper, we propose a novel multi-task learning approach, called HSMTL-Net, for human segmentation that incorporates classification and regression tasks along with segmentation. The proposed model is inspired with the U-Net architecture but with less layers and same depth features, which consists of three modules: Seg-Module, Gender-Module, and Seg-Tuner-Module. The Seg-Module performs the segmentation task, while the Gender-Module performs the gender classification task, and the Seg-Tuner-Module performs the tasks that relates to age and other parameters estimation. The proposed approach achieved an exceptional performance for human segmentation, and in addition to that, it includes gender classification and age estimation. Multi-task learning enables the model to learn multiple related tasks simultaneously using shared features and parameters, thereby improving the generalization and efficiency of the model. As a result, the proposed model achieved a Dice score of 97.83% and a mean IoU of 96.21%, which outperformed all previous approaches.\u003c/p\u003e","manuscriptTitle":"HSMTL-Net: A Multi-Task Learning Framework for Human Segmentation with Integrated Regression and Classification Modules","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-02-03 08:41:11","doi":"10.21203/rs.3.rs-5919455/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"8b92abad-270f-4530-935b-88d65c315dc7","owner":[],"postedDate":"February 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-03-16T14:23:26+00:00","versionOfRecord":[],"versionCreatedAt":"2025-02-03 08:41:11","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5919455","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5919455","identity":"rs-5919455","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}