A UNET++ and CoGAN-based method to remove face masks from the masked faces

A UNET++ and CoGAN-based method to remove face masks from the masked faces | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A UNET++ and CoGAN-based method to remove face masks from the masked faces Akhil Kumar, Divyam Gupta, Manisha Kaushal, Akashdeep Sharma This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3351025/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Image-to-image translation emerges as a significant utility of conditional Generative Adversarial Networks (CoGANs). This research introduces a fresh application of conditional GANs, aiming to uncover hidden facial attributes. Our methodology involves enhancing the Pix2Pix GAN framework through the integration of a modified UNET + + architecture, which serves as an inventive generator model. In this setup, the Pix2Pix model employs a PatchGAN architecture within the discriminator, producing an activation map with values utilized to authenticate depicted faces. Through the incorporation of the UNET + + architecture into the generator, we effectively narrow down the semantic gap between the encoder and decoder feature maps. This strategic adjustment results in a noticeable enhancement in gradient flow. To gauge the effectiveness of our proposed approach, we conducted experiments on a bespoke dataset intentionally crafted for training paired image-to-image translation GANs. Our model is comprehensively compared against other leading models designed for revealing concealed facial features. Significantly, our proposed model convincingly surpasses these alternatives across a range of evaluation criteria. CoGAN Pix2Pix UNET++ PatchGAN image to image translation Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 1. Introduction Generative Adversarial Networks (GANs) [ 1 ] constitute a deep learning approach to generative modeling, often utilizing convolutional neural networks. The GAN framework encompasses a pair of neural networks, specifically the Generator and the Discriminator, engaged in a competitive dynamic to create novel images. The Generator produces batches of samples, juxtaposed with genuine domain instances, and feeds them to the Discriminator for classification into real or synthetic categories. This adversarial interplay reflects a zero-sum game within game theory. The prowess of Generative Adversarial Networks (GANs) in producing remarkably authentic and finely detailed images has fueled their exceptional performance in image generation. Among the array of successful image translation applications for GANs, such as CoGAN [ 2 ], cycleGAN [ 3 ], stackGAN [ 4 ], and more, this study centers on harnessing the Pix2Pix [ 5 ] network for its research objectives. However, a constraint of the GAN model lies in its potential to generate arbitrary domain images. The conditional GAN, or CoGAN, counteracts this by enabling targeted image generation of specific categories. This involves introducing an additional input layer in the form of one-hot-encoded image labels, guiding the generator during training to synthesize images of a designated type. In the context of face mask removal using GANs, the process entails two steps: (1) identifying the mask, generally by delineating the mask region's boundary, and (2) generating the concealed region obscured by the mask. These steps collectively contribute to the final synthesized image. The present study puts forth a variant of the Pix2Pix GAN to facilitate the generation of unmasked facial images from obscured and blurred inputs, employing a pair-to-pair image translation paradigm. Furthermore, we introduce a meticulously curated dataset that establishes a direct correspondence between input and output images, embodying the transformation from masked to unmasked representations. The principal contributions of this paper encompass: 1) We introduce a novel GAN architecture designed to unveil concealed facial features. In pursuit of effective image-to-image translation, we present an innovative adaptation of the CoGAN-based Pix2Pix GAN framework. Our approach involves the integration of the UNET + + architecture into the generator model, alongside a customized array of convolutional networks for the discriminator models. The outcomes of our proposed model are noteworthy, with SSIM values of 0.91115 and 0.85752, as well as PSNR values of 31.7251 and 24.9883, observed on the training and test sets, respectively. These results are indicative of its ability to uphold both structural fidelity and signal clarity amidst noise when reconstructing the facial region obscured by a mask. 2) Furthermore, we introduce a dataset comprising depictions of individuals donning masks juxtaposed with corresponding images of the same individuals without masks. This comprehensive dataset encompasses more than 30,000 facial images, capturing diverse instances of individuals wearing three distinct types of masks. 3) Comparative analyses reveal the notably superior performance and heightened suitability of the proposed GAN within real-world contexts. When contrasted with analogous approaches, our GAN model achieved a significant enhancement, showcasing an improvement ranging from 1.2–5.9% in SSIM scores and a further advancement of 2.82–6.35% in PSNR metric scores. This work encompasses the subsequent sections: In Section 2 , an exploration of the pertinent literature within the realms of image editing and face mask removal is presented. Section 3 delves into the materials and methodologies employed, elucidating the formulation of the dataset and the development of the model. The experimental assessments, outcomes, and limitations of this endeavor are expounded upon in Section 4 . Finally, Section 5 encapsulates the conclusion drawn from this study and outlines prospects for future advancements within the domain. 2. Related work GANs have established their presence across diverse domains, encompassing tasks such as object manipulation and generation, image and sound synthesis, and high-resolution image creation. This section provides an overview of GAN advancements with a specific focus on image generation, particularly concerning concealed portions of images and facial structures. Yu et al. [ 6 ] introduced an image editing technique centered on GANs, employing a contextual attention module in conjunction with a coarse-to-fine GAN-based strategy for image inpainting. Their work was validated through the utilization of Places2, CelebA, and ImageNet [ 7 ] datasets. Nazeri et al. [ 8 ] proposed a two-stage adversarial network for image editing, comprising an edge generator and an image editing module. The CelebA, Places2, and Paris StreetView [ 9 ] datasets served as the basis for evaluation. Khan et al. [ 10 ] utilized a coarse-to-fine Generative Adversarial Network to remove microphones from facial images, leveraging synthesized images from the CelebA dataset. Isola et al. [ 5 ] introduced a straightforward framework for image generation, featuring a "U-Net" based generator architecture and a convolutional "PatchGAN" classifier discriminator. Their results highlighted the potential of conditional adversarial networks for image-to-image translation tasks. A stream of GAN-based works has surfaced in recent times, specifically addressing face inpainting and reconstruction with and without masks. Ud Din et al. [ 11 ] employed a U-NET [ 12 ] architecture for image binarization, utilizing a single generator and two discriminators to generate mask-free face images from masked counterparts within the CelebA dataset. Jiang et al. [ 13 ] proposed a dual GAN approach for the reconstruction of concealed facial areas, building synthetic datasets with and without masks. Farahanipad et al. [ 14 ] introduced masked face unmasking using GANs, leveraging the "MaskTheFace" tool [ 15 ] for dataset creation and the YOLO algorithm [ 16 ] for region detection, ultimately employing a CycleGAN [ 17 ] model for unmasking. Subsequently, others followed similar approaches [ 18 ] [ 19 ] using the Pix2Pix GAN, generating datasets and employing binary segmentation in their modified models. Kumar et al. [ 21 ] introduced a spatial attention module within the C-GAN framework for unmasking masked faces, yielding high-accuracy results. Examining related research on object removal, image editing, and face mask removal and reconstruction underscores the scarcity of techniques proficient in identifying face masks, removing them, and accurately reconstructing the unmasked image. Addressing these gaps, our work endeavours to devise a method capable of adeptly removing face masks, facilitating accurate synthesis of unmasked facial images. 3. Materials and methods 3.1 Dataset The proposed approach employs a technique of paired image-to-image translation, necessitating a specialized dataset that ensures a one-to-one correspondence between input and output images. As such, the creation of a tailored dataset featuring these two distinct image categories was imperative. The input images depict individuals with masks covering their faces, while the target images portray the same individuals without masks, representing their unobstructed facial appearance. Given the limited availability of suitable publicly accessible datasets, our work took the initiative to curate a customized dataset utilizing images of renowned Bollywood celebrities, conveniently accessible on Kaggle [ 20 ]. This dataset encompasses a comprehensive collection of 12,355 images showcasing the top 100 Bollywood personalities. To enhance the dataset's realism, we also incorporated sample images from the dataset used by authors in [ 21 ], thus augmenting the dataset's diversity and mitigating the risk of neural networks converging to local minima. Figure 1 offers a glimpse of the proposed dataset's structure. The dataset preparation involved cropping facial regions and eliminating redundant images, resulting in a final count of 11,262 images without face masks. To introduce variety, the dataset featured three distinct types of face masks superimposed on the images: white face masks, Anti-COVID face masks, and 3M face masks. Consequently, the modified dataset encompassed 11,262 images of unmasked faces, as well as equal numbers of images for each type of mask. It's important to note that the method employed for mask removal from facial regions draws upon both the images of unmasked faces and faces wearing masks. Incorporating OpenCV's DNN module [ 24 ], we proficiently render face masks onto facial regions. This module aids in extracting facial landmarks, generating a comprehensive set of 68 landmarks across the face, encompassing regions like eyes, eyebrows, nose, mouth, and jawline. These landmarks subsequently serve as reference points for precisely overlaying face masks as needed. To ensure optimal mask alignment, we thoughtfully annotated mask points corresponding to key landmarks along the jawline's base and the top of the nose. The alignment process employs the OpenCV library to determine transformations between matched key points, mask annotations, and corresponding facial landmarks. Subsequently, the identified transformation matrix is applied to map these points accurately. To bolster the proposed model's proficiency in predicting face mask regions, we undertook the training of a tiny YOLO v4-SPP face mask detector [ 22 ]. The face mask detector utilized in this study was employed to identify areas with face masks in a publicly available face mask detection dataset [ 23 ]. This dataset encompasses a substantial compilation of 52,635 images featuring individuals both with and without face masks. The regions identified by the tiny YOLO v4-SPP face mask detector were extracted, resulting in cropped images of the detected face mask areas within the images that depicted mask-wearing. These identified face mask regions, denoted by black rectangular boxes, were subsequently overlaid onto the modified Bollywood Celebrity dataset. By strategically integrating these highlighted mask regions into the altered images, we facilitated the precise identification of various types and complexities of face masks by the proposed model. This streamlined the process of removing masks from obscured faces. The comprehensive process of mask area detection via the tiny YOLO v4-SPP face mask detector is elucidated in Fig. 2 , while the intricate steps of dataset creation are meticulously elaborated upon in Fig. 3 . 3.2 Approach In this section, we provide an in-depth insight into the proposed GAN architecture, which is founded upon a modified UNET + + encoder-decoder design for the purpose of unveiling concealed facial features. The operational intricacies of this architecture are expounded upon. Figure 4 visually represents the comprehensive architecture adopted within this study. The underlying model takes an input image and proceeds to identify the presence of a facial mask. Subsequently, it undertakes the task of reconstructing the obscured facial region that lies behind the detected mask. The subsequent sections delve into the architecture's specifics and mechanics, unraveling its inner workings in detail. a) Discriminator module The discriminator's role involves processing both a source image (a facial image with a mask) and a target image (a facial image without a mask). It then assesses the likelihood that the target image is either a genuine or a synthesized transformation of the source image. The architecture of the discriminator is intricately tied to the effective receptive field of the model, which determines the ratio between the model's output and the number of input image pixels. For this endeavor, the PatchGAN architecture was chosen for the discriminator. This design meticulously aligns each output prediction of the model to a 70x70 square patch within the input image. This strategy offers the advantage of adaptable applicability to input images of varying sizes, whether larger or smaller than 256x256 pixels. The model's output can take the form of a singular value or a square activation map, featuring values that assess the authenticity of each patch in the input image—whether it's real or fake. These values can be averaged to provide an overall probability or classification score if desired. The PatchGAN's structure is influenced by the effective receptive field size, often referred to as the receptive field. This term denotes the relationship between a model's output activation and a region within the input image. Notably, the receptive field isn't equivalent to the discriminator model's output size. Instead, it pertains to a model that outputs a pixel in the activation map corresponding to a pixel in the input image. Traditionally, the receptive field is defined based on the size of an individual convolutional layer's activation map relative to its input, filter size, and stride value. The 70x70 PatchGAN adheres to a fixed number of three layers (excluding the output and second-to-last layers), regardless of the input image's dimensions. The formula for calculating the one-dimensional receptive field is: receptive field = (output_size − 1) * stride + kernel_size , where output_size represents the previous layer's activation map size, stride signifies the filter's application shift, and kernel_size indicates the filter's dimensions. Furthermore, the discriminator module processes two concatenated input images and generates a prediction matrix, as depicted in Fig. 5 . It consists of a series of layers including Convolution, Batch Normalization, and LeakyReLU. The proposed discriminator module encompasses six Convolution layers, with a kernel size of 4 and a stride of 2 for the initial four layers. The training of the discriminator employs the Adam optimizer with a learning rate of 0.0001, a beta_1 hyperparameter of 0.5, and the Poisson loss function. The model comprises a total of 6,968,257 parameters (2,816 trainable and 6,965,441 non-trainable parameters). The model's weights are initialized with a normal distribution characterized by a standard deviation of 0.02. b) Generator module The generator module within the proposed CoGAN-based Pix2Pix architecture adopts the UNET architecture, characterized by its distinctive "U" shape. This symmetrical structure comprises two integral parts: the contracting path, situated on the left, and the expansive path forming the right side of the U. The contracting path employs convolution processes, while the expansive path employs 2D transpose convolution layers. This encoder-decoder model employs skip connections, linking equivalent layers between the encoder and decoder blocks that share matching feature map sizes. In this work, we incorporate a modification of the UNET architecture known as UNET++. UNET + + refines the traditional skip connections present in UNET by addressing the semantic gap between encoder and decoder feature maps before concatenation. This architecture utilizes nested and dense skip connections, effectively capturing intricate details of 2D images. The UNET + + structure comprises encoder and decoder blocks interconnected through a sequence of nested dense convolution blocks, as depicted in Fig. 6 . It differs from the original UNET architecture in several key ways: 1) It integrates convolution layers within skip pathways, narrowing the semantic gap between encoder and decoder feature maps. 2) It introduces dense skip connections on skip pathways to enhance gradient flow. 3) It incorporates deep supervision, enabling model pruning and performance enhancement. The UNET + + architecture enhances segmentation accuracy by incorporating Dense blocks and Convolution layers between the encoder and decoder. It achieves this by redesigning the skip pathways to bridge the semantic gap between encoder and decoder sub-paths. These convolution layers aim to minimize the semantic gap between the feature maps of the encoder and decoder sub-networks, simplifying optimization for the optimizer. Dense skip connections, inspired by DenseNet, are implemented within UNET + + as skip pathways between the encoder and decoder. These Dense blocks accumulate previous feature maps and deliver them to the current node through the dense convolution block along each skip pathway. This generates high-resolution feature maps at multiple semantic levels. Capitalizing on UNET++'s advantages, we propose a modified version of the original Pix2Pix GAN, featuring a generator module built on the UNET + + architecture. The generator module comprises an encoder sub-network or backbone followed by a decoder sub-network. Each encoder block is composed of Convolution layers, BatchNormalization layers, and ReLU activation layers. The encoder network consists of five such blocks, all with a kernel size and stride of 2 and identical padding. The decoder block's upsampling layers employ Convolution transpose layers with a kernel size and stride of 2 and similar padding. The final output layer employs a convolution layer with a kernel size of 1 and a Tanh activation function. This choice of activation ensures pixel values in the generated image lie within the range of [-1, 1]. The generator module is updated using a weighted combination of Poisson and Mean Squared Error losses, with the weighting skewed toward Mean Squared Error. The Adam Optimizer is employed to minimize the losses, featuring a learning rate of 0.0001 and beta_1 of 0.5. To propose a method for unmasking masked faces, we integrate the previously described discriminator and generator modules, creating the UNET + + and CoGAN-based Pix2Pix GAN model. This combined GAN model operates with a substantial 23,023,172 parameters, out of which 16,053,603 are trainable, and 6,969,569 are non-trainable. c) Loss function The GAN model we propose employs a combination of Poisson and Mean Squared Error losses to address issues of vanishing gradients and convergence. The specifics of these two loss functions are outlined as follows. 1) Poisson loss In the context of our proposed GAN model, the Poisson loss function serves as a tool for regression, particularly suited for modeling count data. This loss function gauges the dissimilarity between the projected outcome and the observed actual output. Mathematically, the Poisson loss can be expressed through Eq. ( 1 ). $$L\left(y, \widehat{y}\right)= \frac{1}{N} \sum _{i=0}^{N}({\widehat{y}}_{i}-{y}_{i}log{\widehat{y}}_{i})$$ 1 The act of minimizing the Poisson loss is synonymous with maximizing the likelihood of the data, assuming that the target follows a Poisson distribution given the input. This loss is applicable when there's a belief that the target value adheres to a Poisson distribution and the aim is to model the rate parameter based on a particular input. 2) Mean Squared Error The Mean Squared Error (MSE) serves as a widely used loss function in regression scenarios. This loss function quantifies the average of squared disparities between predicted and actual values across the provided data. MSE is particularly responsive to outliers; when confronted with numerous instances featuring identical input features, the optimal prediction is their mean target value. This stands in contrast to the Mean Absolute Error, where the median becomes the optimal prediction. MSE is especially suitable when the target data, contingent on the input, adheres to a normal distribution centered around a mean value. Furthermore, it's beneficial when the intention is to penalize outliers effectively. The mathematical representation of the Mean Squared Error (MSE) loss is captured by Eq. ( 2 ). $$L\left(y, \widehat{y}\right)= \frac{1}{N} \sum _{i=0}^{N}{(y-{\widehat{y}}_{i})}^{2}$$ 2 In Eq. ( 1 ) for Poisson loss and (2) for Mean Squared Error (MSE) loss, \(y\) represents the actual output whereas, \(\widehat{y}\) is the predicted expected value for \(i\) data count and \(N\) computations. The previously outlined generator and discriminator, coupled with the described loss functions, exhibited a favorable loss curve during training. This curve reflects successful training on the training set, contributing to the accomplishment of the image-to-image translation objective. Graphical representations of the generator loss, discriminator loss, and the comprehensive GAN model loss are depicted in Fig. 7. 4. Experiments and Results In this section, we delve into the results garnered from a series of experiments conducted to validate the efficacy of the proposed approach. The UNET + + and CoGAN-based Pix2Pix model, designed for unmasking masked faces, was implemented on a system featuring the following configuration: Intel® Core i7-6850K CPU @ 3.60 GHz x 12, coupled with 64 GB of RAM, and two NVIDIA TITAN XP 12 GB GPUs. The dataset was partitioned into training and testing subsets, maintaining a 55:45 ratio. The training subset, constituting 55% of the images, was employed for generating realistic images of unmasked faces using the generator module. To facilitate the discrimination between generator module outputs and real images of unmasked faces, the discriminator module was presented with outputs from the generator module along with a collection of 17,603 genuine unmasked face images. For evaluation purposes, a distinct set of 14,000 images from the testing subset, exclusive of the training set's masked face images, was utilized. The training phase of the proposed approach spanned 300 iterations, employing a batch size of 4. The training process was facilitated by the Adam optimizer, leveraging a learning rate of 0.0001 and a beta_1 value of 0.5. In the generator module of the proposed GAN model, the Mean Squared Error (MSE) loss and Poisson loss were utilized, while the discriminator module relied solely on the Poisson loss. Further elaboration includes insight into the evaluation metrics, the sequence of experiments, the resultant evaluation outcomes, and a comparative analysis against other relevant methodologies. These aspects are meticulously presented in the subsequent subsections. 4.1 Evaluation metrics The central emphasis of this investigation centers on image-to-image translation, making the evaluation of structural similarity between images and their signal-to-noise ratio of utmost importance. To quantify the structural similarity, the SSIM (Structural Similarity Index) metric has been employed. Concurrently, the PSNR (Peak Signal-to-Noise Ratio) metric has been adopted for evaluating signal-to-noise ratio. The SSIM metric, denoting the Structural Similarity Index, is a robust tool for gauging the quality of digital images and videos. Its purpose lies in assessing the likeness between two images, utilizing criteria such as luminance intensity, contrast, and structural characteristics. This metric encapsulates these factors to provide an estimate of similarity between the images. Mathematically, the SSIM metric is articulated through Eq. ( 3 ). $$SSIM\left(x,y\right)=l\left(x, y\right)*c\left(x, y\right)*s(x, y)$$ 3 In Eq. ( 3 ), the variable " \(l\) " denotes the luminance, which is used to compare the brightness of two images. The variable " \(c\) " represents contrast, indicating the variation in intensity range between the brightest and darkest regions of the two images. Meanwhile, " \(s\) " stands for structure, enabling a comparison of the local luminance between the two images to ascertain their similarity. The coordinates ( \(x, y\) ) symbolize the two images being analyzed. Another metric employed in this study, the PSNR (Peak Signal-to-Noise Ratio), is indicative of the highest achievable signal-to-noise ratio. It fulfils the role of quantifying quality assessment in situations involving quality degradation due to various codecs and image compression methods. The PSNR metric involves treating the original image as the signal and the distortion caused by image compression as the disparity. The computation of PSNR involves determining the Mean Squared Error (MSE) – a measure of the squared differences between a noise-free monochrome image and its noise-affected approximation. The mathematical expression for Mean Squared Error (MSE) in the context of a given noise-free image and its noise approximation is elucidated by Eq. ( 4 ). $$MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{[I\left(i, j\right)-K(i, j\left)\right]}^{2}$$ 4 In Eq. ( 4 ), \(m\) and \(n\) are dimensions of a monochrome image \(I\) and \(K\) is the value of noise approximation. The PSNR can be expressed using Eq. ( 5 ). $$PSNR=10\bullet {log}_{10}\left(\frac{{MAX}_{I}^{2}}{MSE}\right)$$ 5 In Eq. ( 5 ), \({MAX}_{I}^{2}\) is the maximum possible pixel value of the image. 4.2 Evaluation results The method we put forth underwent training on an approximate count of 17,000 images encompassing both masked and unmasked faces, subsequently undergoing testing on nearly 14,000 images. The training process was conducted over the span of 300 epochs, employing a batch size of 4. This choice of a smaller batch size was deliberate, aimed at averting potential out-of-memory issues. To assess the effectiveness of the proposed GAN model, its performance was gauged using evaluation metrics including SSIM and PSNR, as detailed in Table 1 . This assessment was carried out on both the training and test datasets, thereby encompassing images akin to those in the training set and images distinct from it present in the test set. Table 1 Metrics findings for the proposed GAN model Dataset SSIM PSNR Training Set 0.91115 31.7251 Testing Set 0.85752 24.9883 As illustrated in Table 1 , the GAN model introduced in this study achieved notable results with an SSIM value of 0.91115 and a PSNR value of 31.7251 on the training set. Similarly, on the test set, the model yielded an SSIM value of 0.85752 and a PSNR value of 24.9883. These outcomes underscore the model's capability to reconstruct the concealed facial regions behind masks, both on similar and diverse images, demonstrating commendable structural similarity and adeptly preserving signal integrity amidst noise. To qualitatively assess the model's performance, we subjected select images from the training set to scrutiny, evaluating its efficacy in reconstructing the obscured facial regions. These reconstructions were then juxtaposed against the ground truth. The results from this qualitative assessment, as depicted in Fig. 8 , were promising and augured well for the model's performance. 4.3 Ablation experiments The outcomes derived from the proposed GAN model showcased remarkable results in both quantitative and qualitative aspects. However, for the sake of validating the model's performance, we undertook an exploration of five additional combinations, each involving alterations in the number of layers within the generator and discriminator modules. These explorations encompassed diverse configurations of encoder and decoder layers, resulting in variations in the tally of trainable and non-trainable parameters. Nonetheless, the empirical investigations established that the most favorable results, as measured by SSIM and PSNR, were consistently attained with the initially proposed GAN model configuration. This specific configuration entailed a generator module comprised of 4 encoder layers, 1 bottleneck layer, and 10 decoder layers, alongside a discriminator module equipped with 6 layers. Detailed insights into the configuration of these custom experiment setups can be gleaned from Table 2 . Further quantified assessments of performance, relying on SSIM and PSNR metrics, can be observed in Tables 3 and 4 . In addition, Fig. 9 offers a visual representation of the qualitative outcomes derived from employing various model configurations. These experimental endeavors provide valuable insights into the judicious selection of generator and discriminator layers, a decision-making process pertinent not only to the specific challenge of face mask removal addressed in this study, but also applicable to broader image-to-image translation tasks. Table 2 Description of the custom GANs trained for comparison Model Generator Layers Discriminator Layers Trainable Parameters Non-Trainable Parameters Total Parameters Model 1 6 encoder, 1 bottleneck, 6 decoder 6 67,097,603 11,177,473 78,275,076 Model 2 7 encoder, 1 bottleneck, 7 decoder 6 54,423,299 11,179,329 65,602,628 Model 3 12 encoder, 1 bottleneck, 12 decoder 6 41,250,691 11,179,073 52,429,764 Model 4 43 encoder, 1 bottleneck, 43 decoder 14 48,885,315 19,314,433 68,199,748 Model 5 7 encoder, 1 bottleneck, 7 decoder 6 82,736,899 11,180,353 93,917,252 Proposed 4 encoder, 1 bottleneck, 10 decoder 6 16,053,603 6,969,569 23,023,172 Table 3 Evaluation results of custom GANs on training set Model SSIM PSNR Model 1 0.85640 28.4696 Model 2 0.88068 30.1880 Model 3 0.75869 23.8032 Model 4 0.76087 22.2019 Model 5 0.87019 29.4226 Proposed 0.91115 31.7251 Table 4 Evaluation results of custom GANs on test set Model SSIM PSNR Model 1 0.83119 25.8888 Model 2 0.84232 26.1952 Model 3 0.74868 22.8974 Model 4 0.75919 22.0925 Model 5 0.83647 25.9400 Proposed 0.85752 24.9883 4.3.1 Limitations In order to gauge the performance of the proposed GAN model, we conducted a qualitative assessment on images that were dissimilar to any of those encompassed within the training set. Within this evaluation, certain image samples underwent a process of facial area reconstruction by the proposed GAN model, resulting in distorted outcomes. These distortions could potentially be attributed to factors such as the network's failure to facilitate gradient propagation across all layers. The detailed presentation of these distorted images generated by the proposed GAN model, juxtaposed against their corresponding ground truths, can be observed in Fig. 10 . It is pertinent to note that despite the distortions, the generated images remain identifiable by human observers, predominantly due to the preservation of significant facial features. 4.3.2 Face recognition challenge To assess the capability of the proposed method in generating images that closely resemble actual facial images, we conducted experiments involving face recognition using edge devices compatible with deep learning-based classifiers, specifically MobileNet [ 26 ] and NasNet [ 27 ]. This evaluation was carried out with the utilization of paired images: genuine facial images and synthetic facial images generated by the proposed GAN model. For the purpose of training and testing the classifiers, a dataset was meticulously curated, comprising 1000 real facial images and an equivalent count of plausible facial images. This dataset was divided into an 80:20 ratio for training and testing purposes. The performance assessment of the deep learning classifiers was carried out through metrics including Accuracy, Precision, Recall, and F-1 Score, as explicated by equations (6–9). The outcomes of these performance evaluations for face recognition, employing deep learning classifiers, are comprehensively detailed in Table 5 . Accuracy = \(\frac{TP + FP}{TP + TN + FP + FN}\) (6) Precision = \(\frac{TP}{TP + FP}\) (7) Recall = \(\frac{TP}{TP + FN}\) (8) F-1 Score = \(2*\frac{Precision * Recall}{Precision+Recall}\) (9) Table 5 Deep learning classifiers results for facial recognition Classifier Class Precision Recall F-1 Score Accuracy MobileNet Real 96% 96% 97% 96.25% Plausible 94% 95% 95% NasNet Real 93% 94% 94% 94.50% Plausible 94% 95% 95% In the context of the classification task aimed at distinguishing between authentic and credible faces, the MobilNet classifier showcased a precision of 96.25%, whereas the NasNet classifier achieved an accuracy of 94.50%. Moreover, impressive results were obtained across a spectrum of performance metrics. These findings emphasize that the synthetic faces produced by the suggested model exhibit discernible attributes, enabling them to be distinguished from genuine faces. As delineated in Table 5 , the obtained results affirm that the images produced by the proposed UNET + + and CoGAN-based Pix2Pix model exhibit accuracy and possess the potential to be utilized in face recognition systems for the purpose of discerning between genuine and fabricated faces. To evaluate the generalization of the proposed method, we tested the proposed method on test images of faces with masks extracted from the face mask detection (FMD) dataset [ 23 ] and evaluated for SSIM and PSNR metrics. The quantitative and qualitative results of the test are presented in Table 6 and Fig. 11. Table 6 Quantitative results on FMD dataset Method SSIM PSNR UNET + + and CoGAN (Proposed) 0.90872 29.4723 For the test conducted to determine the generalization of the proposed method, an SSIM value of 0.90872 was achieved that indicates for random masked faces a significant structural similarity is retained by the proposed method. Further, for the PSNR metric, it achieved a value of 29.4723 that indicate a significant amount of signal in images is retained over noise thus, making the proposed method capable of reconstructing facial area behind masks with significant accuracy. Since, the proposed method works on the principle of image-to-image translation therefore, better results can be obtained in paired images with the availability of ground truths and masked faces. Furthermore, to gauge the validity of the proposed method in removing face masks from low resolution images, we trained the proposed method with a few images of faces with synthetic masks captured from distance and low-resolution camera and tested for its performance. The results for the distant and low-resolution images in removing and reconstructing the face area behind masks is presented in Fig. 12 . 4.4 Comparison with related work To conduct a thorough comparative analysis between the proposed method and recent contributions in the field, we subjected benchmark face mask removal techniques to training and testing using the curated dataset. The prevailing trends in face mask removal approaches predominantly involve the utilization of GANs, albeit with variations in architectural designs. The comprehensive breakdown of this comparative analysis is provided in Table 7 . Table 7 Comparative analysis with previous studies Work Method SSIM PSNR Ud Din et al. [ 11 ] GAN employing a single generator and dual discriminators 0.85237 25.3752 Jiang et al. [ 13 ] Single generator and discriminator-based GAN 0.89981 28.9085 Farahanipad et al. [ 14 ] Cycle-GAN 0.86314 26.1518 Ours UNET + + and CoGAN-based Pix2Pix 0.91115 31.7251 The outcomes presented in Table 7 distinctly demonstrate that the proposed approach outperforms the GAN architectures outlined in the related studies, as evidenced by superior SSIM and PSNR metric scores. In particular, the method put forth demonstrates enhancements spanning from 1.2–5.9% for SSIM and 2.82–6.35% for PSNR metrics, when contrasted with the GAN architectures advocated in earlier studies. It's worth highlighting that the technique advanced by Jiang et al. [ 21 ], which hinges on a single generator and discriminator, bears striking resemblance to the framework of the suggested approach. Nonetheless, the results continue to underscore the superiority of the proposed method in terms of structural similarity and signal preservation over noise, surpassing other GAN-based methodologies when evaluated on the dataset employed within this study. 5. Conclusions and Future Work Generative Adversarial Networks (GANs) have emerged as a potent instrument for crafting realistic images, audio, and videos, with applications proliferating across various sectors like healthcare, design, and computer gaming. In this research, we present an innovative approach that amalgamates UNET + + and CoGAN-based Pix2Pix GAN for the task of revealing concealed faces. The UNET + + architecture is meticulously crafted using a fusion of convolution layers, convolution transpose layers, max pooling layers, and batch normalization layers, each equipped with distinct kernel sizes and strides. Our devised method not only surpasses several state-of-the-art models but also exhibits impressive proficiency in removing masks. The incorporation of a modified UNET + + model as the generator notably enhances the overall performance of our GAN, particularly reflected in elevated SSIM and PSNR scores. A striking attribute of our model is its adaptability in eliminating various mask types, transcending the constraints of specific training masks. Furthermore, we have curated an original dataset tailored for paired image-to-image translation, which uniquely encompasses images of individuals within the criminal domain, augmenting the dataset's relevance to real-world scenarios. In the future, our research direction could focus on harnessing the proposed GAN model for tasks like identity recognition and preservation. This might involve the utilization of models such as StyleGAN to infuse distinctive aesthetic characteristics into generated facial images, thereby broadening the horizon of potential applications. Declarations Availability of data and materials 1. The original Bollywood Celebrity Face dataset is available at: https://www.kaggle.com/datasets/havingfun/100-bollywood-celebrity-faces. 2. The created dataset and code of the UNET++ and CoGAN method is available on request. Competing interests The authors have no competing interests to disclose. Funding The authors express their gratitude to the All India Council of Technical Education for providing financial support for this study. This research has been sponsored under the Research Promotion Scheme of AICTE, India, with reference file number 8-108/FDC/RPS(POLICY-1/2019-20). Authors’ contribution Akhil Kumar (AK) contributed to methodology design, implementation, formal analysis and writing and editing original and final manuscript. Divyam Gupta (DG) contributed to methodology design, implementation and formal analysis. Manisha Kaushal (MK) contributed to editing and reviewing original and final draft of the manuscript. Akashdeep Sharma (AS) supervised the entire work with additional responsibility of conceptualization and methodology design. Acknowledgements Not applicable References Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu, B, Warde-Farley D, Ozair S, Courville Bengio Y (2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems, (Neural information processing systems foundation), pp. 2672–2680. https://doi.org/10.1007/978-3-658-40442-0_9. Mirza M, Osindero S (2018) Conditional Generative Adversarial Nets. In arXiv. http://arxiv.org/abs/1411.1784. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision (Vol. 2017-October, pp. 2242–2251. https://doi.org/10.1109/ICCV.2017.244. Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas D (2017) StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5908–5916. https://doi.org/10.1109/ICCV.2017.629. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition. pp. 5967–5976. https://doi.org/10.1109/CVPR.2017.632. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative Image Inpainting with Contextual Attention. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 5505–5514. https://doi.org/10.1109/CVPR.2018.00577. Deng J, Dong W, Socher R, Li L-J, Kai Li, Li Fei-Fei (2010) ImageNet: A large-scale hierarchical image database. https://doi.org/10.1109/cvpr.2009.5206848. Nazeri K, Ng E, Joseph T, Qureshi F, Ebrahimi M (2019) Edgeconnect: Structure guided image inpainting using edge prediction. In: Proceedings - 2019 International Conference on Computer Vision Workshop. pp. 3265–3274. https://doi.org/10.1109/ICCVW.2019.00408. Doersch C, Singh S, Gupta A, Sivic J, Efros A (2015) What makes Paris look like Paris? Commun. ACM 58(12):103–110. https://doi.org/10.1145/2830541. Khan MKJ, Ud Din N, Bae S, Yi J (2019) Interactive removal of microphone object in facial images. Electronics 8(10). https://doi.org/10.3390/electronics8101115. Ud Din N, Javed K, Bae S, Yi J (2020) A Novel GAN-Based Network for Unmasking of Masked Face. IEEE Access. 8:44276–44287. https://doi.org/10.1109/ACCESS.2020.2977386. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Lecture Notes in Computer Science. pp. 234–241. Springer Verlag. https://doi.org/10.1007/978-3-319-24574-4_28. Jiang Y, Yang F, Bian Z, Lu C, Xia S (2022) Mask removal : Face inpainting via attributes. Multimedia Tools Appl. 81(21):29785–29797. https://doi.org/10.1007/s11042-022-12912-1. Farahanipad F, Rezaei M, Nasr M, Kamangar F, Athitsos V (2022) GAN-based Face Reconstruction for Masked-Face. In: ACM International Conference Proceeding Series. pp. 583–587. https://doi.org/10.1145/3529190.3534774. Anwar A, Raychowdhury A (2020) Masked Face Recognition for Secure Authentication. In arXiv. https://arxiv.org/abs/2008.11104. Redmon J, Farhadi, A (2018) YOLO v3.0: An Incremental Improvement. In arXiv. https://doi.org/10.48550/arXiv.1804.02767. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2242–2251. https://doi.org/10.1109/ICCV.2017.244. Hamzah Ordiyasa IW, Najib MHR. (2023) DeepMask: face mask detection using GAN algorithm. Iran J.Comput. Sci. 6(1): 13–19. https://doi.org/10.1007/s42044-022-00114-9. Ye L (2022) Mask Removal Algorithm Using GAN Based Models. In: Proceedings - 2022 3rd International Conference on Electronic Communication and Artificial Intelligence. pp. 281–285. https://doi.org/10.1109/IWECAI55315.2022.00060. Bollywood Celebrity Faces Dataset (2020) https://www.kaggle.com/datasets/havingfun/100-bollywood-celebrity-faces. Accessed 20 May 2023. Kumar A, Kaushal M, Sharma A (2023) SAM C-GAN: a method for removal of face masks from masked faces. Signal, Image Video Process. https://doi.org/10.1007/s11760-023-02602-2. Kumar A, Kalia A, Sharma A, Kaushal M (2023) A hybrid tiny YOLO v4-SPP module based improved face mask detection vision system. J. Ambient Intell. Hum. Comput. 14(6):6783–6796. https://doi.org/10.1007/s12652-021-03541-x. Kumar A, Kalia A, Verma K, Sharma A, Kaushal M (2021) Scaling up face masks detection with YOLO on a novel dataset. Optik, 239. https://doi.org/10.1016/j.ijleo.2021.166744. OpenCV DNN Module. https://docs.opencv.org. Accessed 10 June 2023 Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Lecture Notes in Computer Science. pp. 3–11. Springer Verlag. https://doi.org/10.1007/978-3-030-00889-5_1. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. In arXiv. https://arxiv.org/abs/1704.04861. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning Transferable Architectures for Scalable Image Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 8697–8710. https://doi.org/10.1109/CVPR.2018.00907. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3351025","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":232771737,"identity":"bdc6662a-a1a8-4ad4-9383-1476d3ceeab6","order_by":0,"name":"Akhil Kumar","email":"","orcid":"","institution":"SCSET, Bennett University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Akhil","middleName":"","lastName":"Kumar","suffix":""},{"id":232771738,"identity":"6c9d936c-ca1b-4af6-95af-677430e378fe","order_by":1,"name":"Divyam Gupta","email":"","orcid":"","institution":"CSE, UIET, Panjab University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Divyam","middleName":"","lastName":"Gupta","suffix":""},{"id":232771739,"identity":"08bbbace-02bc-411c-bf74-53db91e878ce","order_by":2,"name":"Manisha Kaushal","email":"","orcid":"","institution":"Thapar Institute of Engineering \u0026 Technology","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Manisha","middleName":"","lastName":"Kaushal","suffix":""},{"id":232771741,"identity":"6ef3a5c9-174d-4c28-b454-1e26fdb9f213","order_by":3,"name":"Akashdeep Sharma","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1ElEQVRIiWNgGAWjYDACHgYGZgY2IIO9ASLARrwWngMka5FIINJd/D2HH38uKLORM5d8e0yCocaOgU+6Ab8WibNtZtIzzqUZW87OS5NgOJbMwCZzgIA15xnMmHnbDiduuJ1jJsHAdoCBjZAL5c+zf/7M2/a/fsPNM0At/4jQYnC2x0Cat+1AgsENHjMJxjYitBieOVMmzXMu2XBnT16yRWJfMg9BLXJn0jd/5imzkzdnP3vwxodvdnLyMwhoQbgQFEMJ4HgiFhiQongUjIJRMApGFgAAuMs6+bWp5Q0AAAAASUVORK5CYII=","orcid":"","institution":"CSE, UIET, Panjab University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Akashdeep","middleName":"","lastName":"Sharma","suffix":""}],"badges":[],"createdAt":"2023-09-13 07:59:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3351025/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3351025/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":43304825,"identity":"c155e73d-8f80-44b5-8d53-d69eb76126ed","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":340259,"visible":true,"origin":"","legend":"\u003cp\u003eDataset description\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/83e8784d6467692e9dcfa84f.png"},{"id":43306018,"identity":"7bae9153-4fc2-4333-b9d6-1cd6a3a6221b","added_by":"auto","created_at":"2023-09-18 16:02:46","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":110322,"visible":true,"origin":"","legend":"\u003cp\u003eDetection of face mask regions via the tiny YOLO v4-SPP detector\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/d5805e207139c376ec29a47c.png"},{"id":43304826,"identity":"b8b945b8-7272-443c-b187-8d85de36a086","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":330301,"visible":true,"origin":"","legend":"\u003cp\u003eDataset creation process\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/c83d8af5e4fedcbe9d046ed6.png"},{"id":43306014,"identity":"fce6d77f-0f74-4072-bf37-fe54ffcf2cfc","added_by":"auto","created_at":"2023-09-18 16:02:46","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":206286,"visible":true,"origin":"","legend":"\u003cp\u003eArchitecture of proposed model\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/44245e7aa698f4c5c0870450.png"},{"id":43304824,"identity":"8ae14acf-85c7-4e35-82fd-a4fcbcf8364c","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":88347,"visible":true,"origin":"","legend":"\u003cp\u003eDiscriminator model\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/7423a5812f0f1a403337c4d9.png"},{"id":43306237,"identity":"a9b8b60e-44b6-4944-96eb-aa4a5672df5b","added_by":"auto","created_at":"2023-09-18 16:10:46","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":176988,"visible":true,"origin":"","legend":"\u003cp\u003eUNET++ architecture\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/2b5f52e560c3185bf97018af.png"},{"id":43306016,"identity":"3857966c-2835-4b71-b485-d8f374a06555","added_by":"auto","created_at":"2023-09-18 16:02:46","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":221159,"visible":true,"origin":"","legend":"\u003cp\u003ePlots for loss\u003c/p\u003e\n\u003cp\u003ea) Generator loss \u0026nbsp;\u0026nbsp;\u0026nbsp;b) Discriminator loss \u0026nbsp;c) Proposed method loss\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/f553c2e4a51793ae220af523.png"},{"id":43308096,"identity":"50aa44cf-bf74-431c-85ae-2e8ecee7e49e","added_by":"auto","created_at":"2023-09-18 16:18:46","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":2331994,"visible":true,"origin":"","legend":"\u003cp\u003eImages generated by proposed GAN for faces without mask\u003c/p\u003e","description":"","filename":"floatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/c28f58a035adc361a4a38a3a.png"},{"id":43304828,"identity":"939fb6dc-dbcf-4ac5-90ca-eeb048158f3b","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":144772,"visible":true,"origin":"","legend":"\u003cp\u003ePlots for SSIM and PSNR values with custom GANs\u003c/p\u003e\n\u003cp\u003ea) SSIM values \u0026nbsp;b) PSNR values\u003c/p\u003e","description":"","filename":"floatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/b0bc1d61a2eb5314ccabf09c.png"},{"id":43304832,"identity":"12fe5e10-9593-4674-ab9d-dd4e64cd8008","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":1712217,"visible":true,"origin":"","legend":"\u003cp\u003eFailed examples with the proposed GAN model\u003c/p\u003e","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/b30db077c7d4958a40b74aa3.png"},{"id":43304835,"identity":"764c41f0-302f-49a0-82a5-13821a6f8e39","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":336788,"visible":true,"origin":"","legend":"\u003cp\u003eQualitative results of test on FMD dataset\u003c/p\u003e","description":"","filename":"11.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/cae1cf6b0ca73829908ce236.png"},{"id":43304834,"identity":"5d61bf08-d943-45b7-ac41-60de19ab38f6","added_by":"auto","created_at":"2023-09-18 15:54:46","extension":"png","order_by":12,"title":"Figure 12","display":"","copyAsset":false,"role":"figure","size":412195,"visible":true,"origin":"","legend":"\u003cp\u003eProposed method results on low-resolution images\u003c/p\u003e","description":"","filename":"12.png","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/06a5365e43667fac9e99fa3f.png"},{"id":44098924,"identity":"3f900112-82e1-4877-be3f-7899f105637e","added_by":"auto","created_at":"2023-10-04 17:52:36","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5903340,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3351025/v1/48d67754-bdb2-4da2-98e5-ff91cf5426bd.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A UNET++ and CoGAN-based method to remove face masks from the masked faces","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eGenerative Adversarial Networks (GANs) [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] constitute a deep learning approach to generative modeling, often utilizing convolutional neural networks. The GAN framework encompasses a pair of neural networks, specifically the Generator and the Discriminator, engaged in a competitive dynamic to create novel images. The Generator produces batches of samples, juxtaposed with genuine domain instances, and feeds them to the Discriminator for classification into real or synthetic categories. This adversarial interplay reflects a zero-sum game within game theory. The prowess of Generative Adversarial Networks (GANs) in producing remarkably authentic and finely detailed images has fueled their exceptional performance in image generation. Among the array of successful image translation applications for GANs, such as CoGAN [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], cycleGAN [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], stackGAN [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], and more, this study centers on harnessing the Pix2Pix [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] network for its research objectives. However, a constraint of the GAN model lies in its potential to generate arbitrary domain images. The conditional GAN, or CoGAN, counteracts this by enabling targeted image generation of specific categories. This involves introducing an additional input layer in the form of one-hot-encoded image labels, guiding the generator during training to synthesize images of a designated type. In the context of face mask removal using GANs, the process entails two steps: (1) identifying the mask, generally by delineating the mask region's boundary, and (2) generating the concealed region obscured by the mask. These steps collectively contribute to the final synthesized image.\u003c/p\u003e \u003cp\u003eThe present study puts forth a variant of the Pix2Pix GAN to facilitate the generation of unmasked facial images from obscured and blurred inputs, employing a pair-to-pair image translation paradigm. Furthermore, we introduce a meticulously curated dataset that establishes a direct correspondence between input and output images, embodying the transformation from masked to unmasked representations. The principal contributions of this paper encompass:\u003c/p\u003e \u003cp\u003e1) We introduce a novel GAN architecture designed to unveil concealed facial features. In pursuit of effective image-to-image translation, we present an innovative adaptation of the CoGAN-based Pix2Pix GAN framework. Our approach involves the integration of the UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture into the generator model, alongside a customized array of convolutional networks for the discriminator models. The outcomes of our proposed model are noteworthy, with SSIM values of 0.91115 and 0.85752, as well as PSNR values of 31.7251 and 24.9883, observed on the training and test sets, respectively. These results are indicative of its ability to uphold both structural fidelity and signal clarity amidst noise when reconstructing the facial region obscured by a mask.\u003c/p\u003e \u003cp\u003e2) Furthermore, we introduce a dataset comprising depictions of individuals donning masks juxtaposed with corresponding images of the same individuals without masks. This comprehensive dataset encompasses more than 30,000 facial images, capturing diverse instances of individuals wearing three distinct types of masks.\u003c/p\u003e \u003cp\u003e3) Comparative analyses reveal the notably superior performance and heightened suitability of the proposed GAN within real-world contexts. When contrasted with analogous approaches, our GAN model achieved a significant enhancement, showcasing an improvement ranging from 1.2\u0026ndash;5.9% in SSIM scores and a further advancement of 2.82\u0026ndash;6.35% in PSNR metric scores.\u003c/p\u003e \u003cp\u003eThis work encompasses the subsequent sections: In Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, an exploration of the pertinent literature within the realms of image editing and face mask removal is presented. Section \u003cspan refid=\"Sec3\" class=\"InternalRef\"\u003e3\u003c/span\u003e delves into the materials and methodologies employed, elucidating the formulation of the dataset and the development of the model. The experimental assessments, outcomes, and limitations of this endeavor are expounded upon in Section \u003cspan refid=\"Sec8\" class=\"InternalRef\"\u003e4\u003c/span\u003e. Finally, Section \u003cspan refid=\"Sec15\" class=\"InternalRef\"\u003e5\u003c/span\u003e encapsulates the conclusion drawn from this study and outlines prospects for future advancements within the domain.\u003c/p\u003e"},{"header":"2. Related work","content":"\u003cp\u003eGANs have established their presence across diverse domains, encompassing tasks such as object manipulation and generation, image and sound synthesis, and high-resolution image creation. This section provides an overview of GAN advancements with a specific focus on image generation, particularly concerning concealed portions of images and facial structures.\u003c/p\u003e \u003cp\u003eYu et al. [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] introduced an image editing technique centered on GANs, employing a contextual attention module in conjunction with a coarse-to-fine GAN-based strategy for image inpainting. Their work was validated through the utilization of Places2, CelebA, and ImageNet [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] datasets. Nazeri et al. [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] proposed a two-stage adversarial network for image editing, comprising an edge generator and an image editing module. The CelebA, Places2, and Paris StreetView [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] datasets served as the basis for evaluation. Khan et al. [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] utilized a coarse-to-fine Generative Adversarial Network to remove microphones from facial images, leveraging synthesized images from the CelebA dataset. Isola et al. [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] introduced a straightforward framework for image generation, featuring a \"U-Net\" based generator architecture and a convolutional \"PatchGAN\" classifier discriminator. Their results highlighted the potential of conditional adversarial networks for image-to-image translation tasks.\u003c/p\u003e \u003cp\u003eA stream of GAN-based works has surfaced in recent times, specifically addressing face inpainting and reconstruction with and without masks. Ud Din et al. [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] employed a U-NET [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] architecture for image binarization, utilizing a single generator and two discriminators to generate mask-free face images from masked counterparts within the CelebA dataset. Jiang et al. [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] proposed a dual GAN approach for the reconstruction of concealed facial areas, building synthetic datasets with and without masks. Farahanipad et al. [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] introduced masked face unmasking using GANs, leveraging the \"MaskTheFace\" tool [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] for dataset creation and the YOLO algorithm [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] for region detection, ultimately employing a CycleGAN [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] model for unmasking. Subsequently, others followed similar approaches [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] using the Pix2Pix GAN, generating datasets and employing binary segmentation in their modified models. Kumar et al. [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] introduced a spatial attention module within the C-GAN framework for unmasking masked faces, yielding high-accuracy results.\u003c/p\u003e \u003cp\u003eExamining related research on object removal, image editing, and face mask removal and reconstruction underscores the scarcity of techniques proficient in identifying face masks, removing them, and accurately reconstructing the unmasked image. Addressing these gaps, our work endeavours to devise a method capable of adeptly removing face masks, facilitating accurate synthesis of unmasked facial images.\u003c/p\u003e"},{"header":"3. Materials and methods","content":"\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Dataset\u003c/h2\u003e \u003cp\u003eThe proposed approach employs a technique of paired image-to-image translation, necessitating a specialized dataset that ensures a one-to-one correspondence between input and output images. As such, the creation of a tailored dataset featuring these two distinct image categories was imperative. The input images depict individuals with masks covering their faces, while the target images portray the same individuals without masks, representing their unobstructed facial appearance.\u003c/p\u003e \u003cp\u003eGiven the limited availability of suitable publicly accessible datasets, our work took the initiative to curate a customized dataset utilizing images of renowned Bollywood celebrities, conveniently accessible on Kaggle [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. This dataset encompasses a comprehensive collection of 12,355 images showcasing the top 100 Bollywood personalities. To enhance the dataset's realism, we also incorporated sample images from the dataset used by authors in [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], thus augmenting the dataset's diversity and mitigating the risk of neural networks converging to local minima. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e offers a glimpse of the proposed dataset's structure. The dataset preparation involved cropping facial regions and eliminating redundant images, resulting in a final count of 11,262 images without face masks. To introduce variety, the dataset featured three distinct types of face masks superimposed on the images: white face masks, Anti-COVID face masks, and 3M face masks. Consequently, the modified dataset encompassed 11,262 images of unmasked faces, as well as equal numbers of images for each type of mask. It's important to note that the method employed for mask removal from facial regions draws upon both the images of unmasked faces and faces wearing masks.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIncorporating OpenCV's DNN module [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], we proficiently render face masks onto facial regions. This module aids in extracting facial landmarks, generating a comprehensive set of 68 landmarks across the face, encompassing regions like eyes, eyebrows, nose, mouth, and jawline. These landmarks subsequently serve as reference points for precisely overlaying face masks as needed. To ensure optimal mask alignment, we thoughtfully annotated mask points corresponding to key landmarks along the jawline's base and the top of the nose. The alignment process employs the OpenCV library to determine transformations between matched key points, mask annotations, and corresponding facial landmarks. Subsequently, the identified transformation matrix is applied to map these points accurately. To bolster the proposed model's proficiency in predicting face mask regions, we undertook the training of a tiny YOLO v4-SPP face mask detector [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. The face mask detector utilized in this study was employed to identify areas with face masks in a publicly available face mask detection dataset [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. This dataset encompasses a substantial compilation of 52,635 images featuring individuals both with and without face masks. The regions identified by the tiny YOLO v4-SPP face mask detector were extracted, resulting in cropped images of the detected face mask areas within the images that depicted mask-wearing. These identified face mask regions, denoted by black rectangular boxes, were subsequently overlaid onto the modified Bollywood Celebrity dataset. By strategically integrating these highlighted mask regions into the altered images, we facilitated the precise identification of various types and complexities of face masks by the proposed model. This streamlined the process of removing masks from obscured faces. The comprehensive process of mask area detection via the tiny YOLO v4-SPP face mask detector is elucidated in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, while the intricate steps of dataset creation are meticulously elaborated upon in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Approach\u003c/h2\u003e \u003cp\u003eIn this section, we provide an in-depth insight into the proposed GAN architecture, which is founded upon a modified UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;encoder-decoder design for the purpose of unveiling concealed facial features. The operational intricacies of this architecture are expounded upon. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e visually represents the comprehensive architecture adopted within this study. The underlying model takes an input image and proceeds to identify the presence of a facial mask. Subsequently, it undertakes the task of reconstructing the obscured facial region that lies behind the detected mask. The subsequent sections delve into the architecture's specifics and mechanics, unraveling its inner workings in detail.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003ea) Discriminator module\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe discriminator's role involves processing both a source image (a facial image with a mask) and a target image (a facial image without a mask). It then assesses the likelihood that the target image is either a genuine or a synthesized transformation of the source image. The architecture of the discriminator is intricately tied to the effective receptive field of the model, which determines the ratio between the model's output and the number of input image pixels. For this endeavor, the PatchGAN architecture was chosen for the discriminator. This design meticulously aligns each output prediction of the model to a 70x70 square patch within the input image. This strategy offers the advantage of adaptable applicability to input images of varying sizes, whether larger or smaller than 256x256 pixels. The model's output can take the form of a singular value or a square activation map, featuring values that assess the authenticity of each patch in the input image\u0026mdash;whether it's real or fake. These values can be averaged to provide an overall probability or classification score if desired. The PatchGAN's structure is influenced by the effective receptive field size, often referred to as the receptive field. This term denotes the relationship between a model's output activation and a region within the input image. Notably, the receptive field isn't equivalent to the discriminator model's output size. Instead, it pertains to a model that outputs a pixel in the activation map corresponding to a pixel in the input image. Traditionally, the receptive field is defined based on the size of an individual convolutional layer's activation map relative to its input, filter size, and stride value. The 70x70 PatchGAN adheres to a fixed number of three layers (excluding the output and second-to-last layers), regardless of the input image's dimensions. The formula for calculating the one-dimensional receptive field is: \u003cem\u003ereceptive field = (output_size \u0026minus;\u0026thinsp;1) * stride\u0026thinsp;+\u0026thinsp;kernel_size\u003c/em\u003e, where output_size represents the previous layer's activation map size, stride signifies the filter's application shift, and kernel_size indicates the filter's dimensions. Furthermore, the discriminator module processes two concatenated input images and generates a prediction matrix, as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e. It consists of a series of layers including Convolution, Batch Normalization, and LeakyReLU. The proposed discriminator module encompasses six Convolution layers, with a kernel size of 4 and a stride of 2 for the initial four layers. The training of the discriminator employs the Adam optimizer with a learning rate of 0.0001, a beta_1 hyperparameter of 0.5, and the Poisson loss function. The model comprises a total of 6,968,257 parameters (2,816 trainable and 6,965,441 non-trainable parameters). The model's weights are initialized with a normal distribution characterized by a standard deviation of 0.02.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eb) Generator module\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe generator module within the proposed CoGAN-based Pix2Pix architecture adopts the UNET architecture, characterized by its distinctive \"U\" shape. This symmetrical structure comprises two integral parts: the contracting path, situated on the left, and the expansive path forming the right side of the U. The contracting path employs convolution processes, while the expansive path employs 2D transpose convolution layers. This encoder-decoder model employs skip connections, linking equivalent layers between the encoder and decoder blocks that share matching feature map sizes.\u003c/p\u003e \u003cp\u003eIn this work, we incorporate a modification of the UNET architecture known as UNET++. UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;refines the traditional skip connections present in UNET by addressing the semantic gap between encoder and decoder feature maps before concatenation. This architecture utilizes nested and dense skip connections, effectively capturing intricate details of 2D images.\u003c/p\u003e \u003cp\u003eThe UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;structure comprises encoder and decoder blocks interconnected through a sequence of nested dense convolution blocks, as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e. It differs from the original UNET architecture in several key ways:\u003c/p\u003e \u003cp\u003e1) It integrates convolution layers within skip pathways, narrowing the semantic gap between encoder and decoder feature maps.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e2) It introduces dense skip connections on skip pathways to enhance gradient flow.\u003c/h3\u003e\n\n\u003ch3\u003e3) It incorporates deep supervision, enabling model pruning and performance enhancement.\u003c/h3\u003e\n\u003cp\u003eThe UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture enhances segmentation accuracy by incorporating Dense blocks and Convolution layers between the encoder and decoder. It achieves this by redesigning the skip pathways to bridge the semantic gap between encoder and decoder sub-paths. These convolution layers aim to minimize the semantic gap between the feature maps of the encoder and decoder sub-networks, simplifying optimization for the optimizer.\u003c/p\u003e \u003cp\u003eDense skip connections, inspired by DenseNet, are implemented within UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;as skip pathways between the encoder and decoder. These Dense blocks accumulate previous feature maps and deliver them to the current node through the dense convolution block along each skip pathway. This generates high-resolution feature maps at multiple semantic levels.\u003c/p\u003e \u003cp\u003eCapitalizing on UNET++'s advantages, we propose a modified version of the original Pix2Pix GAN, featuring a generator module built on the UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture. The generator module comprises an encoder sub-network or backbone followed by a decoder sub-network. Each encoder block is composed of Convolution layers, BatchNormalization layers, and ReLU activation layers. The encoder network consists of five such blocks, all with a kernel size and stride of 2 and identical padding. The decoder block's upsampling layers employ Convolution transpose layers with a kernel size and stride of 2 and similar padding. The final output layer employs a convolution layer with a kernel size of 1 and a Tanh activation function. This choice of activation ensures pixel values in the generated image lie within the range of [-1, 1]. The generator module is updated using a weighted combination of Poisson and Mean Squared Error losses, with the weighting skewed toward Mean Squared Error. The Adam Optimizer is employed to minimize the losses, featuring a learning rate of 0.0001 and beta_1 of 0.5.\u003c/p\u003e \u003cp\u003eTo propose a method for unmasking masked faces, we integrate the previously described discriminator and generator modules, creating the UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN-based Pix2Pix GAN model. This combined GAN model operates with a substantial 23,023,172 parameters, out of which 16,053,603 are trainable, and 6,969,569 are non-trainable.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003ec) Loss function\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe GAN model we propose employs a combination of Poisson and Mean Squared Error losses to address issues of vanishing gradients and convergence. The specifics of these two loss functions are outlined as follows.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003e1) Poisson loss\u003c/strong\u003e \u003cp\u003eIn the context of our proposed GAN model, the Poisson loss function serves as a tool for regression, particularly suited for modeling count data. This loss function gauges the dissimilarity between the projected outcome and the observed actual output. Mathematically, the Poisson loss can be expressed through Eq.\u0026nbsp;(\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv id=\"Equ1\" class=\"Equation\"\u003e \u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$L\\left(y, \\widehat{y}\\right)= \\frac{1}{N} \\sum _{i=0}^{N}({\\widehat{y}}_{i}-{y}_{i}log{\\widehat{y}}_{i})$$\u003c/div\u003e \u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe act of minimizing the Poisson loss is synonymous with maximizing the likelihood of the data, assuming that the target follows a Poisson distribution given the input. This loss is applicable when there's a belief that the target value adheres to a Poisson distribution and the aim is to model the rate parameter based on a particular input.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003e2) Mean Squared Error\u003c/strong\u003e \u003cp\u003eThe Mean Squared Error (MSE) serves as a widely used loss function in regression scenarios. This loss function quantifies the average of squared disparities between predicted and actual values across the provided data. MSE is particularly responsive to outliers; when confronted with numerous instances featuring identical input features, the optimal prediction is their mean target value. This stands in contrast to the Mean Absolute Error, where the median becomes the optimal prediction. MSE is especially suitable when the target data, contingent on the input, adheres to a normal distribution centered around a mean value. Furthermore, it's beneficial when the intention is to penalize outliers effectively. The mathematical representation of the Mean Squared Error (MSE) loss is captured by Eq.\u0026nbsp;(\u003cspan refid=\"Equ2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv id=\"Equ2\" class=\"Equation\"\u003e \u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$L\\left(y, \\widehat{y}\\right)= \\frac{1}{N} \\sum _{i=0}^{N}{(y-{\\widehat{y}}_{i})}^{2}$$\u003c/div\u003e \u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eIn Eq.\u0026nbsp;(\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) for Poisson loss and (2) for Mean Squared Error (MSE) loss, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(y\\)\u003c/span\u003e\u003c/span\u003e represents the actual output whereas, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{y}\\)\u003c/span\u003e\u003c/span\u003e is the predicted expected value for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e data count and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(N\\)\u003c/span\u003e\u003c/span\u003ecomputations.\u003c/p\u003e \u003cp\u003eThe previously outlined generator and discriminator, coupled with the described loss functions, exhibited a favorable loss curve during training. This curve reflects successful training on the training set, contributing to the accomplishment of the image-to-image translation objective. Graphical representations of the generator loss, discriminator loss, and the comprehensive GAN model loss are depicted in Fig.\u0026nbsp;7.\u003c/p\u003e "},{"header":"4. Experiments and Results","content":"\u003cp\u003eIn this section, we delve into the results garnered from a series of experiments conducted to validate the efficacy of the proposed approach. The UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN-based Pix2Pix model, designed for unmasking masked faces, was implemented on a system featuring the following configuration: Intel\u0026reg; Core i7-6850K CPU @ 3.60 GHz x 12, coupled with 64 GB of RAM, and two NVIDIA TITAN XP 12 GB GPUs. The dataset was partitioned into training and testing subsets, maintaining a 55:45 ratio. The training subset, constituting 55% of the images, was employed for generating realistic images of unmasked faces using the generator module. To facilitate the discrimination between generator module outputs and real images of unmasked faces, the discriminator module was presented with outputs from the generator module along with a collection of 17,603 genuine unmasked face images. For evaluation purposes, a distinct set of 14,000 images from the testing subset, exclusive of the training set\u0026apos;s masked face images, was utilized. The training phase of the proposed approach spanned 300 iterations, employing a batch size of 4. The training process was facilitated by the Adam optimizer, leveraging a learning rate of 0.0001 and a beta_1 value of 0.5. In the generator module of the proposed GAN model, the Mean Squared Error (MSE) loss and Poisson loss were utilized, while the discriminator module relied solely on the Poisson loss. Further elaboration includes insight into the evaluation metrics, the sequence of experiments, the resultant evaluation outcomes, and a comparative analysis against other relevant methodologies. These aspects are meticulously presented in the subsequent subsections.\u003c/p\u003e\n\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e\n \u003ch2\u003e4.1 Evaluation metrics\u003c/h2\u003e\n \u003cp\u003eThe central emphasis of this investigation centers on image-to-image translation, making the evaluation of structural similarity between images and their signal-to-noise ratio of utmost importance. To quantify the structural similarity, the SSIM (Structural Similarity Index) metric has been employed. Concurrently, the PSNR (Peak Signal-to-Noise Ratio) metric has been adopted for evaluating signal-to-noise ratio. The SSIM metric, denoting the Structural Similarity Index, is a robust tool for gauging the quality of digital images and videos. Its purpose lies in assessing the likeness between two images, utilizing criteria such as luminance intensity, contrast, and structural characteristics. This metric encapsulates these factors to provide an estimate of similarity between the images. Mathematically, the SSIM metric is articulated through Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e\n \u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e$$SSIM\\left(x,y\\right)=l\\left(x, y\\right)*c\\left(x, y\\right)*s(x, y)$$\u003c/div\u003e\n \u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eIn Eq. (\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e), the variable \u0026quot;\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e\u0026quot; denotes the luminance, which is used to compare the brightness of two images. The variable \u0026quot;\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(c\\)\u003c/span\u003e\u003c/span\u003e\u0026quot; represents contrast, indicating the variation in intensity range between the brightest and darkest regions of the two images. Meanwhile, \u0026quot;\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e\u0026quot; stands for structure, enabling a comparison of the local luminance between the two images to ascertain their similarity. The coordinates (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x, y\\)\u003c/span\u003e\u003c/span\u003e) symbolize the two images being analyzed.\u003c/p\u003e\n \u003cp\u003eAnother metric employed in this study, the PSNR (Peak Signal-to-Noise Ratio), is indicative of the highest achievable signal-to-noise ratio. It fulfils the role of quantifying quality assessment in situations involving quality degradation due to various codecs and image compression methods. The PSNR metric involves treating the original image as the signal and the distortion caused by image compression as the disparity. The computation of PSNR involves determining the Mean Squared Error (MSE) \u0026ndash; a measure of the squared differences between a noise-free monochrome image and its noise-affected approximation. The mathematical expression for Mean Squared Error (MSE) in the context of a given noise-free image and its noise approximation is elucidated by Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e\n \u003cdiv id=\"Equ4\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equ4\" name=\"EquationSource\"\u003e$$MSE=\\frac{1}{mn}\\sum _{i=0}^{m-1}\\sum _{j=0}^{n-1}{[I\\left(i, j\\right)-K(i, j\\left)\\right]}^{2}$$\u003c/div\u003e\n \u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eIn Eq. (\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e), \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(m\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(n\\)\u003c/span\u003e\u003c/span\u003e are dimensions of a monochrome image \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(I\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(K\\)\u003c/span\u003e\u003c/span\u003e is the value of noise approximation.\u003c/p\u003e\n \u003cp\u003eThe PSNR can be expressed using Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e\n \u003cdiv id=\"Equ5\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equ5\" name=\"EquationSource\"\u003e$$PSNR=10\\bullet {log}_{10}\\left(\\frac{{MAX}_{I}^{2}}{MSE}\\right)$$\u003c/div\u003e\n \u003cdiv class=\"EquationNumber\"\u003e5\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eIn Eq. (\u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e), \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({MAX}_{I}^{2}\\)\u003c/span\u003e\u003c/span\u003e is the maximum possible pixel value of the image.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e\n \u003ch2\u003e4.2 Evaluation results\u003c/h2\u003e\n \u003cp\u003eThe method we put forth underwent training on an approximate count of 17,000 images encompassing both masked and unmasked faces, subsequently undergoing testing on nearly 14,000 images. The training process was conducted over the span of 300 epochs, employing a batch size of 4. This choice of a smaller batch size was deliberate, aimed at averting potential out-of-memory issues. To assess the effectiveness of the proposed GAN model, its performance was gauged using evaluation metrics including SSIM and PSNR, as detailed in Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e. This assessment was carried out on both the training and test datasets, thereby encompassing images akin to those in the training set and images distinct from it present in the test set.\u003c/p\u003e\u0026nbsp;\u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eMetrics findings for the proposed GAN model\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDataset\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePSNR\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eTraining Set\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.91115\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e31.7251\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eTesting Set\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.85752\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e24.9883\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003eAs illustrated in Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e, the GAN model introduced in this study achieved notable results with an SSIM value of 0.91115 and a PSNR value of 31.7251 on the training set. Similarly, on the test set, the model yielded an SSIM value of 0.85752 and a PSNR value of 24.9883. These outcomes underscore the model\u0026apos;s capability to reconstruct the concealed facial regions behind masks, both on similar and diverse images, demonstrating commendable structural similarity and adeptly preserving signal integrity amidst noise. To qualitatively assess the model\u0026apos;s performance, we subjected select images from the training set to scrutiny, evaluating its efficacy in reconstructing the obscured facial regions. These reconstructions were then juxtaposed against the ground truth. The results from this qualitative assessment, as depicted in Fig. \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e, were promising and augured well for the model\u0026apos;s performance.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n \u003ch2\u003e4.3 Ablation experiments\u003c/h2\u003e\n \u003cp\u003eThe outcomes derived from the proposed GAN model showcased remarkable results in both quantitative and qualitative aspects. However, for the sake of validating the model\u0026apos;s performance, we undertook an exploration of five additional combinations, each involving alterations in the number of layers within the generator and discriminator modules. These explorations encompassed diverse configurations of encoder and decoder layers, resulting in variations in the tally of trainable and non-trainable parameters.\u003c/p\u003e\n \u003cp\u003eNonetheless, the empirical investigations established that the most favorable results, as measured by SSIM and PSNR, were consistently attained with the initially proposed GAN model configuration. This specific configuration entailed a generator module comprised of 4 encoder layers, 1 bottleneck layer, and 10 decoder layers, alongside a discriminator module equipped with 6 layers. Detailed insights into the configuration of these custom experiment setups can be gleaned from Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e. Further quantified assessments of performance, relying on SSIM and PSNR metrics, can be observed in Tables \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e and \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e. In addition, Fig. 9 offers a visual representation of the qualitative outcomes derived from employing various model configurations. These experimental endeavors provide valuable insights into the judicious selection of generator and discriminator layers, a decision-making process pertinent not only to the specific challenge of face mask removal addressed in this study, but also applicable to broader image-to-image translation tasks.\u0026nbsp;\u003c/p\u003e\u0026nbsp;\u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eDescription of the custom GANs trained for comparison\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eModel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eGenerator Layers\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDiscriminator Layers\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTrainable Parameters\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eNon-Trainable Parameters\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTotal Parameters\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 encoder, 1 bottleneck, 6 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e67,097,603\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e11,177,473\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e78,275,076\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 encoder, 1 bottleneck, 7 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e54,423,299\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e11,179,329\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e65,602,628\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12 encoder, 1 bottleneck, 12 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e41,250,691\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e11,179,073\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e52,429,764\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e43 encoder, 1 bottleneck, 43 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e48,885,315\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e19,314,433\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e68,199,748\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 encoder, 1 bottleneck, 7 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e82,736,899\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e11,180,353\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93,917,252\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eProposed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 encoder, 1 bottleneck, 10 decoder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e16,053,603\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e6,969,569\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e23,023,172\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003e\u003c/p\u003e\u0026nbsp;\u003ctable id=\"Tab3\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eEvaluation results of custom GANs on training set\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eModel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePSNR\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.85640\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e28.4696\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.88068\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e30.1880\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.75869\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e23.8032\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.76087\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e22.2019\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.87019\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e29.4226\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eProposed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.91115\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e31.7251\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003e\u003c/p\u003e\u0026nbsp;\u003ctable id=\"Tab4\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eEvaluation results of custom GANs on test set\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eModel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePSNR\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.83119\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e25.8888\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.84232\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e26.1952\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.74868\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e22.8974\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.75919\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e22.0925\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModel 5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.83647\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e25.9400\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eProposed\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.85752\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e24.9883\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cdiv id=\"Sec12\" class=\"Section3\"\u003e\n \u003ch2\u003e4.3.1 Limitations\u003c/h2\u003e\n \u003cp\u003eIn order to gauge the performance of the proposed GAN model, we conducted a qualitative assessment on images that were dissimilar to any of those encompassed within the training set. Within this evaluation, certain image samples underwent a process of facial area reconstruction by the proposed GAN model, resulting in distorted outcomes. These distortions could potentially be attributed to factors such as the network\u0026apos;s failure to facilitate gradient propagation across all layers. The detailed presentation of these distorted images generated by the proposed GAN model, juxtaposed against their corresponding ground truths, can be observed in Fig. \u003cspan class=\"InternalRef\"\u003e10\u003c/span\u003e. It is pertinent to note that despite the distortions, the generated images remain identifiable by human observers, predominantly due to the preservation of significant facial features.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec13\" class=\"Section3\"\u003e\n \u003ch2\u003e4.3.2 Face recognition challenge\u003c/h2\u003e\n \u003cp\u003eTo assess the capability of the proposed method in generating images that closely resemble actual facial images, we conducted experiments involving face recognition using edge devices compatible with deep learning-based classifiers, specifically MobileNet [\u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e] and NasNet [\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e]. This evaluation was carried out with the utilization of paired images: genuine facial images and synthetic facial images generated by the proposed GAN model.\u003c/p\u003e\n \u003cp\u003eFor the purpose of training and testing the classifiers, a dataset was meticulously curated, comprising 1000 real facial images and an equivalent count of plausible facial images. This dataset was divided into an 80:20 ratio for training and testing purposes. The performance assessment of the deep learning classifiers was carried out through metrics including Accuracy, Precision, Recall, and F-1 Score, as explicated by equations (6\u0026ndash;9). The outcomes of these performance evaluations for face recognition, employing deep learning classifiers, are comprehensively detailed in Table \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eAccuracy = \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{TP + FP}{TP + TN + FP + FN}\\)\u003c/span\u003e\u003c/span\u003e (6)\u003c/p\u003e\n \u003cp\u003ePrecision = \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{TP}{TP + FP}\\)\u003c/span\u003e\u003c/span\u003e (7)\u003c/p\u003e\n \u003cp\u003eRecall = \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{TP}{TP + FN}\\)\u003c/span\u003e\u003c/span\u003e (8)\u003c/p\u003e\n \u003cp\u003eF-1 Score = \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(2*\\frac{Precision * Recall}{Precision+Recall}\\)\u003c/span\u003e\u003c/span\u003e (9)\u0026nbsp;\u003c/p\u003e\u0026nbsp;\u003ctable id=\"Tab5\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eDeep learning classifiers results for facial recognition\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eClassifier\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eClass\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eF-1 Score\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eAccuracy\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eMobileNet\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eReal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e96%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e96%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" rowspan=\"2\"\u003e\n \u003cp\u003e96.25%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003ePlausible\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e94%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eNasNet\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eReal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e93%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e94%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e94%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" rowspan=\"2\"\u003e\n \u003cp\u003e94.50%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003ePlausible\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e94%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003eIn the context of the classification task aimed at distinguishing between authentic and credible faces, the MobilNet classifier showcased a precision of 96.25%, whereas the NasNet classifier achieved an accuracy of 94.50%. Moreover, impressive results were obtained across a spectrum of performance metrics. These findings emphasize that the synthetic faces produced by the suggested model exhibit discernible attributes, enabling them to be distinguished from genuine faces. As delineated in Table \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e, the obtained results affirm that the images produced by the proposed UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN-based Pix2Pix model exhibit accuracy and possess the potential to be utilized in face recognition systems for the purpose of discerning between genuine and fabricated faces.\u003c/p\u003e\n \u003cp\u003eTo evaluate the generalization of the proposed method, we tested the proposed method on test images of faces with masks extracted from the face mask detection (FMD) dataset [\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e] and evaluated for SSIM and PSNR metrics. The quantitative and qualitative results of the test are presented in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e and Fig. 11.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab6\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eQuantitative results on FMD dataset\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethod\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePSNR\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN (Proposed)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.90872\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e29.4723\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cp\u003eFor the test conducted to determine the generalization of the proposed method, an SSIM value of 0.90872 was achieved that indicates for random masked faces a significant structural similarity is retained by the proposed method. Further, for the PSNR metric, it achieved a value of 29.4723 that indicate a significant amount of signal in images is retained over noise thus, making the proposed method capable of reconstructing facial area behind masks with significant accuracy. Since, the proposed method works on the principle of image-to-image translation therefore, better results can be obtained in paired images with the availability of ground truths and masked faces. Furthermore, to gauge the validity of the proposed method in removing face masks from low resolution images, we trained the proposed method with a few images of faces with synthetic masks captured from distance and low-resolution camera and tested for its performance. The results for the distant and low-resolution images in removing and reconstructing the face area behind masks is presented in Fig. \u003cspan class=\"InternalRef\"\u003e12\u003c/span\u003e.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\n \u003ch2\u003e4.4 Comparison with related work\u003c/h2\u003e\n \u003cp\u003eTo conduct a thorough comparative analysis between the proposed method and recent contributions in the field, we subjected benchmark face mask removal techniques to training and testing using the curated dataset. The prevailing trends in face mask removal approaches predominantly involve the utilization of GANs, albeit with variations in architectural designs. The comprehensive breakdown of this comparative analysis is provided in Table \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\n \u003cdiv align=\"char\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\u0026nbsp;\u003ctable id=\"Tab7\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 7\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eComparative analysis with previous studies\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"4\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eWork\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethod\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSSIM\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePSNR\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUd Din et al. [\u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eGAN employing a single generator and dual discriminators\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.85237\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e25.3752\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eJiang et al. [\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSingle generator and discriminator-based GAN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.89981\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e28.9085\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFarahanipad et al. [\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCycle-GAN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.86314\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e26.1518\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOurs\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN-based Pix2Pix\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.91115\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e31.7251\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cp\u003eThe outcomes presented in Table \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e distinctly demonstrate that the proposed approach outperforms the GAN architectures outlined in the related studies, as evidenced by superior SSIM and PSNR metric scores. In particular, the method put forth demonstrates enhancements spanning from 1.2\u0026ndash;5.9% for SSIM and 2.82\u0026ndash;6.35% for PSNR metrics, when contrasted with the GAN architectures advocated in earlier studies. It\u0026apos;s worth highlighting that the technique advanced by Jiang et al. [\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e], which hinges on a single generator and discriminator, bears striking resemblance to the framework of the suggested approach. Nonetheless, the results continue to underscore the superiority of the proposed method in terms of structural similarity and signal preservation over noise, surpassing other GAN-based methodologies when evaluated on the dataset employed within this study.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"5. Conclusions and Future Work","content":"\u003cp\u003eGenerative Adversarial Networks (GANs) have emerged as a potent instrument for crafting realistic images, audio, and videos, with applications proliferating across various sectors like healthcare, design, and computer gaming. In this research, we present an innovative approach that amalgamates UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and CoGAN-based Pix2Pix GAN for the task of revealing concealed faces. The UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture is meticulously crafted using a fusion of convolution layers, convolution transpose layers, max pooling layers, and batch normalization layers, each equipped with distinct kernel sizes and strides. Our devised method not only surpasses several state-of-the-art models but also exhibits impressive proficiency in removing masks. The incorporation of a modified UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;model as the generator notably enhances the overall performance of our GAN, particularly reflected in elevated SSIM and PSNR scores. A striking attribute of our model is its adaptability in eliminating various mask types, transcending the constraints of specific training masks.\u003c/p\u003e \u003cp\u003eFurthermore, we have curated an original dataset tailored for paired image-to-image translation, which uniquely encompasses images of individuals within the criminal domain, augmenting the dataset's relevance to real-world scenarios. In the future, our research direction could focus on harnessing the proposed GAN model for tasks like identity recognition and preservation. This might involve the utilization of models such as StyleGAN to infuse distinctive aesthetic characteristics into generated facial images, thereby broadening the horizon of potential applications.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e1. The original Bollywood Celebrity Face dataset is available at: \u003cu\u003ehttps://www.kaggle.com/datasets/havingfun/100-bollywood-celebrity-faces.\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003e2. The created dataset and code of the UNET++ and CoGAN method is available on request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors have no competing interests to disclose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors express their gratitude to the All India Council of Technical Education for providing financial support for this study. This research has been sponsored under the Research Promotion Scheme of AICTE, India, with reference file number 8-108/FDC/RPS(POLICY-1/2019-20).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contribution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAkhil Kumar (AK) contributed to methodology design, implementation, formal analysis and writing and editing original and final manuscript. Divyam Gupta (DG) contributed to methodology design, implementation and formal analysis. Manisha Kaushal (MK) contributed to editing and reviewing original and final draft of the manuscript. Akashdeep Sharma (AS) supervised the entire work with additional responsibility of conceptualization and methodology design.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eGoodfellow IJ, Pouget-Abadie J, Mirza M, Xu, B, Warde-Farley D, Ozair S, Courville Bengio Y (2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems, (Neural information processing systems foundation), pp. 2672\u0026ndash;2680. https://doi.org/10.1007/978-3-658-40442-0_9.\u003c/li\u003e\n\u003cli\u003eMirza M, Osindero S (2018) Conditional Generative Adversarial Nets. In arXiv. http://arxiv.org/abs/1411.1784.\u003c/li\u003e\n\u003cli\u003eZhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision (Vol. 2017-October, pp. 2242\u0026ndash;2251. https://doi.org/10.1109/ICCV.2017.244.\u003c/li\u003e\n\u003cli\u003eZhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas D (2017) StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5908\u0026ndash;5916. https://doi.org/10.1109/ICCV.2017.629.\u003c/li\u003e\n\u003cli\u003eIsola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition. pp. 5967\u0026ndash;5976. https://doi.org/10.1109/CVPR.2017.632.\u003c/li\u003e\n\u003cli\u003eYu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative Image Inpainting with Contextual Attention. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 5505\u0026ndash;5514. https://doi.org/10.1109/CVPR.2018.00577.\u003c/li\u003e\n\u003cli\u003eDeng J, Dong W, Socher R, Li L-J, Kai Li, Li Fei-Fei (2010) ImageNet: A large-scale hierarchical image database. https://doi.org/10.1109/cvpr.2009.5206848.\u003c/li\u003e\n\u003cli\u003eNazeri K, Ng E, Joseph T, Qureshi F, Ebrahimi M (2019) Edgeconnect: Structure guided image inpainting using edge prediction. In: Proceedings - 2019 International Conference on Computer Vision Workshop. pp. 3265\u0026ndash;3274. https://doi.org/10.1109/ICCVW.2019.00408.\u003c/li\u003e\n\u003cli\u003eDoersch C, Singh S, Gupta A, Sivic J, Efros A (2015) What makes Paris look like Paris? Commun. ACM 58(12):103\u0026ndash;110. https://doi.org/10.1145/2830541.\u003c/li\u003e\n\u003cli\u003eKhan MKJ, Ud Din N, Bae S, Yi J (2019) Interactive removal of microphone object in facial images. Electronics 8(10). https://doi.org/10.3390/electronics8101115.\u003c/li\u003e\n\u003cli\u003eUd Din N, Javed K, Bae S, Yi J (2020) A Novel GAN-Based Network for Unmasking of Masked Face. IEEE Access. 8:44276\u0026ndash;44287. https://doi.org/10.1109/ACCESS.2020.2977386.\u003c/li\u003e\n\u003cli\u003eRonneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Lecture Notes in Computer Science. pp. 234\u0026ndash;241. Springer Verlag. https://doi.org/10.1007/978-3-319-24574-4_28.\u003c/li\u003e\n\u003cli\u003eJiang Y, Yang F, Bian Z, Lu C, Xia S (2022) Mask removal : Face inpainting via attributes. Multimedia Tools Appl. 81(21):29785\u0026ndash;29797. https://doi.org/10.1007/s11042-022-12912-1.\u003c/li\u003e\n\u003cli\u003eFarahanipad F, Rezaei M, Nasr M, Kamangar F, Athitsos V (2022) GAN-based Face Reconstruction for Masked-Face. In: ACM International Conference Proceeding Series. pp. 583\u0026ndash;587. https://doi.org/10.1145/3529190.3534774.\u003c/li\u003e\n\u003cli\u003eAnwar A, Raychowdhury A (2020) Masked Face Recognition for Secure Authentication. In arXiv. https://arxiv.org/abs/2008.11104.\u003c/li\u003e\n\u003cli\u003eRedmon J, Farhadi, A (2018) YOLO v3.0: An Incremental Improvement. In arXiv. https://doi.org/10.48550/arXiv.1804.02767.\u003c/li\u003e\n\u003cli\u003eZhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2242\u0026ndash;2251. https://doi.org/10.1109/ICCV.2017.244.\u003c/li\u003e\n\u003cli\u003eHamzah Ordiyasa IW, Najib MHR. (2023) DeepMask: face mask detection using GAN algorithm. Iran J.Comput. Sci. 6(1): 13\u0026ndash;19. https://doi.org/10.1007/s42044-022-00114-9.\u003c/li\u003e\n\u003cli\u003eYe L (2022) Mask Removal Algorithm Using GAN Based Models. In: Proceedings - 2022 3rd International Conference on Electronic Communication and Artificial Intelligence. pp. 281\u0026ndash;285. https://doi.org/10.1109/IWECAI55315.2022.00060.\u003c/li\u003e\n\u003cli\u003eBollywood Celebrity Faces Dataset (2020) https://www.kaggle.com/datasets/havingfun/100-bollywood-celebrity-faces. Accessed 20 May 2023.\u003c/li\u003e\n\u003cli\u003eKumar A, Kaushal M, Sharma A (2023) SAM C-GAN: a method for removal of face masks from masked faces. Signal, Image Video Process. https://doi.org/10.1007/s11760-023-02602-2.\u003c/li\u003e\n\u003cli\u003eKumar A, Kalia A, Sharma A, Kaushal M (2023) A hybrid tiny YOLO v4-SPP module based improved face mask detection vision system. J. Ambient Intell. Hum. Comput. 14(6):6783\u0026ndash;6796. https://doi.org/10.1007/s12652-021-03541-x.\u003c/li\u003e\n\u003cli\u003eKumar A, Kalia A, Verma K, Sharma A, Kaushal M (2021) Scaling up face masks detection with YOLO on a novel dataset. Optik, 239. https://doi.org/10.1016/j.ijleo.2021.166744.\u003c/li\u003e\n\u003cli\u003eOpenCV DNN Module. https://docs.opencv.org. Accessed 10 June 2023\u003c/li\u003e\n\u003cli\u003eZhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Lecture Notes in Computer Science. pp. 3\u0026ndash;11. Springer Verlag. https://doi.org/10.1007/978-3-030-00889-5_1.\u003c/li\u003e\n\u003cli\u003eHoward AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. In arXiv. https://arxiv.org/abs/1704.04861.\u003c/li\u003e\n\u003cli\u003eZoph B, Vasudevan V, Shlens J, Le QV (2018) Learning Transferable Architectures for Scalable Image Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 8697\u0026ndash;8710. https://doi.org/10.1109/CVPR.2018.00907.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"CoGAN, Pix2Pix, UNET++, PatchGAN, image to image translation","lastPublishedDoi":"10.21203/rs.3.rs-3351025/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3351025/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eImage-to-image translation emerges as a significant utility of conditional Generative Adversarial Networks (CoGANs). This research introduces a fresh application of conditional GANs, aiming to uncover hidden facial attributes. Our methodology involves enhancing the Pix2Pix GAN framework through the integration of a modified UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture, which serves as an inventive generator model. In this setup, the Pix2Pix model employs a PatchGAN architecture within the discriminator, producing an activation map with values utilized to authenticate depicted faces. Through the incorporation of the UNET\u0026thinsp;+\u0026thinsp;+\u0026thinsp;architecture into the generator, we effectively narrow down the semantic gap between the encoder and decoder feature maps. This strategic adjustment results in a noticeable enhancement in gradient flow. To gauge the effectiveness of our proposed approach, we conducted experiments on a bespoke dataset intentionally crafted for training paired image-to-image translation GANs. Our model is comprehensively compared against other leading models designed for revealing concealed facial features. Significantly, our proposed model convincingly surpasses these alternatives across a range of evaluation criteria.\u003c/p\u003e","manuscriptTitle":"A UNET++ and CoGAN-based method to remove face masks from the masked faces","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-09-18 15:54:41","doi":"10.21203/rs.3.rs-3351025/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3f05e750-f3b4-4efc-8b93-7935408950ba","owner":[],"postedDate":"September 18th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-10-04T17:44:26+00:00","versionOfRecord":[],"versionCreatedAt":"2023-09-18 15:54:41","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3351025","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3351025","identity":"rs-3351025","version":["v1"]},"buildId":"-HB7Z8yhvgn0wM9Nzuekk","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}