Constrained Convolutional Neural Network Models for Optimizing Fully Connected Layer Weights in CNNs

Abstract This study proposes constrained convolutional neural network models for determining the initial connection weights in the Fully Connected Network (FCN) layer within the Convolutional Neural Network (CNN) model, resulting in an increase in the CNN model's performance. A literature review indicates that the constrained method is used in conjunction with CNN. However, previous studies have typically focused on using the constrained method before feature selection in CNN. In contrast, this study aims to calculate the initial values of the connection weights, one of the hyperparameters in the FCN, by using the constrained method between feature selection and the FCN layer. Five different models are proposed: the Constrained Difference CNN (D-CNN), the Sample Constrained CNN (C-CNN), the Constrained Sum CNN (S-CNN), the Random Sum CNN (RS-CNN), and the Constrained Mixed CNN (M-CNN). These proposed models and classical CNN, have been applied to the MNIST, MNIST Fashion, and CIFAR-10 datasets then the results have been examined. According to the average accuracy results, the C-CNN model achieved the highest performance in the MNIST dataset with an accuracy rate of 99.03\%. In the MNIST Fashion dataset, the best result was obtained by the D-CNN model with an accuracy rate of 91.80\%. Similarly, the D-CNN model achieved the highest performance in the CIFAR-10 dataset with an accuracy rate of 71.44\%. D-CNN and C-CNN models have outperformed the other proposed models and the classical CNN. The reason for the better performance of these two methods is considered to be their calculation based on the differential operation.