LapEfficientDepth: lightweight model for monocular depth estimation based on small samples

LapEfficientDepth: lightweight model for monocular depth estimation based on small samples | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article LapEfficientDepth: lightweight model for monocular depth estimation based on small samples Guangyuan Zhang, Hongliang Guan, Kefeng Li, Zhenfei Wang, Peng Wang, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3998413/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This study proposes an improved LapDepth model, named LapEfficientDepth, for the monocular depth estimation field. The objective is to reduce the error in model-predicted relative depths and enhance depth estimation accuracy, addressing the issues of substantial resource consumption and large parameter count inherent in the original model. By incorporating lightweight modules, the LapEfficientDepth model significantly reduces model the complexity and resource requirements of the model while maintaining high accuracy estimation. Specifically, the parameter count of the LapEfficientDepth model has been reduced to 6M, constituting only 8.2% of the parameter volume found in the original LapDepth model, while achieving an approximate 1% improvement in accuracy compared to the Lite-Mono-8M model, which has a similar number of parameters. In addition, the LapEfficientDepth model exhibits exceptional transfer learning capabilities. After pre-training on the KITTI dataset and further training on the ETH3D-S dataset, the model achieved a1, a2, and a3 metrics of 0.706, 0.997, and 0.999, respectively, proving its rapid adaptability and learning ability on small sample datasets. This offers an effective solution for high-performance, lightweight monocular depth estimation network models. Monocular depth estimation Lightweight Small samples Transfer learning Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Monocular depth estimation refers to the process of estimating the depth information of various objects in a scene using images captured by a single camera 1 . This technique is critically important in the fields of computer vision and robotics, as it provides a cost-effective and flexible solution for applications such as environmental perception, 3D reconstruction, autonomous driving, and augmented reality. In recent years, the development of deep learning technologies has greatly advanced research in monocular depth estimation 2 . Methods based on Convolutional Neural Networks (CNNs) have become the mainstream approach, These methods typically rely on supervised learning with a large volume of labeled depth maps to achieve accurate depth estimation 3,4 . Currently, datasets for monocular depth estimation are somewhat limited, with the most common datasets including the KITTI dataset 5 , jointly created by the Karlsruhe Institute of Technology in Germany and the Toyota Technological Institute at Chicago, and the NYU dataset 6 , developed by the Department of Computer Science at New York University. The KITTI dataset is primarily used for research in visual perception for mobile robots and autonomous vehicles, while the NYU dataset supports depth estimation and 3D reconstruction research in indoor environments. Establishing datasets for depth estimation, especially those of high quality and large scale, involves significant hardware and annotation costs. Collecting depth data requires specialized hardware equipment, such as LiDAR (Light Detection and Ranging) and high-precision depth cameras, the cost of which can range from several thousand to tens of thousands of dollars. Furthermore, depth datasets require not only the collection of raw data but also precise annotation and processing, including depth calibration and the registration of image and depth data. This process is often labor-intensive, requiring extensive expertise, thereby incurring high labor costs. In recent years, few-shot learning has emerged as an important research direction in the field of deep learning, aiming to design more intelligent algorithms that enable models to learn effective feature representations from limited data, thereby reducing the reliance on large-scale annotated datasets 7,8 . This is particularly important for depth estimation, as it addresses the issue of acquiring and annotating high-quality depth data, which is both time-consuming and expensive, to some extent. Evaluating the performance of a monocular depth estimation model primarily involves analyzing metrics from two perspectives 9,10,11 . One is the analysis of error and accuracy from the perspective of the model's ability to reconstruct depth maps. The other is the analysis of the model's size in terms of the number of parameters 12,13,14 . Since the introduction of the ResNet model in 2015, convolutional neural networks have been able to improve model accuracy by stacking more layers 15 . The introduction of the ResNet model has also greatly aided in reducing the error and improving the accuracy of monocular depth estimation models. Since then, the quality of depth maps generated by monocular depth estimation models has reached a higher standard. MonoDepth2 model proposed by Godard C, et al., which uses ResNet50 as the feature extraction network, achieved an absolute error of 0.115 on the KITTI dataset, with metrics of a1( ), a2( 2 ), and a3( 3 ) at 0.879, 0.961, and 0.982, respectively 16 . Song and others introduced the LapDepth monocular depth estimation model, which achieved outstanding evaluation metrics on the KITTI dataset, with the error reduced to 0.059 and the a1 metric improved to 0.962, reaching the highest metrics among similar models at the time 17 . LapDepth utilizes the ResNext101 feature extraction network. The ResNext network improves performance and efficiency by introducing "grouped convolution". The ResNeXt models are designed based on the basic framework of ResNet, incorporating a new dimension called "cardinality", which refers to the number of parallel paths within each residual block 18 . Through this approach, ResNeXt can enhance the model's representational capability without significantly increasing computational complexity. The introduction of the MobileNet lightweight model 19 began to drive the development of lightweight monocular depth estimation models. Wofk D, et al. used MobileNet as the feature extraction network for the FastDepth 20 , resulting in the MobileNet-NNConv5 model with a parameter count of 20.6M. The latest monocular depth estimation model, Lite-Mono-8M 21 , has reduced its total parameter count to 8.7M, with an a1of 0.897, achieving a good balance between model lightweighting and depth estimation accuracy. Research on models for monocular depth estimation based on small samples remains a topic area with significant research potential and practical value 22,23,24 . In previous work 25 , by studying the characteristics and differences between various convolutional neural network models, we designed a lightweight model, the Res-Efficient model, which achieved good results in pest and disease recognition 26 . Further experimental validation of this model revealed that it performs excellently in identifying the main features in images within specific scenarios. This characteristic is particularly suitable for small-sample learning in specific contexts. This paper builds on the in-depth study of this model and combines experience from the field of monocular depth estimation to explore lightweight models for small-sample monocular depth estimation, proposing the LapEfficientDepth model. The main contributions of this study are as follows: 1. The LapEfficientDepth model, a novel lightweight model that can produce high-quality depth maps after being trained on small sample datasets, is proposed. 2. The model's generalization ability was tested using transfer learning. Experiments demonstrate that the proposed model has strong generalization capabilities, with a slight improvement in error and accuracy after transfer learning of the pre-trained model. 3. The proposed model is high-performing. On the KITTI dataset, compared to currently known monocular depth estimation models with parameter counts under 10M, the LapEfficientDepth model has the smallest error and the highest accuracy. Experiments Experimental Environment and Datasets The experimental environment for this study was conducted on Tencent Cloud's HAI high-performance server, with a GPU base type computing power, equipped with an Nvidia T4 16GB graphics card, 32GB of memory, and configured with Ubuntu 20.04, Python 3.8, ChatGLM2-6b, CUDA 11.7, cuDNN 8, and PyTorch 2. The datasets used in the experiments include the KITTI dataset and the ETH3D dataset 27 . The division of the KITTI dataset was constructed following the experimental setup of MonoDepth2. The KITTI dataset serves two purposes in this study: one is to evaluate the performance of the LapEfficientDepth model, and the other is as the dataset for the pre-trained model. The ETH3D-S dataset was divided into a training set and a test set at a 9:1 ratio. The details of the dataset sample numbers are shown in Table 1. The ETH3D dataset is a high-quality 3D reconstruction dataset developed and maintained by ETH Zurich, Switzerland. It aims to provide a rich testing and evaluation platform for research into stereo vision and multi-view stereo algorithms. The ETH3D-S dataset used in this paper is selected from the ETH3D dataset, consisting of data from 3 classic scenes, serving as a small-sample dataset for specific scenarios. The sample numbers for the 3 scenes are shown in Table 2. Model The LapEfficientDepth model proposed in this article modifies the feature extraction network of the LapDepth network 11 , employing a feature extraction network composed of FusedMBConv modules, MBConv modules 28 , and residual blocks, replacing the original ResNext101 feature extraction network in the LapDepth model to form a new monocular depth estimation network structure. The network structure of the LapEfficientDepth monocular depth estimation model is illustrated in Figure 1. LapEfficientDepth adopts an encoder-decoder structure. The encoding layer of the model is the modified feature extraction network. In Figure 1, 'S' represents the size of the image input into the model. As the image passes through the feature extraction network, it undergoes four downsampling operations, resulting in a feature map that is 1/16th the size of the original image dimensions. The encoding layer of the LapEfficientDepth model features two types of convolutional blocks: FusedMBConv and MBConv. The FusedMBConv layer is constructed from two FusedMBConv modules connected in series, with strides of 2 and 1, respectively, to effectively capture and downsample features from the input image. Similarly, the MBConv layer consists of two MBConv modules in series, also with strides of 2 and 1, enhancing the model's ability to extract detailed features while reducing spatial dimensions. This paper introduces two variants of the monocular depth estimation model, differentiated by the parameters of their decoding layers: LapEfficientDepth-m and LapEfficientDepth-s. The LapEfficientDepth-m model's encoding layer module parameters are detailed in Table 3, designed to offer a balance between performance and computational efficiency. On the other hand, the LapEfficientDepth-s model simplifies the architecture by setting the repeat parameter of all modules to 1, aiming for a more compact and faster model suitable for environments where computational resources are limited or where rapid depth estimation is required. This distinction allows for flexible application across different scenarios, catering to the specific needs of the task at hand. The decoding layer first utilizes an Atrous Spatial Pyramid Pooling (ASPP) layer to achieve a larger receptive field without significantly losing resolution 29 . Subsequent operations involve concatenation and connection at the same scale to produce corresponding scale feature maps and depth maps. In Figure 1, represents the generated depth residual feature maps, while denotes the difference feature maps of the corresponding scale color images generated by Laplacian residuals, as illustrated in Figure 2. The generation of difference feature maps enriches texture and feature information through concatenation and connection operations with feature maps of the same scale in the decoding layer, resulting in corresponding scale depth residual feature maps. Finally, the corresponding scale residual feature maps are added to the feature maps obtained from the previous layer's upsampling, enhancing object boundaries' clarity and ultimately generating depth maps of the same scale. This process intricately blends various techniques to refine the depth estimation, emphasizing the model's capability to preserve detail and enhance the accuracy of the depth maps produced. Experimental Design This paper focuses on exploring the performance of the LapEfficientDepth model, which uses lightweight modules to construct the feature extraction layers of the network. LapEfficientDepth is designed in medium and small versions.The experiments will evaluate the model's performance from two aspects. Firstly, the performance of the LapEfficientDepth model will be assessed based on the evaluation metrics obtained from a pre-trained model on KITTI, a publicly available large-scale depth estimation dataset. Secondly, the evaluation will consider the metrics and the quality of depth maps generated from small-sample transfer learning based on the pre-trained model. The design of comparative experiments for LapEfficientDepth is outlined in Table 4. Where represents the model's predicted output values, is the true value of the image depth, and is the total number of pixels in the depth image. Additional parameters used in the experiment are detailed in Table 5. Results and Analysis Evaluation of the LapEfficientDepth model on the KITTI dataset The experiment began with training on the KITTI dataset, where both LapEfficientDepth-m and LapEfficientDepth-s models underwent 20 training epochs to obtain their respective pre-trained versions. Upon validation with the KITTI test set, the performance metrics obtained were compared with those of other models. The comparative results are showcased in Table 6. From Table 6, it is evident that ResNext101 has the highest number of parameters, indicating it is the most complex model. In contrast, LapEfficientDepth_s has the fewest parameters. The number of parameters reflects the model's complexity, implying that LapEfficientDepth_s might be more suitable as a monocular depth estimation model in resource-constrained situations. The absolute relative error measures the accuracy of the model's relative depth predictions; the lower the absolute relative error, the more accurate the model's predictions. This comparison underscores the trade-off between model complexity and performance, where lightweight models like LapEfficientDepth_s aim to achieve competitive accuracy with significantly reduced computational resources. In terms of the absolute relative error (abs rel) metric, the LapDepth model utilizing ResNext101 and ConvNeXt_tiny 30 as feature extraction networks performed the best, while Lite-Mono-8M showed the poorest performance. LapEfficientDepth_m and LapEfficientDepth_s did not perform as well as ResNext101 and ConvNeXt_tiny on this metric but were superior to Lite-Mono-8M. Regarding the accuracy thresholds a1, a2, and a3, which measure the consistency between the predicted and actual depth values, the closer the values are to 1, the more accurate the model's predictions. Across these three accuracy metrics, all models performed quite well, especially on a3, where, except for Lite-Mono-8M, all other models reached or exceeded 0.995. The LapEfficientDepth models were slightly below the top three models in a1 and a2 but still demonstrated high accuracy. Root Mean Square Error (RMSE) and Root Mean Square Logarithmic Error (RMSE Log) measure the magnitude of the prediction error, with lower values indicating smaller errors and more accurate predictions. ResNext101 and ConvNeXt_tiny had the best performance on the RMSE metric, while Lite-Mono_8M had the worst. The performance of LapEfficientDepth_m and LapEfficientDepth_s on these metrics was intermediate compared to the other models. Overall, the main advantages of the LapEfficientDepth_m and LapEfficientDepth_s models compared to other models lie in their low number of parameters, implying they may be more suitable in situations with limited computational resources. Although their performance in absolute relative error, accuracy thresholds, as well as RMSE and RMSE Log metrics, may not be the best, considering their smaller model size, these results are still quite impressive. Especially for applications that need to run on edge devices or in scenarios requiring high efficiency under computational resource constraints, the LapEfficientDepth models could be a more optimal choice. This balance between model complexity, efficiency, and performance underscores the significance of lightweight models in advancing depth estimation tasks, particularly in resource-limited environments. Transfer learning of LapEfficientDepth in small sample data sets The transfer learning experiments in this paper involved applying the LapEfficientDepth pre-trained model, obtained from the KITTI dataset, to the ETH3D-S small-sample dataset for full-parameter transfer learning. Table 7 would provide a performance comparison of the LapEfficientDepth model on the ETH3D-S dataset, including two different variants (medium and small), as well as their results after optimization through transfer learning. The models utilizing transfer learning, LapEfficientDepth-m-transfer and LapEfficientDepth-s-transfer, demonstrated lower errors on the absolute relative error (abs_rel) and log10 metrics, indicating that transfer learning contributes to enhancing the model's prediction accuracy. Notably, LapEfficientDepth-m-transfer achieved the lowest abs_rel at 0.222 among the four configurations, indicating the smallest average absolute error in predictions. In terms of accuracy thresholds, the transfer learning models performed better on a1, especially LapEfficientDepth-m-transfer, which reached 0.706 on a1. This means that over 70% of the predictions had an error smaller than 25%. All models performed very well on a2 and a3, nearing or achieving perfection. Regarding the RMSE and RMSE Log metrics, the models optimized with transfer learning also showed superior performance, with LapEfficientDepth-m-transfer having the lowest RMSE at 1.28 among the four configurations, indicating the smallest average error in predictions. Transfer learning also helped reduce the root mean square logarithmic error, with LapEfficientDepth-m-transfer and LapEfficientDepth-s-transfer achieving 0.207 and 0.209, respectively. In summary, the application of transfer learning on the LapEfficientDepth model significantly improved its performance on the ETH3D-S dataset, particularly in terms of absolute relative error, accuracy thresholds, and root mean square error. This demonstrates that pre-training a model on related tasks and then fine-tuning it for specific tasks (i.e., transfer learning) can effectively enhance the model's accuracy and generalization ability. This approach leverages the knowledge gained from a comprehensive dataset to improve performance in more specialized scenarios, illustrating the practical value of transfer learning in enhancing deep learning models' adaptability and efficiency across various tasks. Training loss curve analysis Training loss curves for monocular depth estimation models can reflect the trend of the model's improvement over time on the training data. Typically, these curves show a downward trend, indicating that as training progresses, the model's performance on the training set improves, meaning the error decreases. Figure 3 would showcase the training loss curves for two different configurations of the LapEfficientDepth model. Each configuration includes two versions: one without the use of transfer learning and one with the application of transfer learning. From the upper two graphs in Figure 3, it is observed that without the use of transfer learning, the LapEfficientDepth-m model starts with a relatively high loss that gradually decreases with an increase in the number of iterations. Initially, the loss decreases rapidly, indicating effective learning by the model during this phase. However, as the number of iterations increases, the rate of loss decrease slows down, typically indicating that the model is beginning to saturate, and the amount of new information learned from the data diminishes. With the application of transfer learning, as shown in the lower two graphs, both the LapEfficientDepth-m and LapEfficientDepth-s models start from a lower value of loss. This demonstrates that transfer learning enables the models to perform better at the onset of training. More importantly, the loss curves show a very rapid decline initially, implying that the models are able to quickly adapt to the new dataset. This is especially evident for the LapEfficientDepth-m model, where the training loss rapidly reduces to a very low level after applying transfer learning. LapEfficientDepth depth map effect and analysis To investigate the generalization ability of the LapEfficientDepth monocular depth estimation model, as well as the effectiveness of transfer learning on a small-sample dataset and the visual quality of the generated depth maps, the experiment compares depth maps generated by the LapEfficientDepth model on the ETH3D-S small-sample test set. The comparison images are presented in Figure 4. Figure 4, showcasing three different scenes with a real label and four different depth map results for each, demonstrates the capabilities of the LapEfficientDepth model variants before and after applying transfer learning. Each row represents a scene, and each column corresponds to a depth map generated by a different model configuration. Observations from Figure 4 indicate that depth maps produced by the LapEfficientDepth-s and LapEfficientDepth-m models estimate broad changes in depth well, with the depth maps appearing smooth and the color gradients soft. However, these models sometimes handle edge regions less delicately. The depth maps generated by the LapEfficientDepth-s-transfer and LapEfficientDepth-m-transfer models, which have undergone transfer learning training, show improved detail retention while smoothing the depth field as much as possible. This improvement is especially noticeable in areas with good depth continuity, and these models reveal more details, particularly near edges. The depth maps displayed in Figure 4 reveal the positive impact of transfer learning on the accuracy of depth estimation. The LapEfficientDepth models that have undergone transfer learning seem to perform better in maintaining details and edges than their original versions. This suggests that the LapEfficientDepth models are capable of performing monocular depth estimation tasks in specific scenarios through transfer learning. These models can recognize common edges and shapes using already learned features and adapt to scenarios that were uncommon or entirely unseen during the training process. Such adaptability brings significant value to applications, highlighting the potential of transfer learning to enhance model performance across diverse and challenging environments. Conclusion This paper explores lightweight models for monocular depth estimation in small-sample scenarios and designs the LapEfficientDepth lightweight monocular depth estimation model based on the LapDepth model. Two versions of the model were developed: LapEfficientDepth-m and LapEfficientDepth-s, with parameter counts of 17.6M and 6.0M, respectively. After training on the KITTI dataset, their performance on the KITTI test set achieves an a1 metric of approximately 0.9 and an a3 metric of 0.995 and above, indicating a high consistency between the predicted and actual depth values of the LapEfficientDepth model. On the ETH3D-S small-sample dataset, the LapEfficientDepth also demonstrated exceptional performance. Analysis of experimental results shows that the LapEfficientDepth model, when trained from scratch on a small-sample dataset, can achieve a depth estimation model with low relative error and high precision. After applying transfer learning, the model effectively transfers knowledge from the pre-trained model, resulting in rapid loss reduction and faster convergence. Since the computational part of the pre-trained model can be reused, transfer learning not only enhances the model's performance on small-sample datasets but also significantly reduces the time and resource consumption needed for model training. The LapEfficientDepth model is capable of generating depth maps with clear subjects and soft color gradients, estimating broad changes in the overall depth of images. Transfer learning has a positive impact on the generation of depth maps, with the LapEfficientDepth model showing significant improvement, especially in edge retention and detail depiction. This indicates that the model has learned to extract useful features from larger datasets and successfully adapt these features to new, smaller datasets. The proposed LapEfficientDepth-m monocular depth estimation model serves as a lightweight foundational model capable of handling tasks that involve training on small-sample datasets. Through transfer learning, the model's performance on small-sample datasets is enhanced, reducing dependency on the quantity of samples. The other model, LapEfficientDepth-s, is an even more lightweight monocular depth estimation model. Combined with transfer learning, its performance can reach or even exceed that of the LapEfficientDepth-m model trained from scratch. This makes the LapEfficientDepth-s model broadly deployable on lightweight devices and mobile platforms. References Zhao C, Sun Q, Zhang C, et al. Monocular depth estimation based on deep learning: An overview[J]. Science China Technological Sciences, 63(9), 1612-1627 (2020). Lei C, Zhengyou L, Yu S. A monocular image depth estimation method based on weighted fusion and point‐wise convolution[J]. IET Computer Vision (2023). Khan F, Hussain S, Basak S, et al. An efficient encoder–decoder model for portrait depth estimation from single images trained on pixel-accurate synthetic data[J]. Neural Networks, 142, 479-491(2021). Liu Y, Jin M, Pan S, et al. Graph self-supervised learning: A survey[J]. IEEE Transactions on Knowledge and Data Engineering, 35(6), 5879-5900 (2022). Geiger A, Lenz P, Stiller C, et al. Vision meets robotics: The kitti dataset[J]. The International Journal of Robotics Research, 32(11), 1231-1237 (2013). Couprie C, Farabet C, Najman L, et al. Indoor semantic segmentation using depth information[J]. arXiv preprint arXiv, 1301.3572 (2013). Song Y, Wang T, Cai P, et al. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities[J], ACM Computing Surveys (2023). Wang Y, Yao Q, Kwok J T, et al. Generalizing from a few examples: A survey on few-shot learning[J]. ACM computing surveys (csur), 53(3), 1-34 (2020). Fang Z, Chen X, Chen Y, et al. Towards good practice for CNN-based monocular depth estimation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1091-1100 (2020). Petrovai A, Nedevschi S. Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1578-1588 (2022). Li Z, Yu L, Pan Z. A Monocular SLAM System Based on ResNet Depth Estimation[J], IEEE Sensors Journal, (2023). Dong X, Garratt M A, Anavatti S G, et al. Mobilexnet: An efficient convolutional neural network for monocular depth estimation[J], IEEE Transactions on Intelligent Transportation Systems, 23(11), 20134-20147 (2022). Yucel M K, Dimaridou V, Drosou A, et al. Real-time monocular depth estimation with sparse supervision on mobile[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2428-2437 (2021). Pinasthika K, Utaminingrum F, Lin C Y, et al. Deciphering pixel insights: A deep dive into deep learning strategies for enhanced indoor depth estimation[J], International Journal of Information Management Data Insights, 4(1), 100216 (2024). He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778 (2016). Godard C, Mac Aodha O, Firman M, et al. Digging into self-supervised monocular depth estimation[C]//Proceedings of the IEEE/CVF international conference on computer vision, 3828-3838 (2019). Song M, Lim S, Kim W. Monocular depth estimation using laplacian pyramid-based depth residuals[J]. IEEE transactions on circuits and systems for video technology, 31(11), 4381-4393 (2021). Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 1492-1500 (2017). Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, (2017). Wofk D, Ma F, Yang T J, et al. Fastdepth: Fast monocular depth estimation on embedded systems[C]//2019 International Conference on Robotics and Automation (ICRA). IEEE, 6101-6108 (2019). Zhang N, Nex F, Vosselman G, et al. Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18537-18546 (2023). Zhao W. Research on the deep learning of the small sample data based on transfer learning[C]//AIP conference proceedings. AIP Publishing, 1864(1) (2017). Kokol P, Kokol M, Zagoranski S. Machine learning on small size samples: A synthetic knowledge synthesis[J]. Science Progress, 105(1): 00368504211029777.18 (2022) Safonova A, Ghazaryan G, Stiller S, et al. Ten deep learning techniques to address small data problems with remote sensing[J]. International Journal of Applied Earth Observation and Geoinformation, 125: 103569 (2023). Ming Y, Meng X, Fan C, et al. Deep learning for monocular depth estimation: A review[J]. Neurocomputing, 438, 14-33 (2021). Guan H, Fu C, Zhang G, et al. A lightweight model for efficient identification of plant diseases and pests based on deep learning[J]. Frontiers in Plant Science, 14 (2023). Schops T, Schonberger J L, Galliani S, et al. A multi-view stereo benchmark with high-resolution images and multi-camera videos[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 3260-3269 (2017). Tan M, Le Q. Efficientnetv2: Smaller models and faster training[C]//International conference on machine learning, PMLR, 10096-10106 (2021). He K, Zhang X, Ren S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 37(9), 1904-1916 (2015). Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11976-11986 (2022). Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3998413","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":278201027,"identity":"1e234f0b-7d49-47a0-a102-c289602f3a02","order_by":0,"name":"Guangyuan Zhang","email":"","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Guangyuan","middleName":"","lastName":"Zhang","suffix":""},{"id":278201028,"identity":"3475d31d-c8c3-4e7a-89d5-0323c1e843ea","order_by":1,"name":"Hongliang Guan","email":"","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Hongliang","middleName":"","lastName":"Guan","suffix":""},{"id":278201029,"identity":"5d5f1a7a-e6a8-408d-a814-51c7fa302467","order_by":2,"name":"Kefeng Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6ElEQVRIie3RMWsCMRTA8RcCuSV66+vgd4gciGDpfZUEQZfq3MEh03U56Krfoh8hErgpreuBi+IX6OhQihe7dIoZC+a/Bd6PxyMAqdQ/LKf5wX6fHwGM8W96mzy81uTQq2cdsZFEOEcFZ924oZEE2kWDyHdlf7trEF4mSmcfJijIejlDMd6rjbEMwc2V5ksZJBTlCCXfS+EJqazSyEWQME8M+yx/yU8E4dwVQ80Meb8SHUEwq9SJ1FN/SzGWzbyo+HOYlJZ2H3J+Kvvt9th+rSaDt8yFyZ99BkD66yLnu3IdP5tKpVL31QVbCUmZVQYSrwAAAABJRU5ErkJggg==","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":true,"prefix":"","firstName":"Kefeng","middleName":"","lastName":"Li","suffix":""},{"id":278201030,"identity":"a11c1b39-ef2d-48bc-b842-ebe8142902b4","order_by":3,"name":"Zhenfei Wang","email":"","orcid":"","institution":"Shandong Zhengyuan Yeda Environmental Technology Co., Ltd","correspondingAuthor":false,"prefix":"","firstName":"Zhenfei","middleName":"","lastName":"Wang","suffix":""},{"id":278201031,"identity":"1f6080b0-f52a-455c-b691-eb56f1d4ab80","order_by":4,"name":"Peng Wang","email":"","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Peng","middleName":"","lastName":"Wang","suffix":""},{"id":278201032,"identity":"eaae8457-2d99-42dc-aac8-d2eeb6ac254c","order_by":5,"name":"Chen Fu","email":"","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Chen","middleName":"","lastName":"Fu","suffix":""},{"id":278201033,"identity":"3f3b0aab-11d9-41d6-8c3b-1e36f9b705c7","order_by":6,"name":"Zhenfang Zhu","email":"","orcid":"","institution":"Shandong Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Zhenfang","middleName":"","lastName":"Zhu","suffix":""}],"badges":[],"createdAt":"2024-02-29 03:46:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3998413/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3998413/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":52516184,"identity":"c8c0c3b9-15e1-465c-975f-91bd7401de2a","added_by":"auto","created_at":"2024-03-12 12:59:12","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":110483,"visible":true,"origin":"","legend":"\u003cp\u003eLapEfficientDepth model structure\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3998413/v1/f319bd585f6c412a137ba0ba.png"},{"id":52516183,"identity":"253a1bd5-e788-441a-b751-41af48d434d9","added_by":"auto","created_at":"2024-03-12 12:59:12","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":39359,"visible":true,"origin":"","legend":"\u003cp\u003eImage difference feature map generation process\u003c/p\u003e","description":"","filename":"22.png","url":"https://assets-eu.researchsquare.com/files/rs-3998413/v1/d71733c24439bfdfa2c390d3.png"},{"id":52516185,"identity":"ffcbdea0-07ee-4f60-884c-3b3332a9707c","added_by":"auto","created_at":"2024-03-12 12:59:12","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":223098,"visible":true,"origin":"","legend":"\u003cp\u003eTraining loss curve\u003c/p\u003e","description":"","filename":"33.png","url":"https://assets-eu.researchsquare.com/files/rs-3998413/v1/d45f0e484819ba204731b73d.png"},{"id":52516493,"identity":"7604c332-c98f-47a8-9ecf-a8aafd0d8dd0","added_by":"auto","created_at":"2024-03-12 13:07:12","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":1087150,"visible":true,"origin":"","legend":"\u003cp\u003eDepth map effect generated by LapEfficientDepth model\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3998413/v1/37f62c486573d5a4311d0bb5.png"},{"id":67328435,"identity":"1ca20d75-5952-4215-8bd0-02ff53e76cff","added_by":"auto","created_at":"2024-10-23 18:01:39","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2255240,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3998413/v1/1c113ad4-4afd-496e-b0b4-6f8b16281985.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"LapEfficientDepth: lightweight model for monocular depth estimation based on small samples","fulltext":[{"header":"Introduction","content":"\u003cp\u003eMonocular depth estimation refers to the process of estimating the depth information of various objects in a scene using images captured by a single camera\u003csup\u003e1\u003c/sup\u003e. This technique is critically important in the fields of computer vision and robotics, as it provides a cost-effective and flexible solution for applications such as environmental perception, 3D reconstruction, autonomous driving, and augmented reality. In recent years, the development of deep learning technologies has greatly advanced research in monocular depth estimation\u003csup\u003e2\u003c/sup\u003e. Methods based on Convolutional Neural Networks (CNNs) have become the mainstream approach, \u0026nbsp;These methods typically rely on supervised learning with a large volume of labeled depth maps to achieve accurate depth estimation\u003csup\u003e3,4\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eCurrently, datasets for monocular depth estimation are somewhat limited, with the most common datasets including the KITTI dataset\u003csup\u003e5\u003c/sup\u003e, jointly created by the Karlsruhe Institute of Technology in Germany and the Toyota Technological Institute at Chicago, and the NYU dataset\u003csup\u003e6\u003c/sup\u003e, developed by the Department of Computer Science at New York University. The KITTI dataset is primarily used for research in visual perception for mobile robots and autonomous vehicles, while the NYU dataset supports depth estimation and 3D reconstruction research in indoor environments.\u003c/p\u003e\n\u003cp\u003eEstablishing datasets for depth estimation, especially those of high quality and large scale, involves significant hardware and annotation costs. Collecting depth data requires specialized hardware equipment, such as LiDAR (Light Detection and Ranging) and high-precision depth cameras, the cost of which can range from several thousand to tens of thousands of dollars. Furthermore, depth datasets require not only the collection of raw data but also precise annotation and processing, including depth calibration and the registration of image and depth data. This process is often labor-intensive, requiring extensive expertise, thereby incurring high labor costs.\u003c/p\u003e\n\u003cp\u003eIn recent years, few-shot learning has emerged as an important research direction in the field of deep learning, aiming to design more intelligent algorithms that enable models to learn effective feature representations from limited data, thereby reducing the reliance on large-scale annotated datasets\u003csup\u003e7,8\u003c/sup\u003e. This is particularly important for depth estimation, as it addresses the issue of acquiring and annotating high-quality depth data, which is both time-consuming and expensive, to some extent.\u003c/p\u003e\n\u003cp\u003eEvaluating the performance of a monocular depth estimation model primarily involves analyzing metrics from two perspectives\u003csup\u003e9,10,11\u003c/sup\u003e. One is the analysis of error and accuracy from the perspective of the model\u0026apos;s ability to reconstruct depth maps. The other is the analysis of the model\u0026apos;s size in terms of the number of parameters\u003csup\u003e12,13,14\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eSince the introduction of the ResNet model in 2015, convolutional neural networks have been able to improve model accuracy by stacking more layers\u003csup\u003e15\u003c/sup\u003e. The introduction of the ResNet model has also greatly aided in reducing the error and improving the accuracy of monocular depth estimation models. Since then, the quality of depth maps generated by monocular depth estimation models has reached a higher standard.\u003c/p\u003e\n\u003cp\u003eMonoDepth2 model proposed by Godard C, et al., which uses ResNet50 as the feature extraction network, achieved an absolute error of 0.115 on the KITTI dataset, with metrics of a1(\u0026nbsp;), a2(\u0026nbsp;\u003csup\u003e2\u003c/sup\u003e), and a3(\u0026nbsp;\u003csup\u003e3\u003c/sup\u003e) at 0.879, 0.961, and 0.982, respectively\u003csup\u003e16\u003c/sup\u003e. Song and others introduced the LapDepth monocular depth estimation model, which achieved outstanding evaluation metrics on the KITTI dataset, with the error reduced to 0.059 and the a1 metric improved to 0.962, reaching the highest metrics among similar models at the time\u003csup\u003e17\u003c/sup\u003e. LapDepth utilizes the ResNext101 feature extraction network. The ResNext network improves performance and efficiency by introducing \u0026quot;grouped convolution\u0026quot;. The ResNeXt models are designed based on the basic framework of ResNet, incorporating a new dimension called \u0026quot;cardinality\u0026quot;, which refers to the number of parallel paths within each residual block\u003csup\u003e18\u003c/sup\u003e. Through this approach, ResNeXt can enhance the model\u0026apos;s representational capability without significantly increasing computational complexity.\u003c/p\u003e\n\u003cp\u003eThe introduction of the MobileNet lightweight model\u003csup\u003e19\u003c/sup\u003e began to drive the development of lightweight monocular depth estimation models. Wofk D, et al. used MobileNet as the feature extraction network for the FastDepth\u003csup\u003e20\u003c/sup\u003e, resulting in the MobileNet-NNConv5 model with a parameter count of 20.6M. The latest monocular depth estimation model, Lite-Mono-8M\u003csup\u003e21\u003c/sup\u003e, has reduced its total parameter count to 8.7M, with an a1of 0.897, achieving a good balance between model lightweighting and depth estimation accuracy.\u003c/p\u003e\n\u003cp\u003eResearch on models for monocular depth estimation based on small samples remains a topic area with significant research potential and practical value\u003csup\u003e22,23,24\u003c/sup\u003e. In previous work\u003csup\u003e25\u003c/sup\u003e, by studying the characteristics and differences between various convolutional neural network models, we designed a lightweight model, the Res-Efficient model, which achieved good results in pest and disease recognition\u003csup\u003e26\u003c/sup\u003e. Further experimental validation of this model revealed that it performs excellently in identifying the main features in images within specific scenarios. This characteristic is particularly suitable for small-sample learning in specific contexts. This paper builds on the in-depth study of this model and combines experience from the field of monocular depth estimation to explore lightweight models for small-sample monocular depth estimation, proposing the LapEfficientDepth model.\u003c/p\u003e\n\u003cp\u003eThe main contributions of this study are as follows:\u003c/p\u003e\n\u003cp\u003e1. The LapEfficientDepth model, a novel lightweight model that can produce high-quality depth maps after being trained on small sample datasets, is proposed.\u003c/p\u003e\n\u003cp\u003e2. The model\u0026apos;s generalization ability was tested using transfer learning. Experiments demonstrate that the proposed model has strong generalization capabilities, with a slight improvement in error and accuracy after transfer learning of the pre-trained model.\u003c/p\u003e\n\u003cp\u003e3. The proposed model is high-performing. On the KITTI dataset, compared to currently known monocular depth estimation models with parameter counts under 10M, the LapEfficientDepth model has the smallest error and the highest accuracy.\u003c/p\u003e"},{"header":"Experiments","content":"\u003cp\u003e\u003cstrong\u003eExperimental Environment and Datasets\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe experimental environment for this study was conducted on Tencent Cloud\u0026apos;s HAI high-performance server, with a GPU base type computing power, equipped with an Nvidia T4 16GB graphics card, 32GB of memory, and configured with Ubuntu 20.04, Python 3.8, ChatGLM2-6b, CUDA 11.7, cuDNN 8, and PyTorch 2.\u003c/p\u003e\n\u003cp\u003eThe datasets used in the experiments include the KITTI dataset and the ETH3D dataset\u003csup\u003e27\u003c/sup\u003e. The division of the KITTI dataset was constructed following the experimental setup of MonoDepth2. The KITTI dataset serves two purposes in this study: one is to evaluate the performance of the LapEfficientDepth model, and the other is as the dataset for the pre-trained model. The ETH3D-S dataset was divided into a training set and a test set at a 9:1 ratio. The details of the dataset sample numbers are shown in Table 1.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247490.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThe ETH3D dataset is a high-quality 3D reconstruction dataset developed and maintained by ETH Zurich, Switzerland. It aims to provide a rich testing and evaluation platform for research into stereo vision and multi-view stereo algorithms. The ETH3D-S dataset used in this paper is selected from the ETH3D dataset, consisting of data from 3 classic scenes, serving as a small-sample dataset for specific scenarios. The sample numbers for the 3 scenes are shown in Table 2.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247508.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe LapEfficientDepth model proposed in this article modifies the feature extraction network of the LapDepth network\u003csup\u003e11\u003c/sup\u003e, employing a feature extraction network composed of FusedMBConv modules, MBConv modules\u003csup\u003e28\u003c/sup\u003e, and residual blocks, replacing the original ResNext101 feature extraction network in the LapDepth model to form a new monocular depth estimation network structure. The network structure of the LapEfficientDepth monocular depth estimation model is illustrated in Figure 1.\u003c/p\u003e\n\u003cp\u003eLapEfficientDepth adopts an encoder-decoder structure. The encoding layer of the model is the modified feature extraction network. In Figure 1, \u0026apos;S\u0026apos; represents the size of the image input into the model. As the image passes through the feature extraction network, it undergoes four downsampling operations, resulting in a feature map that is 1/16th the size of the original image dimensions.\u003c/p\u003e\n\u003cp\u003eThe encoding layer of the LapEfficientDepth model features two types of convolutional blocks: FusedMBConv and MBConv. The FusedMBConv layer is constructed from two FusedMBConv modules connected in series, with strides of 2 and 1, respectively, to effectively capture and downsample features from the input image. Similarly, the MBConv layer consists of two MBConv modules in series, also with strides of 2 and 1, enhancing the model\u0026apos;s ability to extract detailed features while reducing spatial dimensions.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247553.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis paper introduces two variants of the monocular depth estimation model, differentiated by the parameters of their decoding layers: LapEfficientDepth-m and LapEfficientDepth-s. The LapEfficientDepth-m model\u0026apos;s encoding layer module parameters are detailed in Table 3, designed to offer a balance between performance and computational efficiency. On the other hand, the LapEfficientDepth-s model simplifies the architecture by setting the repeat parameter of all modules to 1, aiming for a more compact and faster model suitable for environments where computational resources are limited or where rapid depth estimation is required. This distinction allows for flexible application across different scenarios, catering to the specific needs of the task at hand.\u003c/p\u003e\n\u003cp\u003eThe decoding layer first utilizes an Atrous Spatial Pyramid Pooling (ASPP) layer to achieve a larger receptive field without significantly losing resolution\u003csup\u003e29\u003c/sup\u003e. Subsequent operations involve concatenation and connection at the same scale to produce corresponding scale feature maps and depth maps. In Figure 1, \u0026nbsp;represents the generated depth residual feature maps, while \u0026nbsp; denotes the difference feature maps of the corresponding scale color images generated by Laplacian residuals, as illustrated in Figure 2.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe generation of difference feature maps enriches texture and feature information through concatenation and connection operations with feature maps of the same scale in the decoding layer, resulting in corresponding scale depth residual feature maps. Finally, the corresponding scale residual feature maps are added to the feature maps obtained from the previous layer\u0026apos;s upsampling, enhancing object boundaries\u0026apos; clarity and ultimately generating depth maps of the same scale. This process intricately blends various techniques to refine the depth estimation, emphasizing the model\u0026apos;s capability to preserve detail and enhance the accuracy of the depth maps produced.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eExperimental Design\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247615.png\"\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis paper focuses on exploring the performance of the LapEfficientDepth model, which uses lightweight modules to construct the feature extraction layers of the network. LapEfficientDepth is designed in medium and small versions.The experiments will evaluate the model\u0026apos;s performance from two aspects. Firstly, the performance of the LapEfficientDepth model will be assessed based on the evaluation metrics obtained from a pre-trained model on KITTI, a publicly available large-scale depth estimation dataset. Secondly, the evaluation will consider the metrics and the quality of depth maps generated from small-sample transfer learning based on the pre-trained model. The design of comparative experiments for LapEfficientDepth is outlined in Table 4.\u003c/p\u003e\n\u003cp\u003eWhere \u0026nbsp; represents the model\u0026apos;s predicted output values, \u0026nbsp; is the true value of the image depth, and \u0026nbsp; is the total number of pixels in the depth image. Additional parameters used in the experiment are detailed in Table 5.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247699.png\"\u003e\u003cstrong\u003e\u003cbr\u003e\u003c/strong\u003e\u003c/p\u003e"},{"header":"Results and Analysis","content":"\u003cp\u003e\u003cstrong\u003eEvaluation of the LapEfficientDepth model on the KITTI dataset\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe experiment began with training on the KITTI dataset, where both LapEfficientDepth-m and LapEfficientDepth-s models underwent 20 training epochs to obtain their respective pre-trained versions. Upon validation with the KITTI test set, the performance metrics obtained were compared with those of other models. The comparative results are showcased in Table 6.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247748.png\"\u003e\u003c/p\u003e\n\u003cp\u003eFrom Table 6, it is evident that ResNext101 has the highest number of parameters, indicating it is the most complex model. In contrast, LapEfficientDepth_s has the fewest parameters. The number of parameters reflects the model\u0026apos;s complexity, implying that LapEfficientDepth_s might be more suitable as a monocular depth estimation model in resource-constrained situations. The absolute relative error measures the accuracy of the model\u0026apos;s relative depth predictions; the lower the absolute relative error, the more accurate the model\u0026apos;s predictions. This comparison underscores the trade-off between model complexity and performance, where lightweight models like LapEfficientDepth_s aim to achieve competitive accuracy with significantly reduced computational resources.\u003c/p\u003e\n\u003cp\u003eIn terms of the absolute relative error (abs rel) metric, the LapDepth model utilizing ResNext101 and ConvNeXt_tiny\u003csup\u003e30\u003c/sup\u003e as feature extraction networks performed the best, while Lite-Mono-8M showed the poorest performance. LapEfficientDepth_m and LapEfficientDepth_s did not perform as well as ResNext101 and ConvNeXt_tiny on this metric but were superior to Lite-Mono-8M.\u003c/p\u003e\n\u003cp\u003eRegarding the accuracy thresholds a1, a2, and a3, which measure the consistency between the predicted and actual depth values, the closer the values are to 1, the more accurate the model\u0026apos;s predictions. Across these three accuracy metrics, all models performed quite well, especially on a3, where, except for Lite-Mono-8M, all other models reached or exceeded 0.995. The LapEfficientDepth models were slightly below the top three models in a1 and a2 but still demonstrated high accuracy.\u003c/p\u003e\n\u003cp\u003eRoot Mean Square Error (RMSE) and Root Mean Square Logarithmic Error (RMSE Log) measure the magnitude of the prediction error, with lower values indicating smaller errors and more accurate predictions. ResNext101 and ConvNeXt_tiny had the best performance on the RMSE metric, while Lite-Mono_8M had the worst. The performance of LapEfficientDepth_m and LapEfficientDepth_s on these metrics was intermediate compared to the other models.\u003c/p\u003e\n\u003cp\u003eOverall, the main advantages of the LapEfficientDepth_m and LapEfficientDepth_s models compared to other models lie in their low number of parameters, implying they may be more suitable in situations with limited computational resources. Although their performance in absolute relative error, accuracy thresholds, as well as RMSE and RMSE Log metrics, may not be the best, considering their smaller model size, these results are still quite impressive. Especially for applications that need to run on edge devices or in scenarios requiring high efficiency under computational resource constraints, the LapEfficientDepth models could be a more optimal choice. This balance between model complexity, efficiency, and performance underscores the significance of lightweight models in advancing depth estimation tasks, particularly in resource-limited environments.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTransfer learning of LapEfficientDepth in small sample data sets\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe transfer learning experiments in this paper involved applying the LapEfficientDepth pre-trained model, obtained from the KITTI dataset, to the ETH3D-S small-sample dataset for full-parameter transfer learning. Table 7 would provide a performance comparison of the LapEfficientDepth model on the ETH3D-S dataset, including two different variants (medium and small), as well as their results after optimization through transfer learning.\u003c/p\u003e\n\u003cp\u003eThe models utilizing transfer learning, LapEfficientDepth-m-transfer and LapEfficientDepth-s-transfer, demonstrated lower errors on the absolute relative error (abs_rel) and log10 metrics, indicating that transfer learning contributes to enhancing the model\u0026apos;s prediction accuracy. Notably, LapEfficientDepth-m-transfer achieved the lowest abs_rel at 0.222 among the four configurations, indicating the smallest average absolute error in predictions.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/122228_c8a1650c59388082/122228_custom_files/img1710247777.png\"\u003e\u003c/p\u003e\n\u003cp\u003eIn terms of accuracy thresholds, the transfer learning models performed better on a1, especially LapEfficientDepth-m-transfer, which reached 0.706 on a1. This means that over 70% of the predictions had an error smaller than 25%. All models performed very well on a2 and a3, nearing or achieving perfection.\u003c/p\u003e\n\u003cp\u003eRegarding the RMSE and RMSE Log metrics, the models optimized with transfer learning also showed superior performance, with LapEfficientDepth-m-transfer having the lowest RMSE at 1.28 among the four configurations, indicating the smallest average error in predictions. Transfer learning also helped reduce the root mean square logarithmic error, with LapEfficientDepth-m-transfer and LapEfficientDepth-s-transfer achieving 0.207 and 0.209, respectively.\u003c/p\u003e\n\u003cp\u003eIn summary, the application of transfer learning on the LapEfficientDepth model significantly improved its performance on the ETH3D-S dataset, particularly in terms of absolute relative error, accuracy thresholds, and root mean square error. This demonstrates that pre-training a model on related tasks and then fine-tuning it for specific tasks (i.e., transfer learning) can effectively enhance the model\u0026apos;s accuracy and generalization ability. This approach leverages the knowledge gained from a comprehensive dataset to improve performance in more specialized scenarios, illustrating the practical value of transfer learning in enhancing deep learning models\u0026apos; adaptability and efficiency across various tasks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraining loss curve analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTraining loss curves for monocular depth estimation models can reflect the trend of the model\u0026apos;s improvement over time on the training data. Typically, these curves show a downward trend, indicating that as training progresses, the model\u0026apos;s performance on the training set improves, meaning the error decreases. Figure 3 would showcase the training loss curves for two different configurations of the LapEfficientDepth model. Each configuration includes two versions: one without the use of transfer learning and one with the application of transfer learning.\u003c/p\u003e\n\u003cp\u003eFrom the upper two graphs in Figure 3, it is observed that without the use of transfer learning, the LapEfficientDepth-m model starts with a relatively high loss that gradually decreases with an increase in the number of iterations. Initially, the loss decreases rapidly, indicating effective learning by the model during this phase. However, as the number of iterations increases, the rate of loss decrease slows down, typically indicating that the model is beginning to saturate, and the amount of new information learned from the data diminishes.\u003c/p\u003e\n\u003cp\u003eWith the application of transfer learning, as shown in the lower two graphs, both the LapEfficientDepth-m and LapEfficientDepth-s models start from a lower value of loss. This demonstrates that transfer learning enables the models to perform better at the onset of training. More importantly, the loss curves show a very rapid decline initially, implying that the models are able to quickly adapt to the new dataset. This is especially evident for the LapEfficientDepth-m model, where the training loss rapidly reduces to a very low level after applying transfer learning.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLapEfficientDepth depth map effect and analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo investigate the generalization ability of the LapEfficientDepth monocular depth estimation model, as well as the effectiveness of transfer learning on a small-sample dataset and the visual quality of the generated depth maps, the experiment compares depth maps generated by the LapEfficientDepth model on the ETH3D-S small-sample test set. The comparison images are presented in Figure 4.\u003c/p\u003e\n\u003cp\u003eFigure 4, showcasing three different scenes with a real label and four different depth map results for each, demonstrates the capabilities of the LapEfficientDepth model variants before and after applying transfer learning. Each row represents a scene, and each column corresponds to a depth map generated by a different model configuration. Observations from Figure 4 indicate that depth maps produced by the LapEfficientDepth-s and LapEfficientDepth-m models estimate broad changes in depth well, with the depth maps appearing smooth and the color gradients soft. However, these models sometimes handle edge regions less delicately.\u003c/p\u003e\n\u003cp\u003eThe depth maps generated by the LapEfficientDepth-s-transfer and LapEfficientDepth-m-transfer models, which have undergone transfer learning training, show improved detail retention while smoothing the depth field as much as possible. This improvement is especially noticeable in areas with good depth continuity, and these models reveal more details, particularly near edges.\u003c/p\u003e\n\u003cp\u003eThe depth maps displayed in Figure 4 reveal the positive impact of transfer learning on the accuracy of depth estimation. The LapEfficientDepth models that have undergone transfer learning seem to perform better in maintaining details and edges than their original versions. This suggests that the LapEfficientDepth models are capable of performing monocular depth estimation tasks in specific scenarios through transfer learning. These models can recognize common edges and shapes using already learned features and adapt to scenarios that were uncommon or entirely unseen during the training process. Such adaptability brings significant value to applications, highlighting the potential of transfer learning to enhance model performance across diverse and challenging environments.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThis paper explores lightweight models for monocular depth estimation in small-sample scenarios and designs the LapEfficientDepth lightweight monocular depth estimation model based on the LapDepth model. Two versions of the model were developed: LapEfficientDepth-m and LapEfficientDepth-s, with parameter counts of 17.6M and 6.0M, respectively. After training on the KITTI dataset, their performance on the KITTI test set achieves an a1 metric of approximately 0.9 and an a3 metric of 0.995 and above, indicating a high consistency between the predicted and actual depth values of the LapEfficientDepth model.\u003c/p\u003e\n\u003cp\u003eOn the ETH3D-S small-sample dataset, the LapEfficientDepth also demonstrated exceptional performance. Analysis of experimental results shows that the LapEfficientDepth model, when trained from scratch on a small-sample dataset, can achieve a depth estimation model with low relative error and high precision. After applying transfer learning, the model effectively transfers knowledge from the pre-trained model, resulting in rapid loss reduction and faster convergence. Since the computational part of the pre-trained model can be reused, transfer learning not only enhances the model\u0026apos;s performance on small-sample datasets but also significantly reduces the time and resource consumption needed for model training.\u003c/p\u003e\n\u003cp\u003eThe LapEfficientDepth model is capable of generating depth maps with clear subjects and soft color gradients, estimating broad changes in the overall depth of images. Transfer learning has a positive impact on the generation of depth maps, with the LapEfficientDepth model showing significant improvement, especially in edge retention and detail depiction. This indicates that the model has learned to extract useful features from larger datasets and successfully adapt these features to new, smaller datasets.\u003c/p\u003e\n\u003cp\u003eThe proposed LapEfficientDepth-m monocular depth estimation model serves as a lightweight foundational model capable of handling tasks that involve training on small-sample datasets. Through transfer learning, the model\u0026apos;s performance on small-sample datasets is enhanced, reducing dependency on the quantity of samples. The other model, LapEfficientDepth-s, is an even more lightweight monocular depth estimation model. Combined with transfer learning, its performance can reach or even exceed that of the LapEfficientDepth-m model trained from scratch. This makes the LapEfficientDepth-s model broadly deployable on lightweight devices and mobile platforms.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eZhao C, Sun Q, Zhang C, et al. Monocular depth estimation based on deep learning: An overview[J]. Science China Technological Sciences, 63(9), 1612-1627 (2020).\u003c/li\u003e\n\u003cli\u003eLei C, Zhengyou L, Yu S. A monocular image depth estimation method based on weighted fusion and point‐wise convolution[J]. IET Computer Vision (2023).\u003c/li\u003e\n\u003cli\u003eKhan F, Hussain S, Basak S, et al. An efficient encoder\u0026ndash;decoder model for portrait depth estimation from single images trained on pixel-accurate synthetic data[J]. Neural Networks, 142, 479-491(2021).\u003c/li\u003e\n\u003cli\u003eLiu Y, Jin M, Pan S, et al. Graph self-supervised learning: A survey[J]. IEEE Transactions on Knowledge and Data Engineering, 35(6), 5879-5900 (2022).\u003c/li\u003e\n\u003cli\u003eGeiger A, Lenz P, Stiller C, et al. Vision meets robotics: The kitti dataset[J]. The International Journal of Robotics Research, 32(11), 1231-1237 (2013).\u003c/li\u003e\n\u003cli\u003eCouprie C, Farabet C, Najman L, et al. Indoor semantic segmentation using depth information[J]. arXiv preprint arXiv, 1301.3572 (2013).\u003c/li\u003e\n\u003cli\u003eSong Y, Wang T, Cai P, et al. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities[J], ACM Computing Surveys (2023).\u003c/li\u003e\n\u003cli\u003eWang Y, Yao Q, Kwok J T, et al. Generalizing from a few examples: A survey on few-shot learning[J]. ACM computing surveys (csur), 53(3), 1-34 (2020).\u003c/li\u003e\n\u003cli\u003eFang Z, Chen X, Chen Y, et al. Towards good practice for CNN-based monocular depth estimation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1091-1100 (2020).\u003c/li\u003e\n\u003cli\u003ePetrovai A, Nedevschi S. Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1578-1588 (2022).\u003c/li\u003e\n\u003cli\u003eLi Z, Yu L, Pan Z. A Monocular SLAM System Based on ResNet Depth Estimation[J], IEEE Sensors Journal, (2023).\u003c/li\u003e\n\u003cli\u003eDong X, Garratt M A, Anavatti S G, et al. Mobilexnet: An efficient convolutional neural network for monocular depth estimation[J], IEEE Transactions on Intelligent Transportation Systems, 23(11), 20134-20147 (2022).\u003c/li\u003e\n\u003cli\u003eYucel M K, Dimaridou V, Drosou A, et al. Real-time monocular depth estimation with sparse supervision on mobile[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2428-2437 (2021).\u003c/li\u003e\n\u003cli\u003ePinasthika K, Utaminingrum F, Lin C Y, et al. Deciphering pixel insights: A deep dive into deep learning strategies for enhanced indoor depth estimation[J], International Journal of Information Management Data Insights, 4(1), 100216 (2024).\u003c/li\u003e\n\u003cli\u003eHe K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778 (2016).\u003c/li\u003e\n\u003cli\u003eGodard C, Mac Aodha O, Firman M, et al. Digging into self-supervised monocular depth estimation[C]//Proceedings of the IEEE/CVF international conference on computer vision, 3828-3838 (2019).\u003c/li\u003e\n\u003cli\u003eSong M, Lim S, Kim W. Monocular depth estimation using laplacian pyramid-based depth residuals[J]. IEEE transactions on circuits and systems for video technology, 31(11), 4381-4393 (2021).\u003c/li\u003e\n\u003cli\u003eXie S, Girshick R, Doll\u0026aacute;r P, et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 1492-1500 (2017).\u003c/li\u003e\n\u003cli\u003eHoward A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, (2017).\u003c/li\u003e\n\u003cli\u003eWofk D, Ma F, Yang T J, et al. Fastdepth: Fast monocular depth estimation on embedded systems[C]//2019 International Conference on Robotics and Automation (ICRA). IEEE, 6101-6108 (2019).\u003c/li\u003e\n\u003cli\u003eZhang N, Nex F, Vosselman G, et al. Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18537-18546 (2023).\u003c/li\u003e\n\u003cli\u003eZhao W. Research on the deep learning of the small sample data based on transfer learning[C]//AIP conference proceedings. AIP Publishing, 1864(1) (2017).\u003c/li\u003e\n\u003cli\u003eKokol P, Kokol M, Zagoranski S. Machine learning on small size samples: A synthetic knowledge synthesis[J]. Science Progress, 105(1): 00368504211029777.18 (2022) \u003c/li\u003e\n\u003cli\u003eSafonova A, Ghazaryan G, Stiller S, et al. Ten deep learning techniques to address small data problems with remote sensing[J]. International Journal of Applied Earth Observation and Geoinformation, 125: 103569 (2023). \u003c/li\u003e\n\u003cli\u003eMing Y, Meng X, Fan C, et al. Deep learning for monocular depth estimation: A review[J]. Neurocomputing, 438, 14-33 (2021).\u003c/li\u003e\n\u003cli\u003eGuan H, Fu C, Zhang G, et al. A lightweight model for efficient identification of plant diseases and pests based on deep learning[J]. Frontiers in Plant Science, 14 (2023).\u003c/li\u003e\n\u003cli\u003eSchops T, Schonberger J L, Galliani S, et al. A multi-view stereo benchmark with high-resolution images and multi-camera videos[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 3260-3269 (2017).\u003c/li\u003e\n\u003cli\u003eTan M, Le Q. Efficientnetv2: Smaller models and faster training[C]//International conference on machine learning, PMLR, 10096-10106 (2021).\u003c/li\u003e\n\u003cli\u003eHe K, Zhang X, Ren S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 37(9), 1904-1916 (2015).\u003c/li\u003e\n\u003cli\u003eLiu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11976-11986 (2022).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Monocular depth estimation, Lightweight, Small samples, Transfer learning","lastPublishedDoi":"10.21203/rs.3.rs-3998413/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3998413/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This study proposes an improved LapDepth model, named LapEfficientDepth, for the monocular depth estimation field. The objective is to reduce the error in model-predicted relative depths and enhance depth estimation accuracy, addressing the issues of substantial resource consumption and large parameter count inherent in the original model. By incorporating lightweight modules, the LapEfficientDepth model significantly reduces model the complexity and resource requirements of the model while maintaining high accuracy estimation. Specifically, the parameter count of the LapEfficientDepth model has been reduced to 6M, constituting only 8.2% of the parameter volume found in the original LapDepth model, while achieving an approximate 1% improvement in accuracy compared to the Lite-Mono-8M model, which has a similar number of parameters. In addition, the LapEfficientDepth model exhibits exceptional transfer learning capabilities. After pre-training on the KITTI dataset and further training on the ETH3D-S dataset, the model achieved a1, a2, and a3 metrics of 0.706, 0.997, and 0.999, respectively, proving its rapid adaptability and learning ability on small sample datasets. This offers an effective solution for high-performance, lightweight monocular depth estimation network models.","manuscriptTitle":"LapEfficientDepth: lightweight model for monocular depth estimation based on small samples","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-12 12:59:07","doi":"10.21203/rs.3.rs-3998413/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7b3c2e1e-3d27-4fda-89e5-85edc6921cfa","owner":[],"postedDate":"March 12th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-10-23T17:53:29+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-12 12:59:07","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3998413","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3998413","identity":"rs-3998413","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}