A Novel Multimodal and Multiscale Method for Intelligent Operation and Maintenance of Transformers Based on the Improved Deep Visual Large Model DETR+X and Digital Twin

A Novel Multimodal and Multiscale Method for Intelligent Operation and Maintenance of Transformers Based on the Improved Deep Visual Large Model DETR+X and Digital Twin | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Novel Multimodal and Multiscale Method for Intelligent Operation and Maintenance of Transformers Based on the Improved Deep Visual Large Model DETR+X and Digital Twin Xuedong Zhang, Wenlei Sun, Ke Chen, Shijie Song This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5218555/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 02 Jan, 2025 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract To achieve real-time monitoring and intelligent maintenance of transformers, a framework based on deep vision and digital twin has been developed. An enhanced visual detection model, DETR + X, is proposed, implementing multidimensional sample data augmentation through Swin2SR and GAN networks. This model converts one-dimensional DGA data into three-dimensional feature images based on Gram angle fields, facilitating the transformation and fusion of heterogeneous modal information. The Pyramid Vision Transformer (PVT) is innovatively adopted as the backbone for image feature extraction, replacing the traditional ResNet structure. A Deformable Attention mechanism is employed to handle the complex spatial structure of multi-scale features. Testing results indicate that the improved DETR + X model performs well in transformer state recognition tasks, achieving a classification accuracy of 100% for DGA feature maps. In object detection tasks, it surpasses advanced models such as YOLOV8 and Deformable DETR in terms of mAP50 scores, particularly demonstrating significant enhancements in small object detection. Furthermore, the Llava-7b model, fine-tuned based on domain expertise, serves as an expert decision-making tool for transformer maintenance, providing accurate operational recommendations based on visual detection results. Finally, based on digital twin and inference models, a comprehensive platform has been developed to achieve real-time monitoring and intelligent maintenance of transformers. Physical sciences/Engineering Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Physics/Information theory and computation Physical sciences/Physics/Techniques and instrumentation Physical sciences/Energy science and technology/Energy infrastructure Physical sciences/Energy science and technology/Energy infrastructure/Power distribution Physical sciences/Energy science and technology/Energy infrastructure/Power stations Transformer operation and maintenance Vision Detection Large model Digital twin Multi-modal Multi-scale Decision-making suggestions generation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 02 Jan, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 21 Nov, 2024 Reviews received at journal 19 Nov, 2024 Reviews received at journal 18 Nov, 2024 Reviewers agreed at journal 07 Nov, 2024 Reviewers agreed at journal 04 Nov, 2024 Reviewers invited by journal 04 Nov, 2024 Editor assigned by journal 04 Nov, 2024 Editor invited by journal 22 Oct, 2024 Submission checks completed at journal 21 Oct, 2024 First submitted to journal 07 Oct, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5218555","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":380934984,"identity":"9abf269d-cf97-4f9d-bc86-3ff69dcde253","order_by":0,"name":"Xuedong Zhang","email":"","orcid":"","institution":"Xinjiang University","correspondingAuthor":false,"prefix":"","firstName":"Xuedong","middleName":"","lastName":"Zhang","suffix":""},{"id":380934985,"identity":"23dcba13-adca-436c-9993-297d5bd902fe","order_by":1,"name":"Wenlei Sun","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5UlEQVRIiWNgGAWjYFCCA1CaHcj8YGBjR4IWZgaGgzMK0pJJsA2ohZnnwyHGBkIKDQ6eMf7Mu+OwnDkz88HDNgYHmBnYDx/dgE+LZMMZM2neM4eNLZvZEg7nGNzhY+BJS7uBTws/wxkzZt6224kbDvMYALU8Y2aQ4DHDq4WNAeSwttv1Gw7zfzhsYXCYsYGQFqAtBtJALQkGh3kYDjMQo0Wy4ViZ5Ny2/4YbDrMZHOwxSEtmI+QXgxuHN39425Ymb3C8+fGHH39s7PjZDx/Dq4VB4gCG7wgB/gaCSkbBKBgFo2CkAwD5t01f3e44UwAAAABJRU5ErkJggg==","orcid":"","institution":"Xinjiang University","correspondingAuthor":true,"prefix":"","firstName":"Wenlei","middleName":"","lastName":"Sun","suffix":""},{"id":380934986,"identity":"50766015-9fa5-4021-b02f-068d3f9504fd","order_by":2,"name":"Ke Chen","email":"","orcid":"","institution":"Tebian Electric Apparatus (China)","correspondingAuthor":false,"prefix":"","firstName":"Ke","middleName":"","lastName":"Chen","suffix":""},{"id":380934988,"identity":"23045ac9-0815-42e1-af9e-bb0cb9f58b0b","order_by":3,"name":"Shijie Song","email":"","orcid":"","institution":"Xinjiang University","correspondingAuthor":false,"prefix":"","firstName":"Shijie","middleName":"","lastName":"Song","suffix":""}],"badges":[],"createdAt":"2024-10-07 13:38:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5218555/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5218555/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-024-83561-7","type":"published","date":"2025-01-02T15:57:12+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":73093243,"identity":"72f69f11-e99d-4da8-85d2-d040528f0cfe","added_by":"auto","created_at":"2025-01-06 16:11:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1848980,"visible":true,"origin":"","legend":"","description":"","filename":"ANovelMultimodalandMultiscaleMethodforIntelligentOperationandMaintenanceofTransformersBasedontheImprovedDeepVisualLargeModelDETRXandDigitalTwin.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5218555/v1_covered_5b8685c8-48d5-473b-b068-27047f2eab8d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Novel Multimodal and Multiscale Method for Intelligent Operation and Maintenance of Transformers Based on the Improved Deep Visual Large Model DETR+X and Digital Twin","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Transformer operation and maintenance, Vision Detection Large model, Digital twin, Multi-modal, Multi-scale, Decision-making suggestions generation","lastPublishedDoi":"10.21203/rs.3.rs-5218555/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5218555/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eTo achieve real-time monitoring and intelligent maintenance of transformers, a framework based on deep vision and digital twin has been developed. An enhanced visual detection model, DETR\u0026thinsp;+\u0026thinsp;X, is proposed, implementing multidimensional sample data augmentation through Swin2SR and GAN networks. This model converts one-dimensional DGA data into three-dimensional feature images based on Gram angle fields, facilitating the transformation and fusion of heterogeneous modal information. The Pyramid Vision Transformer (PVT) is innovatively adopted as the backbone for image feature extraction, replacing the traditional ResNet structure. A Deformable Attention mechanism is employed to handle the complex spatial structure of multi-scale features. Testing results indicate that the improved DETR\u0026thinsp;+\u0026thinsp;X model performs well in transformer state recognition tasks, achieving a classification accuracy of 100% for DGA feature maps. In object detection tasks, it surpasses advanced models such as YOLOV8 and Deformable DETR in terms of mAP50 scores, particularly demonstrating significant enhancements in small object detection. Furthermore, the Llava-7b model, fine-tuned based on domain expertise, serves as an expert decision-making tool for transformer maintenance, providing accurate operational recommendations based on visual detection results. Finally, based on digital twin and inference models, a comprehensive platform has been developed to achieve real-time monitoring and intelligent maintenance of transformers.\u003c/p\u003e","manuscriptTitle":"A Novel Multimodal and Multiscale Method for Intelligent Operation and Maintenance of Transformers Based on the Improved Deep Visual Large Model DETR+X and Digital Twin","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-12-12 02:08:52","doi":"10.21203/rs.3.rs-5218555/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-11-21T12:29:42+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-11-19T13:26:57+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-11-18T13:17:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"160660506030822604345659385052097357846","date":"2024-11-08T01:59:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"111658042223217130279379637110267031848","date":"2024-11-04T06:40:19+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-11-04T06:04:39+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-11-04T05:34:20+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-10-23T02:01:23+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-10-21T08:49:55+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-10-07T13:23:37+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"22d03ea0-27e8-4b16-ab27-a5b64f61302e","owner":[],"postedDate":"December 12th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":40570820,"name":"Physical sciences/Engineering"},{"id":40570821,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":40570822,"name":"Physical sciences/Physics/Information theory and computation"},{"id":40570823,"name":"Physical sciences/Physics/Techniques and instrumentation"},{"id":40570824,"name":"Physical sciences/Energy science and technology/Energy infrastructure"},{"id":40570825,"name":"Physical sciences/Energy science and technology/Energy infrastructure/Power distribution"},{"id":40570826,"name":"Physical sciences/Energy science and technology/Energy infrastructure/Power stations"}],"tags":[],"updatedAt":"2025-01-06T16:00:04+00:00","versionOfRecord":{"articleIdentity":"rs-5218555","link":"https://doi.org/10.1038/s41598-024-83561-7","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-01-02 15:57:12","publishedOnDateReadable":"January 2nd, 2025"},"versionCreatedAt":"2024-12-12 02:08:52","video":"","vorDoi":"10.1038/s41598-024-83561-7","vorDoiUrl":"https://doi.org/10.1038/s41598-024-83561-7","workflowStages":[]},"version":"v1","identity":"rs-5218555","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5218555","identity":"rs-5218555","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}