Self-Supervised Audio Representation Learning Model Based on Time-Frequency Decoupling and Masked Reconstruction

preprint OA: closed
Full text JSON View at publisher

Abstract

Abstract In the field of audio processing, self-supervised learning has emerged as a key paradigm for learning general audio representations.However, existing Transformer-based models, such as the Audio Spectrogram Transformer (AST), commonly face two major challenges.First, they inherit the fixed input size paradigm from computer vision, leading to suboptimal preprocessing like cropping or padding when handling variable-length audio, which results in the loss of critical information or the introduction of redundancy.Second, these models rely on expensive supervised pre-training or cross-modal knowledge transfer capturing the underlying structural patterns of audio directly from raw data. To resolve these challenges in a unified manner, we introduce a new architecture that incorporates a time-frequency decoupling feature extraction module with a dual-task self-supervised learning framework. The model separates the time and frequency dimensions at the input stage, natively supporting variable-length audio inputs and more effectively capturing the unique time-frequency structure of audio. Simultaneously, by combining a generative masked latent prediction task with a discriminative contrastive learning task, the model ensures the learning of robust general representations that encompass both local details and global semantics. This architecture draws inspiration from supervised time-frequency decoupling in time-frequency decoupled audio models and extends it to an unsupervised paradigm for the first time, enabling from-scratch training on the unlabeled AudioSet-20K dataset.Downstream task evaluations cover benchmarks such as AudioSet-20K and Speech Commands V2. Experimental results demonstrate that, without any external pre-training, our model achieves a linear evaluation accuracy of 0.336 on AudioSet-20K, representing a significant relative improvement of 20.4% over self-supervised baseline models.
Full text 26,484 characters · extracted from preprint-html · click to expand
Self-Supervised Audio Representation Learning Model Based on Time-Frequency Decoupling and Masked Reconstruction | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Self-Supervised Audio Representation Learning Model Based on Time-Frequency Decoupling and Masked Reconstruction Jie Xu, Yuhao Dai, Zhifeng Wang This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8361849/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 13 You are reading this latest preprint version Abstract In the field of audio processing, self-supervised learning has emerged as a key paradigm for learning general audio representations.However, existing Transformer-based models, such as the Audio Spectrogram Transformer (AST), commonly face two major challenges.First, they inherit the fixed input size paradigm from computer vision, leading to suboptimal preprocessing like cropping or padding when handling variable-length audio, which results in the loss of critical information or the introduction of redundancy.Second, these models rely on expensive supervised pre-training or cross-modal knowledge transfer capturing the underlying structural patterns of audio directly from raw data. To resolve these challenges in a unified manner, we introduce a new architecture that incorporates a time-frequency decoupling feature extraction module with a dual-task self-supervised learning framework. The model separates the time and frequency dimensions at the input stage, natively supporting variable-length audio inputs and more effectively capturing the unique time-frequency structure of audio. Simultaneously, by combining a generative masked latent prediction task with a discriminative contrastive learning task, the model ensures the learning of robust general representations that encompass both local details and global semantics. This architecture draws inspiration from supervised time-frequency decoupling in time-frequency decoupled audio models and extends it to an unsupervised paradigm for the first time, enabling from-scratch training on the unlabeled AudioSet-20K dataset.Downstream task evaluations cover benchmarks such as AudioSet-20K and Speech Commands V2. Experimental results demonstrate that, without any external pre-training, our model achieves a linear evaluation accuracy of 0.336 on AudioSet-20K, representing a significant relative improvement of 20.4% over self-supervised baseline models. Self-supervised Learning Audio Representation Learning Transformer Time-Frequency Decoupling Masked Reconstruction Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 17 May, 2026 Reviews received at journal 13 May, 2026 Reviewers agreed at journal 11 May, 2026 Reviewers agreed at journal 09 May, 2026 Reviewers agreed at journal 08 May, 2026 Reviewers agreed at journal 08 May, 2026 Reviews received at journal 04 Apr, 2026 Reviewers agreed at journal 24 Mar, 2026 Reviewers agreed at journal 14 Mar, 2026 Reviewers invited by journal 17 Dec, 2025 Editor assigned by journal 17 Dec, 2025 Submission checks completed at journal 17 Dec, 2025 First submitted to journal 14 Dec, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8361849","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":566262419,"identity":"8a9cfb9f-2ad5-4e2f-9e60-a3accf4fc399","order_by":0,"name":"Jie Xu","email":"","orcid":"","institution":"Shanghai Polytechnic University","correspondingAuthor":false,"prefix":"","firstName":"Jie","middleName":"","lastName":"Xu","suffix":""},{"id":566262420,"identity":"0fc4bc37-70e6-4d94-b0f6-23d57322b919","order_by":1,"name":"Yuhao Dai","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBUlEQVRIie3QsWoCMRjA8e8I5JbUWz+R4itECoeTvkpCwKlDS6EoSAkU4ujaqc9Ql9JRCWS6BxB0qItTB91uuKE57CTkuLFD/ktC4EfyBSAW+4dlhKy5rHBECdl8/x3yRtJdGPFw0EPVSanirQgvCn4+6GnynrEcWxHYisGH/EJCCZ3Mymovdfr6iTDfB0XyJhSXBXb8LG53Y45SM/eM4I5BQlA4Lujlll2irdR4n6PfBAlFuSg9SQxh+WNZedL/aSaMWeDSXAgwWt/CmgmmBrgoUPmHqZ6f5c6wydNQuDAZ2+w0KKcvo/7Sbs7+x26XqV1tT/MwuW4NtF5Ea1CTWCwWi133CyfmVUHEONOZAAAAAElFTkSuQmCC","orcid":"","institution":"Shanghai Polytechnic University","correspondingAuthor":true,"prefix":"","firstName":"Yuhao","middleName":"","lastName":"Dai","suffix":""},{"id":566262421,"identity":"267b5a65-9d0b-4a08-a0e6-e8ee12f21c37","order_by":2,"name":"Zhifeng Wang","email":"","orcid":"","institution":"Shanghai Polytechnic University","correspondingAuthor":false,"prefix":"","firstName":"Zhifeng","middleName":"","lastName":"Wang","suffix":""}],"badges":[],"createdAt":"2025-12-15 04:53:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8361849/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8361849/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":99283886,"identity":"a15c96f2-5428-436a-846f-66281fedb3d8","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5842,"visible":true,"origin":"","legend":"","description":"","filename":"a04f390e00d74d7db960b49c68a2978b.json","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/900b8f6162db7acd2f5b52a3.json"},{"id":99319353,"identity":"29efcd6d-4c87-42e4-8c29-8352bc1b816f","added_by":"auto","created_at":"2025-12-31 16:37:01","extension":"xml","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":68172,"visible":true,"origin":"","legend":"","description":"","filename":"a04f390e00d74d7db960b49c68a2978b1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/3a0a76f83cb0c407db2f5c08.xml"},{"id":99283887,"identity":"5d2fb499-89a1-47ff-91c0-a2f46135af3a","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"latexmkrc","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":115,"visible":true,"origin":"","legend":"","description":"","filename":"file.latexmkrc","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/645e78d9951df11467345a14.latexmkrc"},{"id":99283890,"identity":"9145fef2-e038-49f7-9595-346d94fcd3f3","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"pdf","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":59188,"visible":true,"origin":"","legend":"","description":"","filename":"Pointbypointresponse.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/54b480c2febf6f9bf670c27c.pdf"},{"id":99283889,"identity":"d277953b-cdcc-44f9-b56b-e8e8681e060b","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"pdf","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":93963,"visible":true,"origin":"","legend":"","description":"","filename":"coverletter.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/b5ffe95cab72876c469ac11f.pdf"},{"id":99318925,"identity":"7045b847-9c3b-470b-99ce-3ca20af197c4","added_by":"auto","created_at":"2025-12-31 16:35:44","extension":"pdf","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1392140,"visible":true,"origin":"","legend":"","description":"","filename":"latex.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/cf3d94d4325ef9f52e6c2159.pdf"},{"id":99320542,"identity":"da802351-16c6-4f60-93a6-221ff8bdb297","added_by":"auto","created_at":"2025-12-31 16:38:43","extension":"bst","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":151345,"visible":true,"origin":"","legend":"","description":"","filename":"snapacite.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/672194b73ad9b17b0590b5fc.bst"},{"id":99283893,"identity":"eb7d4680-4e5c-464a-ad5b-d82a589e6176","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"bst","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29828,"visible":true,"origin":"","legend":"","description":"","filename":"snaps.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/38f312dca316950abb43f33d.bst"},{"id":99319111,"identity":"259e31b4-9daf-4556-9cb2-71872a171dc4","added_by":"auto","created_at":"2025-12-31 16:36:20","extension":"aux","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9161,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.aux","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/a65805822cb40cc2671d8256.aux"},{"id":99319312,"identity":"270b4eaf-3f0e-4cc9-ad03-718a7dd06042","added_by":"auto","created_at":"2025-12-31 16:36:54","extension":"bbl","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":14870,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.bbl","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/f0b19464649e11665912516e.bbl"},{"id":99318921,"identity":"53cdb378-1c32-4d8f-aa28-16cb90b604a1","added_by":"auto","created_at":"2025-12-31 16:35:44","extension":"blg","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2354,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.blg","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/49fc4eb6bf65a26ee3c7aada.blg"},{"id":99320815,"identity":"cfd411e8-c42c-4a5e-890b-011cfec1137a","added_by":"auto","created_at":"2025-12-31 16:38:54","extension":"dvi","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":137696,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.dvi","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/a8f14c8b3cbe1b4a7fae252d.dvi"},{"id":99320653,"identity":"3cc7bf8e-3700-4e4a-aa94-35e0e4da1ddb","added_by":"auto","created_at":"2025-12-31 16:38:50","extension":"fdb_latexmk","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17140,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.fdblatexmk","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/0e40d9a493406edefe24b34f.fdb_latexmk"},{"id":99319893,"identity":"cb684f1e-aaf7-42c1-9dc8-e41511be37a0","added_by":"auto","created_at":"2025-12-31 16:37:59","extension":"fls","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":22147,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.fls","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/7f5820c86388bc1e394ddfd6.fls"},{"id":99283900,"identity":"bb0d7a9b-2dd9-40fb-92b4-bab480037886","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"log","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":48890,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.log","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/691dcb9d880176d8a23151af.log"},{"id":99321198,"identity":"d42667ef-8981-4528-9f11-9daf559d965d","added_by":"auto","created_at":"2025-12-31 16:39:16","extension":"out","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3907,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.out","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/d7437de8406b73e7b59cdfaf.out"},{"id":99320418,"identity":"1803142d-70ae-4478-b9a8-f72793f0919a","added_by":"auto","created_at":"2025-12-31 16:38:35","extension":"pdf","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3674622,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/4f6e9a3e1bb67779bd0582ac.pdf"},{"id":99283913,"identity":"a57fe7cd-7cd2-4240-8f8e-8c170faadd7d","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"ps","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16280237,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.ps","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/83b4a85c8a705f545466a617.ps"},{"id":99283896,"identity":"a7aacfa0-593b-4a57-9248-513b099f20c1","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"bst","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":35505,"visible":true,"origin":"","legend":"","description":"","filename":"snbasic.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/a69056da36eb20354dd48b67.bst"},{"id":99319895,"identity":"f5f0620a-e7a7-4116-a0ae-062728fff3bc","added_by":"auto","created_at":"2025-12-31 16:37:59","extension":"bst","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33968,"visible":true,"origin":"","legend":"","description":"","filename":"snchicago.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/f01dee2582c140d85585295e.bst"},{"id":99283905,"identity":"57609621-8fa1-4798-898a-bc34847be7d3","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"cls","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":55857,"visible":true,"origin":"","legend":"","description":"","filename":"snjnl.cls","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/ee2521aaff2a2e2d002c78a8.cls"},{"id":99319289,"identity":"70604948-6cb2-4aff-81f9-2846e644993f","added_by":"auto","created_at":"2025-12-31 16:36:50","extension":"bst","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64023,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysay.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/071ad2fa9f25fe8e48a55f2e.bst"},{"id":99320452,"identity":"83498f55-45ff-4d2b-87f8-0b5cfac414ae","added_by":"auto","created_at":"2025-12-31 16:38:36","extension":"bst","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64164,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysnum.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/972965c0d220b3fe3bd2e27e.bst"},{"id":99283902,"identity":"42164037-87bc-497c-9259-5fa6623098da","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"bst","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39056,"visible":true,"origin":"","legend":"","description":"","filename":"snnature.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/23d74720fed89792534d0688.bst"},{"id":99283906,"identity":"620b49a4-1ca9-4b1c-ba96-e6449e5c1a03","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"bst","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39951,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouveray.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/6647e33df7d48080aae13bf7.bst"},{"id":99319478,"identity":"d0df7d2f-428b-4bdc-8f54-ed2a0116e857","added_by":"auto","created_at":"2025-12-31 16:37:20","extension":"bst","order_by":29,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40758,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouvernum.bst","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/c8367459bb4766cf726bd831.bst"},{"id":99319365,"identity":"945018c5-09e5-41dc-a212-551edb33c4d7","added_by":"auto","created_at":"2025-12-31 16:37:04","extension":"xml","order_by":30,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":75742,"visible":true,"origin":"","legend":"","description":"","filename":"a04f390e00d74d7db960b49c68a2978b1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/3a0dfb7e73339c407e6f1e9b.xml"},{"id":99283910,"identity":"665e7857-296f-446e-875d-39a7c30f0a0e","added_by":"auto","created_at":"2025-12-31 09:04:36","extension":"html","order_by":31,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":84277,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1/7237f47f49ab0ec8c5b09674.html"},{"id":99787779,"identity":"cc339bff-015f-43ef-ba8f-fd84ce432bda","added_by":"auto","created_at":"2026-01-08 12:33:51","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1328728,"visible":true,"origin":"","legend":"","description":"","filename":"latex.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8361849/v1_covered_63259367-997c-4d6e-a64f-f25233ddb297.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Self-Supervised Audio Representation Learning Model Based on Time-Frequency Decoupling and Masked Reconstruction","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":true,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"signal-image-and-video-processing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sivp","sideBox":"Learn more about [Signal, Image and Video Processing](http://link.springer.com/journal/11760)","snPcode":"11760","submissionUrl":"https://submission.nature.com/new-submission/11760/3","title":"Signal, Image and Video Processing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Self-supervised Learning, Audio Representation Learning, Transformer, Time-Frequency Decoupling, Masked Reconstruction","lastPublishedDoi":"10.21203/rs.3.rs-8361849/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8361849/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn the field of audio processing, self-supervised learning has emerged as a key paradigm for learning general audio representations.However, existing Transformer-based models, such as the Audio Spectrogram Transformer (AST), commonly face two major challenges.First, they inherit the fixed input size paradigm from computer vision, leading to suboptimal preprocessing like cropping or padding when handling variable-length audio, which results in the loss of critical information or the introduction of redundancy.Second, these models rely on expensive supervised pre-training or cross-modal knowledge transfer capturing the underlying structural patterns of audio directly from raw data. To resolve these challenges in a unified manner, we introduce a new architecture that incorporates a time-frequency decoupling feature extraction module with a dual-task self-supervised learning framework. The model separates the time and frequency dimensions at the input stage, natively supporting variable-length audio inputs and more effectively capturing the unique time-frequency structure of audio. Simultaneously, by combining a generative masked latent prediction task with a discriminative contrastive learning task, the model ensures the learning of robust general representations that encompass both local details and global semantics. This architecture draws inspiration from supervised time-frequency decoupling in time-frequency decoupled audio models and extends it to an unsupervised paradigm for the first time, enabling from-scratch training on the unlabeled AudioSet-20K dataset.Downstream task evaluations cover benchmarks such as AudioSet-20K and Speech Commands V2. Experimental results demonstrate that, without any external pre-training, our model achieves a linear evaluation accuracy of 0.336 on AudioSet-20K, representing a significant relative improvement of 20.4% over self-supervised baseline models.\u003c/p\u003e","manuscriptTitle":"Self-Supervised Audio Representation Learning Model Based on Time-Frequency Decoupling and Masked Reconstruction","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-31 09:04:31","doi":"10.21203/rs.3.rs-8361849/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-17T18:09:01+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-13T13:31:13+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"275670282298262022640939141746632580423","date":"2026-05-11T08:01:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"59386415085439271230290830381391061136","date":"2026-05-09T18:03:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"296120393821695522281041191430467262775","date":"2026-05-08T07:52:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"144911937238946532547059951753531524677","date":"2026-05-08T07:48:46+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-04T14:50:49+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"131773057216247435405584151342589376785","date":"2026-03-24T04:23:34+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"218801782168480183715018972016661073960","date":"2026-03-14T05:25:25+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-17T20:05:57+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-12-17T12:54:28+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-12-17T12:49:24+00:00","index":"","fulltext":""},{"type":"submitted","content":"Signal, Image and Video Processing","date":"2025-12-15T04:50:50+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"signal-image-and-video-processing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sivp","sideBox":"Learn more about [Signal, Image and Video Processing](http://link.springer.com/journal/11760)","snPcode":"11760","submissionUrl":"https://submission.nature.com/new-submission/11760/3","title":"Signal, Image and Video Processing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"9d4ed533-a059-417a-80b7-ebc050365b28","owner":[],"postedDate":"December 31st, 2025","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-17T18:09:01+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-13T13:31:13+00:00","index":53,"fulltext":""},{"type":"reviewerAgreed","content":"275670282298262022640939141746632580423","date":"2026-05-11T08:01:30+00:00","index":52,"fulltext":""},{"type":"reviewerAgreed","content":"59386415085439271230290830381391061136","date":"2026-05-09T18:03:27+00:00","index":51,"fulltext":""},{"type":"reviewerAgreed","content":"296120393821695522281041191430467262775","date":"2026-05-08T07:52:10+00:00","index":50,"fulltext":""},{"type":"reviewerAgreed","content":"144911937238946532547059951753531524677","date":"2026-05-08T07:48:46+00:00","index":49,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-17T18:23:35+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-31 09:04:31","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8361849","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8361849","identity":"rs-8361849","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}

Text is read by the "Ask this paper" AI Q&A widget below. Extraction quality varies by source — PMC NXML preserves structure cleanly, OA-HTML may include some navigation residue, and OA-PDF can have broken hyphenation. The publisher copy (via DOI) is the canonical version.

My notes (saved in your browser only)

Ask this paper AI returns verbatim quotes from the full text · source: preprint-html

Answers must be backed by verbatim quotes from this paper's full text. Hallucinated quotes are dropped automatically; if no verbatim passage answers the question, we say so. How this works

Citation neighborhood (no data yet)

We don't have any in-corpus citations linked to this paper yet. This is a recent paper (2025) — citers typically take a year or two to land, and the OpenAlex reference graph may still be filling in.

Source provenance

europepmc
last seen: 2026-05-20T01:45:00.602351+00:00