Speaker-Aware Simulation Improves Conversational Speech Recognition

Speaker-Aware Simulation Improves Conversational Speech Recognition | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Speaker-Aware Simulation Improves Conversational Speech Recognition Máté Gedeon, Péter Mihajlik This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8798359/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Automatic speech recognition (ASR) for conversational speech remains challenging due to the limited availability of large-scale, well-annotated multi-speaker dialogue data and the complex temporal dynamics of natural interactions. Speaker-aware simulated conversations (SASC) offer an effective data augmentation strategy by transforming single-speaker recordings into realistic multi-speaker dialogues. However, prior work has primarily focused on English data, leaving questions about the applicability to lower-resource languages. In this paper, we adapt and implement the SASC framework for Hungarian conversational ASR. We further propose C-SASC, an extended variant that incorporates pause modeling conditioned on utterance duration, enabling a more faithful representation of local temporal dependencies observed in human conversation while retaining the simplicity and efficiency of the original approach. We generate synthetic Hungarian dialogues from the BEA-Large corpus and combine them with real conversational data for ASR training. Both SASC and C-SASC are evaluated extensively under a wide range of simulation configurations, using conversational statistics derived from CallHome, BEA-Dialogue, and GRASS corpora. Experimental results show that speaker-aware conversational simulation consistently improves recognition performance over naive concatenation-based augmentation. While the additional duration conditioning in C-SASC yields modest but systematic gains--most notably in character-level error rates--its effectiveness depends on the match between source conversational statistics and the target domain. Overall, our findings confirm the robustness of speaker-aware conversational simulation for Hungarian ASR and highlight the benefits and limitations of increasingly detailed temporal modeling in synthetic dialogue generation. Conversational speech automatic speech recognition simulated conversations data augmentation Full Text Additional Declarations Competing interest reported. Project No. 2025-2.1.2-EKÖP-KDP-2025-00005 has been implemented with the support provided by the Ministry of Culture and Innovation of Hungary from the National Research, Development and Innovation (NRDI) Fund, financed under the EKÖP_KDP-25-1-BME-21 funding scheme. The work was also partially supported by the Hungarian NRDI Fund through the projects NKFIH K143075 and K135038, NKFIH-828-2/2021(MILAB). Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8798359","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":587152887,"identity":"c9ea2de3-dca4-4eb4-bbac-a6a56f78d0e9","order_by":0,"name":"Máté Gedeon","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVRIiWNgGAWjYHACAxAhx8CQwMAM5xKjxZh0LYkNRGuRd2/e+ODjjm3p/ew5BsyFOTYM5hIJ+LUYnjlWbDjzzO3cmT1vDJhnbktjsJxBSMuMHDNp3rbbuRtuAG3h3XaYweAGIS3z35j/BmpJt4do+U9Yi7wEjxkzUEuCgQRYywHCWgx40oolZ7bdNpxx5lnB4ZnbknkMzjwgYEv74Y0fPrbdludvT974uHCbnZzBcUK2HEDigNg8+NWDbGkgqGQUjIJRMApGPAAAavVF2a4Zt+8AAAAASUVORK5CYII=","orcid":"","institution":"Budapest University of Technology and Economics","correspondingAuthor":true,"prefix":"","firstName":"Máté","middleName":"","lastName":"Gedeon","suffix":""},{"id":587152888,"identity":"58d200e4-5851-4bbd-87b3-f46bdc820315","order_by":1,"name":"Péter Mihajlik","email":"","orcid":"","institution":"Budapest University of Technology and Economics","correspondingAuthor":false,"prefix":"","firstName":"Péter","middleName":"","lastName":"Mihajlik","suffix":""}],"badges":[],"createdAt":"2026-02-05 14:40:02","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8798359/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8798359/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104324251,"identity":"ee516785-287b-4a49-a3f1-55df580e510b","added_by":"auto","created_at":"2026-03-10 13:52:12","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":364895,"visible":true,"origin":"","legend":"","description":"","filename":"SASCjournalv2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8798359/v1_covered_634c2ffa-e291-4dec-ac37-2429cf6b67d9.pdf"}],"financialInterests":"Competing interest reported. Project No. 2025-2.1.2-EKÖP-KDP-2025-00005 has been implemented with the support provided by the Ministry of Culture and Innovation of Hungary from the National Research, Development and Innovation (NRDI) Fund, financed under the EKÖP_KDP-25-1-BME-21 funding scheme.\nThe work was also partially supported by the Hungarian NRDI Fund through the projects NKFIH K143075 and K135038, NKFIH-828-2/2021(MILAB).","formattedTitle":"Speaker-Aware Simulation Improves Conversational Speech Recognition","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Conversational speech, automatic speech recognition, simulated conversations, data augmentation","lastPublishedDoi":"10.21203/rs.3.rs-8798359/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8798359/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Automatic speech recognition (ASR) for conversational speech remains challenging due to the limited availability of large-scale, well-annotated multi-speaker dialogue data and the complex temporal dynamics of natural interactions. Speaker-aware simulated conversations (SASC) offer an effective data augmentation strategy by transforming single-speaker recordings into realistic multi-speaker dialogues. However, prior work has primarily focused on English data, leaving questions about the applicability to lower-resource languages.\nIn this paper, we adapt and implement the SASC framework for Hungarian conversational ASR. We further propose C-SASC, an extended variant that incorporates pause modeling conditioned on utterance duration, enabling a more faithful representation of local temporal dependencies observed in human conversation while retaining the simplicity and efficiency of the original approach.\nWe generate synthetic Hungarian dialogues from the BEA-Large corpus and combine them with real conversational data for ASR training. Both SASC and C-SASC are evaluated extensively under a wide range of simulation configurations, using conversational statistics derived from CallHome, BEA-Dialogue, and GRASS corpora. Experimental results show that speaker-aware conversational simulation consistently improves recognition performance over naive concatenation-based augmentation. While the additional duration conditioning in C-SASC yields modest but systematic gains--most notably in character-level error rates--its effectiveness depends on the match between source conversational statistics and the target domain.\nOverall, our findings confirm the robustness of speaker-aware conversational simulation for Hungarian ASR and highlight the benefits and limitations of increasingly detailed temporal modeling in synthetic dialogue generation.","manuscriptTitle":"Speaker-Aware Simulation Improves Conversational Speech Recognition","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-10 13:51:52","doi":"10.21203/rs.3.rs-8798359/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3e714997-495b-4ead-b5d8-adaacbcf63bd","owner":[],"postedDate":"March 10th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-10T13:51:52+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-10 13:51:52","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8798359","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8798359","identity":"rs-8798359","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}