{"paper_id":"4cd57c3a-494d-4fdf-bc4c-e9eb4dc9b5ee","body_text":"Discovering Chemical Space from First Principles with Reinforcement Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Discovering Chemical Space from First Principles with Reinforcement Learning Bjarke Hastrup, Francois Cornet, Tejs Vegge, Arghya Bhowmik This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6900238/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Discovering novel stable molecules without training data remains a grand scientific challenge. Current molecular generative models are trained on large, pre-curated datasets, which introduce biases and limit exploration of novel chemistry. In contrast, we propose a new paradigm: autonomous, generalized agents capable of mapping vast, unknown chemical spaces without any pretraining. For the first time, we present a self-guided agent that autonomously constructs valid 3D isomers under stoichiometric constraints and is trained exclusively online using reinforcement learning. Unlike existing approaches that generally overfit to a specific chemical formula, we establish a multi-composition training scheme that enables a broad generalization across diverse chemistry, guided by energy- and validity-based rewards. Our agent can discover up to an order of magnitude more valid isomers on unseen test formulas than the baseline. These results fulfil the promise of online RL as a powerful paradigm for scalable tabula rasa exploration of the chemical configuration space. Physical sciences/Chemistry/Cheminformatics Physical sciences/Chemistry/Theoretical chemistry/Computational chemistry Physical sciences/Chemistry/Theoretical chemistry/Method development Physical sciences/Chemistry/Theoretical chemistry/Structure prediction Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-6900238\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":false,\"archivedVersions\":[],\"articleType\":\"Article\",\"associatedPublications\":[],\"authors\":[{\"id\":471720191,\"identity\":\"d5521e4e-7264-4730-8011-0bb513f099f9\",\"order_by\":0,\"name\":\"Bjarke Hastrup\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Technical University of Denmark\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Bjarke\",\"middleName\":\"\",\"lastName\":\"Hastrup\",\"suffix\":\"\"},{\"id\":471720192,\"identity\":\"aa03d6df-9b7d-4b96-9e28-188d18045223\",\"order_by\":1,\"name\":\"Francois Cornet\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Technical University of Denmark\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Francois\",\"middleName\":\"\",\"lastName\":\"Cornet\",\"suffix\":\"\"},{\"id\":471720193,\"identity\":\"f0105f21-7d03-4d00-a9e8-a7f57fe64f6b\",\"order_by\":2,\"name\":\"Tejs Vegge\",\"email\":\"\",\"orcid\":\"https://orcid.org/0000-0002-1484-0284\",\"institution\":\"Department of Energy Conversion and Storage, Technical University of Denmark\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Tejs\",\"middleName\":\"\",\"lastName\":\"Vegge\",\"suffix\":\"\"},{\"id\":471720190,\"identity\":\"e7c65504-6ea6-47bd-b431-f3f4dc2a41fe\",\"order_by\":3,\"name\":\"Arghya Bhowmik\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAkElEQVRIiWNgGAWjYFACHjaGD2wgRhoJWhhnsBmQqIWZhyQt/LPPHntsU/YnsYE9LYE4LRLn8tKNc84ZJDbwPDtApDVneMykc9uAWiTSG4jTIQ/SYkmSFgOQFkawljQiHWZ4hi9NsuecsXEbz7ME4rTIneE9JvGjTE62nz3NgDgtcMBGovpRMApGwSgYBfgAADEPJh6wRBjNAAAAAElFTkSuQmCC\",\"orcid\":\"https://orcid.org/0000-0003-3198-5116\",\"institution\":\"Technical University of Denmark\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Arghya\",\"middleName\":\"\",\"lastName\":\"Bhowmik\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-06-15 21:50:15\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-6900238/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-6900238/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":85331788,\"identity\":\"6d3bb504-8849-48d9-9ac5-9eb3a8858909\",\"added_by\":\"auto\",\"created_at\":\"2025-06-24 18:37:57\",\"extension\":\"pdf\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":4453867,\"visible\":true,\"origin\":\"\",\"legend\":\"Article File\",\"description\":\"\",\"filename\":\"NatureCommunicationsRediscoveringChemicalSpacefromFirstPrincipleswithReinforcementLearning1.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6900238/v1_covered_604e1a23-5f54-4000-b2d9-bb80c0ddf831.pdf\"}],\"financialInterests\":\"There is \\u003cb\\u003eNO\\u003c/b\\u003e Competing Interest.\",\"formattedTitle\":\"Discovering Chemical Space from First Principles with Reinforcement Learning\",\"fulltext\":[],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":false,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":true,\"hideJournal\":false,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":true,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":true,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"nature-portfolio\",\"isNatureJournal\":true,\"hasQc\":false,\"allowDirectSubmit\":false,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"\",\"title\":\"Nature Portfolio\",\"twitterHandle\":\"\",\"acdcEnabled\":false,\"dfaEnabled\":false,\"editorialSystem\":\"ejp\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false},\"keywords\":\"\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-6900238/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-6900238/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"Discovering novel stable molecules without training data remains a grand scientific challenge. Current molecular generative models are trained on large, pre-curated datasets, which introduce biases and limit exploration of novel chemistry. In contrast, we propose a new paradigm: autonomous, generalized agents capable of mapping vast, unknown chemical spaces without any pretraining. \\r\\nFor the first time, we present a self-guided agent that autonomously constructs valid 3D isomers under stoichiometric constraints and is trained exclusively online using reinforcement learning. Unlike existing approaches that generally overfit to a specific chemical formula, we establish a multi-composition training scheme that enables a broad generalization across diverse chemistry, guided by energy- and validity-based rewards. Our agent can discover up to an order of magnitude more valid isomers on unseen test formulas than the baseline. These results fulfil the promise of online RL as a powerful paradigm for scalable tabula rasa exploration of the chemical configuration space.\",\"manuscriptTitle\":\"Discovering Chemical Space from First Principles with Reinforcement Learning\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-06-17 08:35:32\",\"doi\":\"10.21203/rs.3.rs-6900238/v1\",\"editorialEvents\":[],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"nature-communications\",\"isNatureJournal\":true,\"hasQc\":false,\"allowDirectSubmit\":false,\"externalIdentity\":\"NCOMMS\",\"sideBox\":\"Learn more about [Nature Communications](http://www.nature.com/ncomms/)\",\"snPcode\":\"\",\"submissionUrl\":\"https://mts-ncomms.nature.com/\",\"title\":\"Nature Communications\",\"twitterHandle\":\"\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"ejp\",\"reportingPortfolio\":\"Nature Communications\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false}}],\"origin\":\"\",\"ownerIdentity\":\"2a57c265-c967-4d8a-8945-2387c873afe7\",\"owner\":[],\"postedDate\":\"June 17th, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"under-review\",\"subjectAreas\":[{\"id\":50089668,\"name\":\"Physical sciences/Chemistry/Cheminformatics\"},{\"id\":50089669,\"name\":\"Physical sciences/Chemistry/Theoretical chemistry/Computational chemistry\"},{\"id\":50089670,\"name\":\"Physical sciences/Chemistry/Theoretical chemistry/Method development\"},{\"id\":50089671,\"name\":\"Physical sciences/Chemistry/Theoretical chemistry/Structure prediction\"}],\"tags\":[],\"updatedAt\":\"2025-11-17T08:27:28+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2025-06-17 08:35:32\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-6900238\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-6900238\",\"identity\":\"rs-6900238\",\"version\":[\"v1\"]},\"buildId\":\"8U1c8b4HqxoKbykW_rLl7\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}