Joint Exploration of Neural Networks and Systolic Hardware for Improved AI Accelerator Performance

Joint Exploration of Neural Networks and Systolic Hardware for Improved AI Accelerator Performance | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Joint Exploration of Neural Networks and Systolic Hardware for Improved AI Accelerator Performance Annina Gutermann, Alexey Serdyuk, Foivos Paraskevas, Hella Toto Kiesa, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9181151/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract When executing common \glspl{nn} on custom AI accelerators, the high performance suggested by advertised Giga or Tera Operations per Second (GOPS/TOPS) is typically not achieved, as low hardware utilization often leads to an effective performance in the single-digit percentage range of the theoretical peak.This discrepancy arises as NNs are typically designed without accounting for the target hardware, leading to inefficient mappings and software optimizations that fail to deliver the expected gains.Addressing this, we present a hardware-aware workflow that integrates accurate latency modeling and design space exploration to optimize both neural network architectures and the underlying systolic-array-based accelerator.We develop and validate two high-precision latency models for two different \gls{rs} dataflows on our target accelerator.Using these models together with a structured search space generation, we generate Pareto-optimal search spaces in terms of achieved GOPS and latency for a given hardware target, and use these for a Bayesian \acrlong{hwnas}. We further explore the accelerator design itself, varying the PE array dimensions and clock ratio to identify hardware configurations that maximize efficiency and minimize inference latency for each network and dataflow.We demonstrate our approach on ResNet-like networks. On the original hardware, the discovered ResNet-50-like architecture achieves an 85% relative increase in hardware utilization, reduces latency by 21% and parameter count by 18%, and maintains baseline ImageNet accuracy.On the jointly optimized hardware, latency and area are further reduced while efficiency is increased by up to 94%, with up to 20% fewer PEs compared to the baseline configuration. For ResNet-34, similar trends are observed, with latency reductions exceeding 33% and efficiency gains up to 43%.To enable reproducibility, we open-source our complete workflow of latency models, search space generation, NAS and training pipeline. Performance Predictors Neural Architecture Search Hardware Acceleration Search Space Design Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9181151","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":627744823,"identity":"e25691aa-1330-493c-a131-ed9ae18942cf","order_by":0,"name":"Annina Gutermann","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAx0lEQVRIiWNgGAWjYFAD9jaGA0CKsYGwUmYozXOM4cAB0rRIpDEwEKVFPiL/4OPKHLs8c8lniYc/MNjIbjhAQIvhjWRmw7PbkostZ6cdADoszZiwlhnJbJKN25gTN9xObwBqOZxIjBb2n43b6hM33DwO0vKfsBZ5iWQ2xsZtQMNvsIEcdoCwFgOex8ZAhx1P3HAmLeHAGYNk45kEbWlPfPixcVt14objx4w/VFTYyfYRtAVVgQEB5WBbGohQNApGwSgYBSMcAABNA02WrfCLmAAAAABJRU5ErkJggg==","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":true,"prefix":"","firstName":"Annina","middleName":"","lastName":"Gutermann","suffix":""},{"id":627744824,"identity":"a4b1509c-eed9-4150-88bd-cf0e4ec5b36a","order_by":1,"name":"Alexey Serdyuk","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Alexey","middleName":"","lastName":"Serdyuk","suffix":""},{"id":627744825,"identity":"f271249b-1e25-455d-a8cc-b89cdc90a4da","order_by":2,"name":"Foivos Paraskevas","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Foivos","middleName":"","lastName":"Paraskevas","suffix":""},{"id":627744827,"identity":"5d5eba4e-1455-4500-9d8d-1f86e8805d3f","order_by":3,"name":"Hella Toto Kiesa","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Hella","middleName":"Toto","lastName":"Kiesa","suffix":""},{"id":627744829,"identity":"e3601964-10db-4a5d-b8ca-e0465e7be5a2","order_by":4,"name":"Fabian Lesniak","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Fabian","middleName":"","lastName":"Lesniak","suffix":""},{"id":627744831,"identity":"9e946d7a-98fc-4552-9051-f6555c3fae32","order_by":5,"name":"Jakob Schwarz","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Jakob","middleName":"","lastName":"Schwarz","suffix":""},{"id":627744832,"identity":"bd8c7bdc-3d8b-4a14-8e98-dcbee24c0a63","order_by":6,"name":"Michael Hartmann","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Hartmann","suffix":""},{"id":627744834,"identity":"5a6dd0e8-9b25-4902-a458-422c7fa966c9","order_by":7,"name":"Tanja Harbaum","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Tanja","middleName":"","lastName":"Harbaum","suffix":""},{"id":627744835,"identity":"935644f5-c237-47f6-8553-22b7430c1cf6","order_by":8,"name":"Juergen Becker","email":"","orcid":"","institution":"Karlsruhe Institute of Technology (KIT)","correspondingAuthor":false,"prefix":"","firstName":"Juergen","middleName":"","lastName":"Becker","suffix":""}],"badges":[],"createdAt":"2026-03-20 17:38:30","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9181151/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9181151/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107707206,"identity":"c9a500fe-5791-4fd6-8465-16498b9f3204","added_by":"auto","created_at":"2026-04-24 09:19:48","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":405346,"visible":true,"origin":"","legend":"","description":"","filename":"NorCASJournalExtension.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9181151/v1_covered_7c56cfda-a157-44e4-b902-36744a625d36.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Joint Exploration of Neural Networks and Systolic Hardware for Improved AI Accelerator Performance","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Performance Predictors, Neural Architecture Search, Hardware Acceleration, Search Space Design","lastPublishedDoi":"10.21203/rs.3.rs-9181151/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9181151/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWhen executing common \\glspl{nn} on custom AI accelerators, the high performance suggested by advertised Giga or Tera Operations per Second (GOPS/TOPS) is typically not achieved, as low hardware utilization often leads to an effective performance in the single-digit percentage range of the theoretical peak.This discrepancy arises as NNs are typically designed without accounting for the target hardware, leading to inefficient mappings and software optimizations that fail to deliver the expected gains.Addressing this, we present a hardware-aware workflow that integrates accurate latency modeling and design space exploration to optimize both neural network architectures and the underlying systolic-array-based accelerator.We develop and validate two high-precision latency models for two different \\gls{rs} dataflows on our target accelerator.Using these models together with a structured search space generation, we generate Pareto-optimal search spaces in terms of achieved GOPS and latency for a given hardware target, and use these for a Bayesian \\acrlong{hwnas}. We further explore the accelerator design itself, varying the PE array dimensions and clock ratio to identify hardware configurations that maximize efficiency and minimize inference latency for each network and dataflow.We demonstrate our approach on ResNet-like networks. On the original hardware, the discovered ResNet-50-like architecture achieves an 85% relative increase in hardware utilization, reduces latency by 21% and parameter count by 18%, and maintains baseline ImageNet accuracy.On the jointly optimized hardware, latency and area are further reduced while efficiency is increased by up to 94%, with up to 20% fewer PEs compared to the baseline configuration. For ResNet-34, similar trends are observed, with latency reductions exceeding 33% and efficiency gains up to 43%.To enable reproducibility, we open-source our complete workflow of latency models, search space generation, NAS and training pipeline.\u003c/p\u003e","manuscriptTitle":"Joint Exploration of Neural Networks and Systolic Hardware for Improved AI Accelerator Performance","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-24 05:37:28","doi":"10.21203/rs.3.rs-9181151/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"889db922-02c0-4fc2-b454-926f01c1f958","owner":[],"postedDate":"April 24th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-24T05:37:28+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-24 05:37:28","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9181151","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9181151","identity":"rs-9181151","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}