Enhancing HPC Job Run Time Predictions leveraging Machine Learning, Historical Job Data, and Metaheuristic Optimization

preprint OA: closed CC-BY-4.0
📄 Open PDF Full text JSON View at publisher

Abstract

Abstract In High Performance Computing (HPC) systems that rely on job schedulers for resource allocation, accurate predictions of job run times is critical for efficient resource utilization. This study attempts to develop data-driven machine learning (ML) models for predicting job run times using three algorithms - Light Gradient Boosting (LGB), Deep Neural Networks (DNN), and Extreme Gradient Boosting (XGB). Unlike approaches that rely solely on job metadata, this work incorporates historical run times of previously executed similar jobs as features, capturing user-specific behavioral patterns and analyzes their impact in improving prediction accuracy. Two metaheuristic algorithms - Genetic Algorithm and Whale Optimization Algorithm are leveraged to optimize the performance of models by selecting a relevant feature subset, hyperparameters and pre-processing techniques. The algorithms employ a sequential selection strategy for historical features, striving to achieve a balance between prediction accuracy and computational cost of extracting the features. Baseline versions of ML models are implemented with Bayesian hyperparameter optimization and embedded feature selection techniques (Ridge Regression and Random Forest). The models are validated using four datasets collected from multiple, heterogeneous HPC systems to ensure adaptability to diverse HPC configurations and workloads. The results show that ML models optimized by metaheuristics consistently outperform baseline models. Optimized XGB models achieved the highest prediction accuracy while using fewer historical features. This work underscores the significance of integrating ML models, historical data and intelligent optimization techniques in developing accurate, efficient and generalized HPC job run time prediction models.
Full text 24,049 characters · extracted from preprint-html · click to expand
Enhancing HPC Job Run Time Predictions leveraging Machine Learning, Historical Job Data, and Metaheuristic Optimization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Enhancing HPC Job Run Time Predictions leveraging Machine Learning, Historical Job Data, and Metaheuristic Optimization Suja Ramachandran, M. L. Jayalal, M. Vasudevan, R. Jehadeesan This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8256818/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In High Performance Computing (HPC) systems that rely on job schedulers for resource allocation, accurate predictions of job run times is critical for efficient resource utilization. This study attempts to develop data-driven machine learning (ML) models for predicting job run times using three algorithms - Light Gradient Boosting (LGB), Deep Neural Networks (DNN), and Extreme Gradient Boosting (XGB). Unlike approaches that rely solely on job metadata, this work incorporates historical run times of previously executed similar jobs as features, capturing user-specific behavioral patterns and analyzes their impact in improving prediction accuracy. Two metaheuristic algorithms - Genetic Algorithm and Whale Optimization Algorithm are leveraged to optimize the performance of models by selecting a relevant feature subset, hyperparameters and pre-processing techniques. The algorithms employ a sequential selection strategy for historical features, striving to achieve a balance between prediction accuracy and computational cost of extracting the features. Baseline versions of ML models are implemented with Bayesian hyperparameter optimization and embedded feature selection techniques (Ridge Regression and Random Forest). The models are validated using four datasets collected from multiple, heterogeneous HPC systems to ensure adaptability to diverse HPC configurations and workloads. The results show that ML models optimized by metaheuristics consistently outperform baseline models. Optimized XGB models achieved the highest prediction accuracy while using fewer historical features. This work underscores the significance of integrating ML models, historical data and intelligent optimization techniques in developing accurate, efficient and generalized HPC job run time prediction models. HPC Machine Learning Feature Selection Genetic Algorithm Whale Optimization Algorithm Full Text Additional Declarations No competing interests reported. Supplementary Files ClusterAdata.txt Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8256818","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":559610891,"identity":"3737fe0d-60fd-4735-ac4d-1f33c82d6965","order_by":0,"name":"Suja Ramachandran","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA40lEQVRIiWNgGAWjYHCCBBAhx4YQYGwgSosxSVrAIJE4ZSAgPyPh4eeKmm3pffyLgQyGWjmDA8xtD/BpMbiRkCx55tjt3DaJZ8aSZxiOGxscYGw3wKtFIiFBsoENpOUMg2QDw7HEmQ2MbRIEHJb8s+Hf7XQ2iTPMP4nSwnAjIU2yse12Aht/DxvQlprEfgYCWgzOPEizbOy7bdgmwWZm2WBwwJifmZDD2nOSbzZ8uy0v33/48c2Gijo5Nvb2Z/gdxsCTAKElQLTBYQYGZvzqgYD9AITmB9N1BNWPglEwCkbByAMAJM9JAjBXdg4AAAAASUVORK5CYII=","orcid":"","institution":"Indira Gandhi Centre for Atomic Research, A CI of Homi Bhabha National Institute","correspondingAuthor":true,"prefix":"","firstName":"Suja","middleName":"","lastName":"Ramachandran","suffix":""},{"id":559610892,"identity":"f602de83-aee6-45f2-8744-b2c8b1db337d","order_by":1,"name":"M. L. Jayalal","email":"","orcid":"","institution":"Indira Gandhi Centre for Atomic Research, A CI of Homi Bhabha National Institute","correspondingAuthor":false,"prefix":"","firstName":"M.","middleName":"L.","lastName":"Jayalal","suffix":""},{"id":559610893,"identity":"fee612d0-4944-4b59-89e5-c27f99c761e5","order_by":2,"name":"M. Vasudevan","email":"","orcid":"","institution":"Indira Gandhi Centre for Atomic Research, A CI of Homi Bhabha National Institute","correspondingAuthor":false,"prefix":"","firstName":"M.","middleName":"","lastName":"Vasudevan","suffix":""},{"id":559610894,"identity":"308a47fe-0099-4daa-bd9c-bacb9657cd26","order_by":3,"name":"R. Jehadeesan","email":"","orcid":"","institution":"Indira Gandhi Centre for Atomic Research, A CI of Homi Bhabha National Institute","correspondingAuthor":false,"prefix":"","firstName":"R.","middleName":"","lastName":"Jehadeesan","suffix":""}],"badges":[],"createdAt":"2025-12-02 06:53:28","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8256818/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8256818/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98433247,"identity":"31718b73-78bc-4458-b47d-3ff02edfeed1","added_by":"auto","created_at":"2025-12-17 16:50:30","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3701643,"visible":true,"origin":"","legend":"","description":"","filename":"hpcmetamlv5.docx","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/33edbb26d4cd9e44d0b32d79.docx"},{"id":98434487,"identity":"a59943f7-d307-429d-a34d-78e7fbca35e3","added_by":"auto","created_at":"2025-12-17 16:52:10","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6431,"visible":true,"origin":"","legend":"","description":"","filename":"a90b704ac58b476bba2e13abdee6236e.json","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/a52280a86acd14b66e1772cc.json"},{"id":98433183,"identity":"81cb39e8-0445-49dd-b8bf-b68940664dc0","added_by":"auto","created_at":"2025-12-17 16:50:23","extension":"txt","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6728838,"visible":true,"origin":"","legend":"","description":"","filename":"ClusterAdata.txt","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/dc09e492cda4d1b549587716.txt"},{"id":98234618,"identity":"72703719-3154-4c02-8bbb-58a0954bb327","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":194004,"visible":true,"origin":"","legend":"","description":"","filename":"a90b704ac58b476bba2e13abdee6236e1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/96510ef5c7037c5bfe0b8f64.xml"},{"id":98432927,"identity":"a733503c-9ca9-4d4c-adef-f660fba33fce","added_by":"auto","created_at":"2025-12-17 16:50:07","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":521458,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/4b7eefd6fdefdb20f60e2c5e.png"},{"id":98234614,"identity":"57717d92-05ac-4210-bf27-3a6f6585e80a","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":402288,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/409e4f215a8ae49fcb1dee94.png"},{"id":98433500,"identity":"441022b0-3feb-41bd-ae0b-1f75198640d9","added_by":"auto","created_at":"2025-12-17 16:50:51","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":178578,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/13c0895891bd0d3662f56d85.png"},{"id":98234612,"identity":"15a33b8d-6bfc-4a84-aaa6-93bd97fbdfa9","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"jpeg","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":254236,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/faba41831b75b81a37f3575a.jpeg"},{"id":98432350,"identity":"e2fc3fbf-b114-459e-aae2-7b56499ee7ee","added_by":"auto","created_at":"2025-12-17 16:49:26","extension":"jpeg","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":139268,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/622fbdf8fff6ec9f3a52a87d.jpeg"},{"id":98433129,"identity":"d87297d1-f01a-447f-8378-774c817fbd8c","added_by":"auto","created_at":"2025-12-17 16:50:18","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":700236,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/a7f3146450e56b76b4b57520.png"},{"id":98234621,"identity":"30b506c1-d8a8-4289-8e26-65036834f147","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1224796,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/0aa997fad46c85e65be668d4.png"},{"id":98434185,"identity":"d8edd21f-58d5-46ce-a632-8ab6eed7d19f","added_by":"auto","created_at":"2025-12-17 16:51:39","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":411412,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/0dfdb28b1a1e1a94f6432e25.png"},{"id":98433120,"identity":"7fb839e2-3725-4f97-aa18-16fe23252f60","added_by":"auto","created_at":"2025-12-17 16:50:17","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":336462,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/ba9e6301d66ac0e48ac8d6be.png"},{"id":98432966,"identity":"9752f543-3687-43a9-ab6e-5120cffb5150","added_by":"auto","created_at":"2025-12-17 16:50:09","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":185791,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/47003deed8dcc23763b756ed.png"},{"id":98432474,"identity":"113e0a62-eb96-451c-a773-bb91527cbf34","added_by":"auto","created_at":"2025-12-17 16:49:35","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":409566,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/20eb26b57d029076ac47ca12.png"},{"id":98433074,"identity":"2cb720c0-81e7-4024-aad1-092e0fcd4db5","added_by":"auto","created_at":"2025-12-17 16:50:13","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":155124,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/285590c959af4a18a4d4dc43.png"},{"id":98234639,"identity":"7e216e3c-a2fd-49b7-b807-8c802e94fd41","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":173302,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/0f71190d57a1fd647aca1a0e.png"},{"id":98234631,"identity":"24015463-bb25-4595-8855-65409cefb9a8","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":70040,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/e956501b64c494883ea6ef83.png"},{"id":98432170,"identity":"8a582499-085a-448b-98e6-7e78e0e3867b","added_by":"auto","created_at":"2025-12-17 16:49:09","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49632,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/67f705035aecaca5e879eb73.png"},{"id":98234636,"identity":"50f1dfb4-ba76-4226-bd3e-396bf17c4416","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":27436,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/b9cd53e9d725315f45686dc5.png"},{"id":98432168,"identity":"51832c2e-6683-4ed0-ba58-7be2da7517f9","added_by":"auto","created_at":"2025-12-17 16:49:09","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":330201,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/97f75b8e9bbee9a925e7c2cb.png"},{"id":98433144,"identity":"e4826260-989f-48e0-8c2b-6a435d5e71f2","added_by":"auto","created_at":"2025-12-17 16:50:20","extension":"png","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":492823,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/aea9703a8e8715ba2b96364b.png"},{"id":98432337,"identity":"30a0f43f-7058-4286-bb3c-159c13f8c78d","added_by":"auto","created_at":"2025-12-17 16:49:25","extension":"png","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":195205,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/1fc6714aee719b3716dc6202.png"},{"id":98434301,"identity":"12698391-114f-4639-b8c2-36524551b669","added_by":"auto","created_at":"2025-12-17 16:51:50","extension":"png","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":143415,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/a18f33ef424adb4aab09aca6.png"},{"id":98433604,"identity":"f9864f5b-492c-44a9-af76-9be33886c174","added_by":"auto","created_at":"2025-12-17 16:50:56","extension":"png","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":70755,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/576ff3e52c6ef7ab4e32a61a.png"},{"id":98234632,"identity":"5d0d4f0f-b2a6-4505-8fc5-cd10637d1a38","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"png","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":174002,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/18abffd98878c4465bb46c8a.png"},{"id":98433158,"identity":"91b41a1e-aef3-4aa3-9969-7c6579861c56","added_by":"auto","created_at":"2025-12-17 16:50:21","extension":"xml","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":195064,"visible":true,"origin":"","legend":"","description":"","filename":"a90b704ac58b476bba2e13abdee6236e1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/727876b38e4682b69ddd6e87.xml"},{"id":98234633,"identity":"73a00676-903c-4aad-88ab-42467175184e","added_by":"auto","created_at":"2025-12-15 14:10:30","extension":"html","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":210707,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/5cfea23c9dd1ed713db4d7d9.html"},{"id":108740032,"identity":"fcbe227a-8d62-43e1-8b71-41a35137fc3b","added_by":"auto","created_at":"2026-05-07 22:24:13","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1334337,"visible":true,"origin":"","legend":"","description":"","filename":"hpcmetamlv5.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1_covered_4a384dca-a960-4541-8075-f4f1b3dbeb5e.pdf"},{"id":98434433,"identity":"2714ec60-6609-46d5-ad78-3e00e6809394","added_by":"auto","created_at":"2025-12-17 16:52:06","extension":"txt","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":6728838,"visible":true,"origin":"","legend":"","description":"","filename":"ClusterAdata.txt","url":"https://assets-eu.researchsquare.com/files/rs-8256818/v1/165584ff7ac47872352d36c5.txt"}],"financialInterests":"No competing interests reported.","formattedTitle":"Enhancing HPC Job Run Time Predictions leveraging Machine Learning, Historical Job Data, and Metaheuristic Optimization","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"HPC, Machine Learning, Feature Selection, Genetic Algorithm, Whale Optimization Algorithm","lastPublishedDoi":"10.21203/rs.3.rs-8256818/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8256818/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn High Performance Computing (HPC) systems that rely on job schedulers for resource allocation, accurate predictions of job run times is critical for efficient resource utilization. This study attempts to develop data-driven machine learning (ML) models for predicting job run times using three algorithms - Light Gradient Boosting (LGB), Deep Neural Networks (DNN), and Extreme Gradient Boosting (XGB). Unlike approaches that rely solely on job metadata, this work incorporates historical run times of previously executed similar jobs as features, capturing user-specific behavioral patterns and analyzes their impact in improving prediction accuracy. Two metaheuristic algorithms - Genetic Algorithm and Whale Optimization Algorithm are leveraged to optimize the performance of models by selecting a relevant feature subset, hyperparameters and pre-processing techniques. The algorithms employ a sequential selection strategy for historical features, striving to achieve a balance between prediction accuracy and computational cost of extracting the features. Baseline versions of ML models are implemented with Bayesian hyperparameter optimization and embedded feature selection techniques (Ridge Regression and Random Forest). The models are validated using four datasets collected from multiple, heterogeneous HPC systems to ensure adaptability to diverse HPC configurations and workloads. The results show that ML models optimized by metaheuristics consistently outperform baseline models. Optimized XGB models achieved the highest prediction accuracy while using fewer historical features. This work underscores the significance of integrating ML models, historical data and intelligent optimization techniques in developing accurate, efficient and generalized HPC job run time prediction models.\u003c/p\u003e","manuscriptTitle":"Enhancing HPC Job Run Time Predictions leveraging Machine Learning, Historical Job Data, and Metaheuristic Optimization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-15 14:10:25","doi":"10.21203/rs.3.rs-8256818/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"403b26b0-6876-4b3e-8c47-e7ad7c7c6e63","owner":[],"postedDate":"December 15th, 2025","published":true,"recentEditorialEvents":[{"type":"decision","content":"Withdrawn","date":"2026-05-07T22:15:46+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-07T22:24:01+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-15 14:10:25","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8256818","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8256818","identity":"rs-8256818","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}

Text is read by the "Ask this paper" AI Q&A widget below. Extraction quality varies by source — PMC NXML preserves structure cleanly, OA-HTML may include some navigation residue, and OA-PDF can have broken hyphenation. The publisher copy (via DOI) is the canonical version.

My notes (saved in your browser only)

Ask this paper AI returns verbatim quotes from the full text · source: preprint-html

Answers must be backed by verbatim quotes from this paper's full text. Hallucinated quotes are dropped automatically; if no verbatim passage answers the question, we say so. How this works

Citation neighborhood (no data yet)

We don't have any in-corpus citations linked to this paper yet. This is a recent paper (2025) — citers typically take a year or two to land, and the OpenAlex reference graph may still be filling in.

Source provenance

europepmc
last seen: 2026-05-20T01:45:00.602351+00:00
unpaywall
last seen: 2026-05-24T02:00:01.246996+00:00
License: CC-BY-4.0