The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach

The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach Mohammed Elshambakey, Aya I. Maiyza, Mona S. Kashkoush, Ghada M. Fathy, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3917850/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 17 Apr, 2024 Read the published version in The Journal of Supercomputing → Version 1 posted 7 You are reading this latest preprint version Abstract Recently, Egypt has recognized the pivotal role of High-Performance Computing in advancing science and innovation. The Egyptian National High Performance Computing Grid (EN-HPCG) advocates for consolidating high-performance computing resources, creating a unified pool accessible to all universities and scientific research centers using the PBS Pro scheduler. Therefore, this paper delves into the architecture and capabilities of the EN-HPCG grid using two different workload management systems: (i) Slurm (Open-Source) and (ii) PBS Pro (Licensed). Additionally, it assesses the grid's performance in specific high-throughput computing (HTC) applications using the NAS Grid parallel benchmark (NGB). The evaluation includes grid-level performance metrics such as throughput, and the number of tasks completed as a function of time. Also, the presented methodology aims to assist potential partners in their decision-making process to join the EN-HPCG grid, with a focus on the site speed-up metric. Our results showed that it is not advisable to integrate a cluster with high-speed hardware with a cluster possessing outdated hardware when using the Slurm scheduler. In contrast, the PBS Pro scheduler takes into account online decision-making in a dynamic environment using a unified grid. High-performance computing (HPC) Grid NGB benchmark HPC cluster speedup ·grid performance metrics EN-HPCG Slurm PBS scheduler HPC grid profiling Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 17 Apr, 2024 Read the published version in The Journal of Supercomputing → Version 1 posted Editorial decision: Revision requested 22 Feb, 2024 Reviews received at journal 12 Feb, 2024 Reviewers agreed at journal 02 Feb, 2024 Reviewers invited by journal 02 Feb, 2024 Editor assigned by journal 02 Feb, 2024 Submission checks completed at journal 02 Feb, 2024 First submitted to journal 01 Feb, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3917850","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":270990427,"identity":"d277b197-8952-4e82-ade2-6c12b6956c9f","order_by":0,"name":"Mohammed Elshambakey","email":"","orcid":"","institution":"Informatics Research institute, City of Scientific Research and Technological Applications (SRTA-City)","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"","lastName":"Elshambakey","suffix":""},{"id":270990428,"identity":"e3b16cd9-1e2f-4cf5-84c8-3016442bf0af","order_by":1,"name":"Aya I. Maiyza","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCElEQVRIiWNgGAWjYJCCA0AsB2EaIEQIajFmYGAGaTEgTgsIJDaAtcCswQf4px1+eJjnj136/NnnD374UfAnsYH/8MMDDBW1OLVI3E4zOMzDk5y74Vwys2SPgUFig0SawQGGM8dxW3M7AahFgjl3Aw8zgzQDWAsPwwHGtmM4dcjfTv9wmMegPl2+h5n5N1gL/xmgln+4tRjczgHaknA4geEMMxvEFoYcoJaGGpxaDG/nFBycc+C44YYzzGaWPQbGxm0gvyQcO4BTi9zt9M0f3vyplpfvYXx848cfOdl+/sOPP3yoqcPtfSBg4kHiOLaByASGw3i1MP5A4thDafy2jIJRMApGwYgCAOqFVsp7ENerAAAAAElFTkSuQmCC","orcid":"","institution":"Informatics Research institute, City of Scientific Research and Technological Applications (SRTA-City)","correspondingAuthor":true,"prefix":"","firstName":"Aya","middleName":"I.","lastName":"Maiyza","suffix":""},{"id":270990429,"identity":"acb2abc3-ade3-427a-8497-fb68c01b7409","order_by":2,"name":"Mona S. Kashkoush","email":"","orcid":"","institution":"Informatics Research institute, City of Scientific Research and Technological Applications (SRTA-City)","correspondingAuthor":false,"prefix":"","firstName":"Mona","middleName":"S.","lastName":"Kashkoush","suffix":""},{"id":270990430,"identity":"e46703d1-5e16-4f11-8210-e19129573d06","order_by":3,"name":"Ghada M. Fathy","email":"","orcid":"","institution":"Informatics Research institute, City of Scientific Research and Technological Applications (SRTA-City)","correspondingAuthor":false,"prefix":"","firstName":"Ghada","middleName":"M.","lastName":"Fathy","suffix":""},{"id":270990431,"identity":"d00e5b47-960d-4b1a-ad16-90c5b896e048","order_by":4,"name":"Hanan A. Hassan","email":"","orcid":"","institution":"Informatics Research institute, City of Scientific Research and Technological Applications (SRTA-City)","correspondingAuthor":false,"prefix":"","firstName":"Hanan","middleName":"A.","lastName":"Hassan","suffix":""}],"badges":[],"createdAt":"2024-02-01 15:03:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3917850/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3917850/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11227-024-06041-9","type":"published","date":"2024-04-17T15:33:13+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":58510244,"identity":"062dfe09-1d5b-4617-8785-8ea7b9e3ff9c","added_by":"auto","created_at":"2024-06-17 15:33:19","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1395153,"visible":true,"origin":"","legend":"","description":"","filename":"ENHPCG.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3917850/v1_covered_3be72dc7-1bf1-4397-ba2c-98b1bd9ad6de.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach\n\n\n","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"High-performance computing (HPC), Grid, NGB benchmark, HPC cluster speedup ·grid performance metrics, EN-HPCG, Slurm, PBS scheduler, HPC grid profiling","lastPublishedDoi":"10.21203/rs.3.rs-3917850/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3917850/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recently, Egypt has recognized the pivotal role of High-Performance Computing in advancing science and innovation. The Egyptian National High Performance Computing Grid (EN-HPCG) advocates for consolidating high-performance computing resources, creating a unified pool accessible to all universities and scientific research centers using the PBS Pro scheduler. Therefore, this paper delves into the architecture and capabilities of the EN-HPCG grid using two different workload management systems: (i) Slurm (Open-Source) and (ii) PBS Pro (Licensed). Additionally, it assesses the grid's performance in specific high-throughput computing (HTC) applications using the NAS Grid parallel benchmark (NGB). The evaluation includes grid-level performance metrics such as throughput, and the number of tasks completed as a function of time. Also, the presented methodology aims to assist potential partners in their decision-making process to join the EN-HPCG grid, with a focus on the site speed-up metric. Our results showed that it is not advisable to integrate a cluster with high-speed hardware with a cluster possessing outdated hardware when using the Slurm scheduler. In contrast, the PBS Pro scheduler takes into account online decision-making in a dynamic environment using a unified grid.","manuscriptTitle":"The Egyptian National HPC Grid (EN-HPCG): Open-Source Slurm Implementation from Cluster to Grid Approach","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-02-05 06:26:12","doi":"10.21203/rs.3.rs-3917850/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-02-22T19:44:51+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-02-12T08:12:18+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"675666a2-f939-46db-9703-bda9a715aeed","date":"2024-02-03T04:50:01+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-02-02T20:09:43+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-02-02T07:17:08+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-02-02T07:17:08+00:00","index":"","fulltext":""},{"type":"submitted","content":"The Journal of Supercomputing","date":"2024-02-01T14:54:16+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"e890ed88-672b-4336-b611-015b08fabef1","owner":[],"postedDate":"February 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-06-17T15:33:13+00:00","versionOfRecord":{"articleIdentity":"rs-3917850","link":"https://doi.org/10.1007/s11227-024-06041-9","journal":{"identity":"the-journal-of-supercomputing","isVorOnly":false,"title":"The Journal of Supercomputing"},"publishedOn":"2024-04-17 15:33:13","publishedOnDateReadable":"April 17th, 2024"},"versionCreatedAt":"2024-02-05 06:26:12","video":"","vorDoi":"10.1007/s11227-024-06041-9","vorDoiUrl":"https://doi.org/10.1007/s11227-024-06041-9","workflowStages":[]},"version":"v1","identity":"rs-3917850","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3917850","identity":"rs-3917850","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}