{"paper_id":"2148c490-c06e-4b40-a854-649a2cf2803b","body_text":"Evaluating ASCERT: Generative AI for Cyber-Range Scenario Generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluating ASCERT: Generative AI for Cyber-Range Scenario Generation M. Palumickas, M. Mudassar Yamin, B. Katt, Chhagan Lal This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7806552/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 15 Dec, 2025 Read the published version in International Journal of Information Security → Version 1 posted 9 You are reading this latest preprint version Abstract In this paper, we worked in collaboration with the ASCERT (AI-based scenario management for cyber-range training) project and its generative AI prototype that generates dynamic and interactive yber-range exercise scenarios. We evaluate the model by focusing on two objectives: (i) its ability to replicate real-world cyber attacks, and (ii) its consistency across multiple simulations that uses same inputs. To assess realism, we examine how well the model reproduces three well-documented cyber incidents namely Colonial Pipeline, Equifax, and SolarWinds, when it is provided with relevant source material for training. We then analyze repeatability by comparing outputs across fixed-input simulation runs. As the evaluation results indicate, overall the model generated varied and context- appropriate scenarios. Moreover, it introduced an interactivity feature that allows users to choose responses and observe consequences in real time. However, the consistency in repeated runs was limited: simulations are not reliably repeatable, although what was interesting is that the variability reflects the unpredictability of real attacks. These findings suggest that, while ASCERT already supports scenario variety and meaningful user interaction, it requires targeted refinements to improve stability and repeatability. With such improvements, the ASCERT model has strong potential to contribute to scalable and adaptive cybersecurity education and training. Generative AI Cybersecurity Scenario Generation Cyber Range Simulation Cyber Attack Simulation Automation Large Language Models (LLMs) Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 15 Dec, 2025 Read the published version in International Journal of Information Security → Version 1 posted Editorial decision: Revision requested 23 Oct, 2025 Reviews received at journal 19 Oct, 2025 Reviews received at journal 15 Oct, 2025 Reviewers agreed at journal 13 Oct, 2025 Reviewers agreed at journal 10 Oct, 2025 Reviewers invited by journal 10 Oct, 2025 Editor assigned by journal 10 Oct, 2025 Submission checks completed at journal 09 Oct, 2025 First submitted to journal 08 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-7806552\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":false,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":533524207,\"identity\":\"99ea388b-3b87-42b4-af97-670eb816799c\",\"order_by\":0,\"name\":\"M. Palumickas\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Norwegian University of Science and Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"M.\",\"middleName\":\"\",\"lastName\":\"Palumickas\",\"suffix\":\"\"},{\"id\":533524208,\"identity\":\"3c18b3df-637f-4550-8e9a-44821f711f86\",\"order_by\":1,\"name\":\"M. Mudassar Yamin\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Norwegian University of Science and Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"M.\",\"middleName\":\"Mudassar\",\"lastName\":\"Yamin\",\"suffix\":\"\"},{\"id\":533524209,\"identity\":\"0fea765a-2a1d-4e6c-9670-38a0dbc7a8a8\",\"order_by\":2,\"name\":\"B. Katt\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Norwegian University of Science and Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"B.\",\"middleName\":\"\",\"lastName\":\"Katt\",\"suffix\":\"\"},{\"id\":533524210,\"identity\":\"0848280f-ac4a-4649-95f7-3fb16676c13d\",\"order_by\":3,\"name\":\"Chhagan Lal\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA00lEQVRIiWNgGAWjYDACCR4IzcfA3PiANC1sDIzNBgwJJGppkyBKC//s3oMfv9TcYWBjP9hWXfjjTmKDdPMB/JbcOZcsLXPsGQMbT2Lb7RkJzxIbZI7ht8tAIsdAWoLtMNBhQC08CYcTG4AihLQY/5b4B9TC/7CtmFgtZpIf24BaJBLbmInSInEjx8yase8wD5vEw2ZpnrTDxm0Safj9wj8jx/jmj2+H5fj5kw9+5rE5LNsvkXwArxYQYAZGDQ+cx0ZQPRAw/iBG1SgYBaNgFIxcAABOMEIJh1RS0AAAAABJRU5ErkJggg==\",\"orcid\":\"\",\"institution\":\"Norwegian University of Science and Technology\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Chhagan\",\"middleName\":\"\",\"lastName\":\"Lal\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-10-08 09:53:40\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-7806552/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-7806552/v1\",\"draftVersion\":[],\"editorialEvents\":[{\"content\":\"https://doi.org/10.1007/s10207-025-01179-w\",\"type\":\"published\",\"date\":\"2025-12-15T15:57:34+00:00\"}],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":94446970,\"identity\":\"4e0831a1-2652-48e2-a640-302a8ad724ba\",\"added_by\":\"auto\",\"created_at\":\"2025-10-27 14:35:11\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":632890,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"IJISPaper.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7806552/v1/74cd5b696fe9158c36750874.pdf\"},{\"id\":94446820,\"identity\":\"35fa82f2-747d-4e8c-af89-721b5757b860\",\"added_by\":\"auto\",\"created_at\":\"2025-10-27 14:35:04\",\"extension\":\"json\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":5519,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"24815bb47cbd4d49b109a8e161897e72.json\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7806552/v1/6002194b5aad74f04abf80c8.json\"},{\"id\":98813903,\"identity\":\"99fb6ee1-8942-484b-9857-88fb083d8933\",\"added_by\":\"auto\",\"created_at\":\"2025-12-22 16:07:21\",\"extension\":\"pdf\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":538037,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"IJISPaper.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7806552/v1_covered_3e18dc2f-5950-452c-af02-4527ff5b7646.pdf\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"Evaluating ASCERT: Generative AI for Cyber-Range Scenario Generation\",\"fulltext\":[],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":false,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":false,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":true,\"isAuthorSuppliedPdf\":true,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":true,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"international-journal-of-information-security\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"ijis\",\"sideBox\":\"Learn more about [International Journal of Information Security](http://link.springer.com/journal/10207)\",\"snPcode\":\"10207\",\"submissionUrl\":\"https://submission.nature.com/new-submission/10207/3\",\"title\":\"International Journal of Information Security\",\"twitterHandle\":\"\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"Springer Hybrid\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false},\"keywords\":\"Generative AI, Cybersecurity, Scenario Generation, Cyber Range, Simulation, Cyber Attack Simulation, Automation, Large Language Models (LLMs)\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-7806552/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-7806552/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"In this paper, we worked in collaboration with the ASCERT (AI-based scenario management for cyber-range training) project and its generative AI prototype that generates dynamic and interactive yber-range exercise scenarios. We evaluate the model by focusing on two objectives: (i) its ability to replicate real-world cyber attacks, and (ii) its consistency across multiple simulations that uses same inputs. To assess realism, we examine how well the model reproduces three well-documented cyber incidents namely Colonial Pipeline, Equifax, and SolarWinds, when it is provided with relevant source material for training. We then analyze repeatability by comparing outputs across fixed-input simulation runs. As the evaluation results indicate, overall the model generated varied and context- appropriate scenarios. Moreover, it introduced an interactivity feature that allows users to choose responses and observe consequences in real time. However, the consistency in repeated runs was limited: simulations are not reliably repeatable, although what was interesting is that the variability reflects the unpredictability of real attacks. These findings suggest that, while ASCERT already supports scenario variety and meaningful user interaction, it requires targeted refinements to improve stability and repeatability. With such improvements, the ASCERT model has strong potential to contribute to scalable and adaptive cybersecurity education and training.\",\"manuscriptTitle\":\"Evaluating ASCERT: Generative AI for Cyber-Range Scenario Generation\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-10-27 11:16:33\",\"doi\":\"10.21203/rs.3.rs-7806552/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0},{\"type\":\"decision\",\"content\":\"Revision requested\",\"date\":\"2025-10-23T09:09:32+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2025-10-19T20:09:42+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2025-10-15T14:52:25+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"226337677974588353016273537745326898401\",\"date\":\"2025-10-13T06:47:56+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"331474893627676439872924551873033616664\",\"date\":\"2025-10-10T07:13:45+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewersInvited\",\"content\":\"\",\"date\":\"2025-10-10T07:07:34+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorAssigned\",\"content\":\"\",\"date\":\"2025-10-10T06:52:03+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"checksComplete\",\"content\":\"\",\"date\":\"2025-10-09T15:51:32+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"submitted\",\"content\":\"International Journal of Information Security\",\"date\":\"2025-10-08T09:51:08+00:00\",\"index\":\"\",\"fulltext\":\"\"}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"international-journal-of-information-security\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"ijis\",\"sideBox\":\"Learn more about [International Journal of Information Security](http://link.springer.com/journal/10207)\",\"snPcode\":\"10207\",\"submissionUrl\":\"https://submission.nature.com/new-submission/10207/3\",\"title\":\"International Journal of Information Security\",\"twitterHandle\":\"\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"Springer Hybrid\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false}}],\"origin\":\"\",\"ownerIdentity\":\"1b291d35-d5bc-4930-b6f7-a83a9ff79f84\",\"owner\":[],\"postedDate\":\"October 27th, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"published-in-journal\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2025-12-22T16:00:49+00:00\",\"versionOfRecord\":{\"articleIdentity\":\"rs-7806552\",\"link\":\"https://doi.org/10.1007/s10207-025-01179-w\",\"journal\":{\"identity\":\"international-journal-of-information-security\",\"isVorOnly\":false,\"title\":\"International Journal of Information Security\"},\"publishedOn\":\"2025-12-15 15:57:34\",\"publishedOnDateReadable\":\"December 15th, 2025\"},\"versionCreatedAt\":\"2025-10-27 11:16:33\",\"video\":\"\",\"vorDoi\":\"10.1007/s10207-025-01179-w\",\"vorDoiUrl\":\"https://doi.org/10.1007/s10207-025-01179-w\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-7806552\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-7806552\",\"identity\":\"rs-7806552\",\"version\":[\"v1\"]},\"buildId\":\"XKTyCvWXoU3ODBz1xrDgd\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}