Transformers as a classifier for solar flare time series: a comparative study

Transformers as a classifier for solar flare time series: a comparative study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Transformers as a classifier for solar flare time series: a comparative study Juliana Sabino Ferreira, André Leon Sampaio Gradvohl, Ana Estela Antunes da Silva, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4093277/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Solar flares are violent and sudden eruptions that occur in the solar atmosphere and release energy in the form of radiation. They can affect technological systems on Earth and in its orbit, causing financial losses and damage to human life. Therefore, it is necessary to predict the occurrence of such flares to mitigate their effects. Specialized instruments gather data for solar activity monitoring. Hence, we can create prediction models using machine learning from this data. From an analysis of the literature, we noticed the prevalence of some algorithms, such as Multi-layer Perceptrons (MLP), Support Vector Machines (SVM), and Long Short-Term Memory (LSTM), which presented good results, mainly considering the True Skill Statistic (TSS) metric. In parallel, in 2017, a new deep learning-based neural network architecture called Transformers emerged. Researchers initially created it for natural language processing. However, Transformers were successfully employed in other domains, such as time series forecasting. Solar activity data is considered a time series due to its continuous capture over time. Consequently, we can employ Transformers to develop a solar flare forecast model. Considering a significant lack of work using Transformers for solar flare forecasting, we ran experiments to test the Transformers' viability and performance in solar flare forecast models. We created models using other algorithms (MLP, SVM, LSTM, Transformers) to investigate the Transformers' performance and compared them using accuracy, TSS, and Area Under the ROC Curve (AUC) metrics. We observed that the Transformers had superior performance compared to the other models. For instance, the Transformers' TSS metric average was 0.9, contrasting the other models' TSS=0.4. The difference was slightly smaller in AUC, where Transformers reached 0.9, and the others reached no more than 0.7. Therefore, we can use the Transformers to classify solar flare data and obtain superior results compared to other models. We also conducted experiments using different forms of data balancing, including unbalanced data, balanced with undersampling, oversampling, and SMOTE techniques. The MLP, SVM, and LSTM models showed significant improvements in balance, where the average TSS increased from 0.1 to 0.4. On the other hand, Transformers were not sensitive to data balancing, presenting the most stable TSS in all cases. Solar flares Machine Learning Forecasting Imbalanced dataset Transformer Architecture Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4093277","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":279591447,"identity":"4a4b19a8-2f60-4620-ad3d-7403d88be4c8","order_by":0,"name":"Juliana Sabino Ferreira","email":"","orcid":"","institution":"Federal Institute of São Paulo","correspondingAuthor":false,"prefix":"","firstName":"Juliana","middleName":"Sabino","lastName":"Ferreira","suffix":""},{"id":279591448,"identity":"863c2741-7f8d-43e3-bf9d-4de551f37808","order_by":1,"name":"André Leon Sampaio Gradvohl","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1klEQVRIiWNgGAWjYJACCQaGAzL8QMYBGJcoLTySDSRrMTiAxMUL5KMPP7zNU3GHx/hG7sEDP2q25TFI9z7Aq8XwXJqxNc+ZZzxmN/ISDvYcu13MIHPcAL+WHgYzyZlth4FacgwOM7DdTmyQSMPvMMMe9m9gLcYzQFr+EaFFnofHTOIjUIuBBFALYxsRWgx4eIotPpw5zCNx5o3Bwd6+28VsMscI2NLDvvFGQsVhOf72HOMPP77dzuOXbiNgywE0gQQ2/BqAtjSgayGkYxSMglEwCkYeAAAW+0lqDh7KrwAAAABJRU5ErkJggg==","orcid":"","institution":"State University of Campinas","correspondingAuthor":true,"prefix":"","firstName":"André","middleName":"Leon Sampaio","lastName":"Gradvohl","suffix":""},{"id":279591449,"identity":"8638a198-65b2-4103-bd4b-696e00a1fbae","order_by":2,"name":"Ana Estela Antunes da Silva","email":"","orcid":"","institution":"State University of Campinas","correspondingAuthor":false,"prefix":"","firstName":"Ana","middleName":"Estela Antunes da","lastName":"Silva","suffix":""},{"id":279591450,"identity":"3e280d83-5bc2-4e29-b150-81af1695ee4d","order_by":3,"name":"Guilherme Palermo Coelho","email":"","orcid":"","institution":"State University of Campinas","correspondingAuthor":false,"prefix":"","firstName":"Guilherme","middleName":"Palermo","lastName":"Coelho","suffix":""},{"id":279591451,"identity":"cd7e105b-4b01-4ebb-9027-6ec3d438e0bd","order_by":4,"name":"Tiago Cinto","email":"","orcid":"","institution":"Instituto Federal Sul-rio-grandense","correspondingAuthor":false,"prefix":"","firstName":"Tiago","middleName":"","lastName":"Cinto","suffix":""}],"badges":[],"createdAt":"2024-03-13 13:16:56","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4093277/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4093277/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":54384252,"identity":"eb73253d-c1b0-448b-b53e-2e72318d1ce0","added_by":"auto","created_at":"2024-04-09 16:58:20","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1089057,"visible":true,"origin":"","legend":"","description":"","filename":"SolarPhysicsTransformersRobusteness.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4093277/v1_covered_f019010b-95f1-4d03-ba12-9a81082a322c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Transformers as a classifier for solar flare time series: a comparative study","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Solar flares, Machine Learning, Forecasting, Imbalanced dataset, Transformer Architecture","lastPublishedDoi":"10.21203/rs.3.rs-4093277/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4093277/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Solar flares are violent and sudden eruptions that occur in the solar atmosphere and release energy in the form of radiation. They can affect technological systems on Earth and in its orbit, causing financial losses and damage to human life. Therefore, it is necessary to predict the occurrence of such flares to mitigate their effects. Specialized instruments gather data for solar activity monitoring. Hence, we can create prediction models using machine learning from this data. From an analysis of the literature, we noticed the prevalence of some algorithms, such as Multi-layer Perceptrons (MLP), Support Vector Machines (SVM), and Long Short-Term Memory (LSTM), which presented good results, mainly considering the True Skill Statistic (TSS) metric. In parallel, in 2017, a new deep learning-based neural network architecture called Transformers emerged. Researchers initially created it for natural language processing. However, Transformers were successfully employed in other domains, such as time series forecasting. Solar activity data is considered a time series due to its continuous capture over time. Consequently, we can employ Transformers to develop a solar flare forecast model. Considering a significant lack of work using Transformers for solar flare forecasting, we ran experiments to test the Transformers' viability and performance in solar flare forecast models. We created models using other algorithms (MLP, SVM, LSTM, Transformers) to investigate the Transformers' performance and compared them using accuracy, TSS, and Area Under the ROC Curve (AUC) metrics. We observed that the Transformers had superior performance compared to the other models. For instance, the Transformers' TSS metric average was 0.9, contrasting the other models' TSS=0.4. The difference was slightly smaller in AUC, where Transformers reached 0.9, and the others reached no more than 0.7. Therefore, we can use the Transformers to classify solar flare data and obtain superior results compared to other models. We also conducted experiments using different forms of data balancing, including unbalanced data, balanced with undersampling, oversampling, and SMOTE techniques. The MLP, SVM, and LSTM models showed significant improvements in balance, where the average TSS increased from 0.1 to 0.4. On the other hand, Transformers were not sensitive to data balancing, presenting the most stable TSS in all cases.","manuscriptTitle":"Transformers as a classifier for solar flare time series: a comparative study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-18 05:12:13","doi":"10.21203/rs.3.rs-4093277/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1581f882-f1d2-4aa5-9f8d-23f2400e98b4","owner":[],"postedDate":"March 18th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-04-09T16:50:11+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-18 05:12:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4093277","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4093277","identity":"rs-4093277","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}