Do We Practice What We Preach? An Exploratory Introspective Analysis of ICSE Research Artifacts

Do We Practice What We Preach? An Exploratory Introspective Analysis of ICSE Research Artifacts | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Do We Practice What We Preach? An Exploratory Introspective Analysis of ICSE Research Artifacts Samuel Thand, Tobias Hansson, Felix Dobslaw, Sergio Rico This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8741708/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 11 You are reading this latest preprint version Abstract Background/Context: Research artifacts in Software Engineering accompany empirical studies and are expected to embody the engineering practices our community advocates.These artifacts serve dual purposes: supporting reproducibility of research findings and demonstrating practical application of software engineering principles.Yet the alignment between advocated practices and actual implementation in research artifacts remains underexamined. Goals: This study provides a quantitative examination of engineering practices in Python research artifacts from the International Conference on Software Engineering (ICSE) (2019--2023), focusing on process metrics, code quality, and testing practices. We contribute empirical evidence to discussions on reproducibility, artifact evaluation, and the role of engineering quality in research artifacts. Methods: We developed the CIRAS (Code Insight and Repository Analysis System) framework and implemented it for Python (PyCIRAS) to conduct repository mining on 90 badged research artifacts. Analysis covered Git process data via the Delta Maintainability Model, static code quality via Pylint, and unit testing metrics, including test-to-production ratios. Results: Findings revealed substantial variability. While artifacts showed reasonable structural maintainability, many exhibited low linting scores (mean 1.72/10), frequent errors (54.66% import errors), sparse documentation (68% undocumented modules), and minimal testing (mean TPS ratio 0.01). No statistically significant correlation emerged between these metrics and repository popularity (GitHub stars). Conclusions: Within our bounded sample of 90 Python artifacts from ICSE 2019--2023, we observe a gap between advocated practices and implementation in research artifacts. The contribution is primarily introspective, demonstrating how repository mining can quantify this gap and provide evidence-based recommendations. We discuss implications for artifact evaluation, training, and the potential role of generative AI in addressing quality gaps. software engineering research artifacts code quality repository mining introspective analysis Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 04 May, 2026 Reviews received at journal 27 Apr, 2026 Reviews received at journal 18 Mar, 2026 Reviews received at journal 16 Mar, 2026 Reviewers agreed at journal 07 Mar, 2026 Reviewers agreed at journal 27 Feb, 2026 Reviewers agreed at journal 23 Feb, 2026 Reviewers invited by journal 17 Feb, 2026 Editor assigned by journal 02 Feb, 2026 Submission checks completed at journal 02 Feb, 2026 First submitted to journal 30 Jan, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8741708","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":592776504,"identity":"b3d3e459-5fea-4157-94e5-804ab4272b4f","order_by":0,"name":"Samuel Thand","email":"","orcid":"","institution":"Mid Sweden University","correspondingAuthor":false,"prefix":"","firstName":"Samuel","middleName":"","lastName":"Thand","suffix":""},{"id":592776511,"identity":"6df68f56-5030-46d6-bd74-af39382eb066","order_by":1,"name":"Tobias Hansson","email":"","orcid":"","institution":"Mid Sweden University","correspondingAuthor":false,"prefix":"","firstName":"Tobias","middleName":"","lastName":"Hansson","suffix":""},{"id":592776520,"identity":"068e93c8-3d0a-4124-870f-503b1c9e8c57","order_by":2,"name":"Felix Dobslaw","email":"","orcid":"","institution":"Mid Sweden University","correspondingAuthor":false,"prefix":"","firstName":"Felix","middleName":"","lastName":"Dobslaw","suffix":""},{"id":592776522,"identity":"43b198c8-abf9-4a4b-9ba6-82cbec8929fa","order_by":3,"name":"Sergio Rico","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzklEQVRIiWNgGAWjYLCCBCBmY2BgfMBgQKIWZgPitUABmwRRyuTbex+/eFDBkM/HfvhY1Y2Cw3kM/IcP4NVicOa4mUXCGQbLNp60tNs5BoeLGSTSEvBrkUhjM0hsYzBgk+AxA2lJbJDgwe8j+fnPEFqKwVr4z3/A75kbbMwPYFqYwVoYcvDrMDiTxsaQcEbCgI0nLVk6xyA9sU0ijYDD2o8xf/xRYWMg33744OecP9aJ/fyHH+C3BhIdSDHCRkg9EDAT8O0oGAWjYBSMeAAAoUo8gF/czuMAAAAASUVORK5CYII=","orcid":"","institution":"Mid Sweden University","correspondingAuthor":true,"prefix":"","firstName":"Sergio","middleName":"","lastName":"Rico","suffix":""}],"badges":[],"createdAt":"2026-01-30 13:38:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8741708/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8741708/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104397274,"identity":"d2c6e45d-0827-40db-8fcc-973cd65ca70e","added_by":"auto","created_at":"2026-03-11 11:45:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1802854,"visible":true,"origin":"","legend":"","description":"","filename":"SQJSubmissionUpdated.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8741708/v1_covered_12e95910-4e08-4804-be53-6303ba60288f.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Do We Practice What We Preach? An Exploratory Introspective Analysis of ICSE Research Artifacts","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"software-quality-journal","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sqjo","sideBox":"Learn more about [Software Quality Journal](http://link.springer.com/journal/11219)","snPcode":"11219","submissionUrl":"https://submission.nature.com/new-submission/11219/3","title":"Software Quality Journal","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"software engineering, research artifacts, code quality, repository mining, introspective analysis","lastPublishedDoi":"10.21203/rs.3.rs-8741708/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8741708/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground/Context:\u003c/h2\u003e\u003cp\u003eResearch artifacts in Software Engineering accompany empirical studies and are expected to embody the engineering practices our community advocates.These artifacts serve dual purposes: supporting reproducibility of research findings and demonstrating practical application of software engineering principles.Yet the alignment between advocated practices and actual implementation in research artifacts remains underexamined.\u003c/p\u003e\u003ch2\u003eGoals:\u003c/h2\u003e\u003cp\u003eThis study provides a quantitative examination of engineering practices in Python research artifacts from the International Conference on Software Engineering (ICSE) (2019--2023), focusing on process metrics, code quality, and testing practices. We contribute empirical evidence to discussions on reproducibility, artifact evaluation, and the role of engineering quality in research artifacts.\u003c/p\u003e\u003ch2\u003eMethods:\u003c/h2\u003e\u003cp\u003eWe developed the CIRAS (Code Insight and Repository Analysis System) framework and implemented it for Python (PyCIRAS) to conduct repository mining on 90 badged research artifacts. Analysis covered Git process data via the Delta Maintainability Model, static code quality via Pylint, and unit testing metrics, including test-to-production ratios.\u003c/p\u003e\u003ch2\u003eResults:\u003c/h2\u003e\u003cp\u003eFindings revealed substantial variability. While artifacts showed reasonable structural maintainability, many exhibited low linting scores (mean 1.72/10), frequent errors (54.66% import errors), sparse documentation (68% undocumented modules), and minimal testing (mean TPS ratio 0.01). No statistically significant correlation emerged between these metrics and repository popularity (GitHub stars).\u003c/p\u003e\u003ch2\u003eConclusions:\u003c/h2\u003e\u003cp\u003eWithin our bounded sample of 90 Python artifacts from ICSE 2019--2023, we observe a gap between advocated practices and implementation in research artifacts. The contribution is primarily introspective, demonstrating how repository mining can quantify this gap and provide evidence-based recommendations. We discuss implications for artifact evaluation, training, and the potential role of generative AI in addressing quality gaps.\u003c/p\u003e","manuscriptTitle":"Do We Practice What We Preach? An Exploratory Introspective Analysis of ICSE Research Artifacts","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-19 06:52:03","doi":"10.21203/rs.3.rs-8741708/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-04T07:39:07+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-27T07:01:56+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-18T13:53:55+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-03-16T09:19:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"304020112865892247481344244672488315121","date":"2026-03-07T09:10:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"82422065036384476230176774321012116215","date":"2026-02-27T12:23:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"288919112003234585797236196973933016969","date":"2026-02-23T11:18:50+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-02-17T08:56:21+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-03T01:15:04+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-03T01:14:04+00:00","index":"","fulltext":""},{"type":"submitted","content":"Software Quality Journal","date":"2026-01-30T13:28:14+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"software-quality-journal","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sqjo","sideBox":"Learn more about [Software Quality Journal](http://link.springer.com/journal/11219)","snPcode":"11219","submissionUrl":"https://submission.nature.com/new-submission/11219/3","title":"Software Quality Journal","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"8e3a168e-87f0-4bc6-b3f3-75b8b7b83596","owner":[],"postedDate":"February 19th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-04T07:39:07+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-04T07:54:50+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-19 06:52:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8741708","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8741708","identity":"rs-8741708","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}