Mapping the Mind of an Instruction-based Image Editing using SMILE

Mapping the Mind of an Instruction-based Image Editing using SMILE | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Mapping the Mind of an Instruction-based Image Editing using SMILE Zeinab Dehghani, Koorosh Aslansefat, Adil Khan, Adin Rivera, Franky George, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5943708/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Despite recent advancements in modifying high-quality images from text, instruction-based image editing models are known as black boxes. Their lack of transparency makes it difficult for users to fully trust these systems. To tackle this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations). This novelty approach is model-agnostic and provides transparent, localized explanations and visual heatmaps, helping users understand how specific textual inputs influence image generation. Our extensive testing across different metrics—stability, accuracy, fidelity, and consistency—shows that SMILE significantly improves interpretability and reliability when applied to popular models such as Instruct-pix2pix, Img2Img-Turbo, and Diffusers-Inpaint. These results highlight the importance of interpretability in making AI more transparent and trustworthy, especially in critical areas like autonomous driving and healthcare. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Statistics Explainable AI image editing interpretability local explanations model-agnostic interpretability SMILE framework statistical interpretability transparency trustworthiness visual heatmap. Full Text Additional Declarations There is NO Competing Interest. Supplementary Files Extendeddata.pdf Extended Data Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5943708","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":418425201,"identity":"b747cbc1-7b74-4d56-a70d-9c477536fa40","order_by":0,"name":"Zeinab Dehghani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHElEQVRIie3RMUvDQBTA8Xc8uC6vuqZEmq9wJWAn61dJyNCpIHRxyHClEBfp3H6L6yTdTg7iEunq4FARnAOCFC3FJKMkraPg/cc7fty7OwCb7W8mEEADILBNcB13BeBEVxt0nKDIs9QXwOQvCQDvLBIM1THiSVy+Xn0+e/0WgEucD++csCDxAESmawmTfOwvZm+91RTAJzobreYlSSMQj7KWINC52741TJnTPCKHj9RTSbgGsa4fjAP1vwpyqQyAIYFDUZF9M6HiFKStCQvCJvMAg4qwRDcO5iAfu21pooIg5DrtqexF6nAWUafh+t7NdPlOO3Oh1rq1DfexJx6i+03+MeieZEH9ZFg+W/JzNTj0kVW7w9s2m832z/sGGW1kJpJgw90AAAAASUVORK5CYII=","orcid":"","institution":"University of Hull","correspondingAuthor":true,"prefix":"","firstName":"Zeinab","middleName":"","lastName":"Dehghani","suffix":""},{"id":418425202,"identity":"9062adbf-9d96-4a06-8482-6410d027f2d0","order_by":1,"name":"Koorosh Aslansefat","email":"","orcid":"","institution":"University of Hull","correspondingAuthor":false,"prefix":"","firstName":"Koorosh","middleName":"","lastName":"Aslansefat","suffix":""},{"id":418425203,"identity":"de0a570f-2a1a-4b8a-9c59-812d235b0ee6","order_by":2,"name":"Adil Khan","email":"","orcid":"","institution":"University of Hull","correspondingAuthor":false,"prefix":"","firstName":"Adil","middleName":"","lastName":"Khan","suffix":""},{"id":418425204,"identity":"22e9cc7b-1729-4fb3-a040-a16e6534e5c2","order_by":3,"name":"Adin Rivera","email":"","orcid":"","institution":"University of Oslo","correspondingAuthor":false,"prefix":"","firstName":"Adin","middleName":"","lastName":"Rivera","suffix":""},{"id":418425205,"identity":"d8cce3cc-8fb0-48b2-b3f3-a3ace005f9af","order_by":4,"name":"Franky George","email":"","orcid":"","institution":"University of Hull","correspondingAuthor":false,"prefix":"","firstName":"Franky","middleName":"","lastName":"George","suffix":""},{"id":418425206,"identity":"fe5244ae-72a1-4c0d-9e06-7eec5cd8683e","order_by":5,"name":"Muhammad Khalid","email":"","orcid":"","institution":"University of Hull","correspondingAuthor":false,"prefix":"","firstName":"Muhammad","middleName":"","lastName":"Khalid","suffix":""}],"badges":[],"createdAt":"2025-02-01 23:15:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5943708/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5943708/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":79176293,"identity":"31e4b88c-3ce2-4f18-8074-dc70a8622315","added_by":"auto","created_at":"2025-03-25 09:59:08","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2881246,"visible":true,"origin":"","legend":"Article File","description":"","filename":"Main.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5943708/v1_covered_441e3ab6-ed20-4961-a6fa-6e0c56a3a6c6.pdf"},{"id":79171420,"identity":"e92c2e9f-5935-4aa0-9620-1f01319da524","added_by":"auto","created_at":"2025-03-25 09:34:58","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":5710978,"visible":true,"origin":"","legend":"Extended Data","description":"","filename":"Extendeddata.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5943708/v1/9f87fa6aaa988c08d238279e.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Mapping the Mind of an Instruction-based Image Editing using SMILE","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Explainable AI, image editing, interpretability, local explanations, model-agnostic interpretability, SMILE framework, statistical interpretability, transparency, trustworthiness, visual heatmap.","lastPublishedDoi":"10.21203/rs.3.rs-5943708/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5943708/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Despite recent advancements in modifying high-quality images from text, instruction-based image editing models are known as black boxes. Their lack of transparency makes it difficult for users to fully trust these systems. To tackle this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations). This novelty approach is model-agnostic and provides transparent, localized explanations and visual heatmaps, helping users understand how specific textual inputs influence image generation. Our extensive testing across different metrics—stability, accuracy, fidelity, and consistency—shows that SMILE significantly improves interpretability and reliability when applied to popular models such as Instruct-pix2pix, Img2Img-Turbo, and Diffusers-Inpaint. These results highlight the importance of interpretability in making AI more transparent and trustworthy, especially in critical areas like autonomous driving and healthcare.","manuscriptTitle":"Mapping the Mind of an Instruction-based Image Editing using SMILE","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-25 09:34:53","doi":"10.21203/rs.3.rs-5943708/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1caf7cf8-92d4-46bc-ba3d-4bd62837f70b","owner":[],"postedDate":"March 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":44612181,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":44612182,"name":"Physical sciences/Mathematics and computing/Statistics"}],"tags":[],"updatedAt":"2025-03-25T09:34:55+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-25 09:34:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5943708","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5943708","identity":"rs-5943708","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}