Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch

Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch Dongjing Shan, Jin Li, Lisha Zhong, Biao Qu, Qi Han This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5364284/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Vision Transformers (ViTs) have achieved impressive results in large-scale image classification. However, when training from scratch on small datasets, there is still a significant performance gap between ViTs and Convolutional Neural Networks (CNNs), which is attributed to the lack of inductive bias. To address this issue, we propose a Graph-based Vision Transformer (GvT) that utilizes graph convolutional projection and graphpooling. In each block, queries and keys are calculated through graph convolutional projection based on the spatial adjacency matrix, while dot-product attention is used in another graph convolution to generate values. When using more attention heads, the queries and keys become lower-dimensional, making their dot product an uninformative matching function. To overcome this low-rank bottleneck in attention heads, we employ talkingheads technology based on bilinear pooled features and sparse selection of attention tensors. This allows interaction among filtered attention scores and enables each attention mechanism to depend on all queries and keys. Additionally, we apply graphpooling between two intermediate blocks to reduce the number of tokens and aggregate semantic information more effectively. Our experimental results show that GvT produces comparable or superior outcomes to deep convolutional networks and surpasses vision transformers without pre-training on large datasets. The code for our proposed model is publicly available on the website: GitHub/GvT. Vision Transformer graph convolution selfattention graph-pooling image classification Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5364284","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":376010926,"identity":"d6ba243c-df22-43d6-902a-8a5b2b7c456a","order_by":0,"name":"Dongjing Shan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+UlEQVRIiWNgGAWjYDACCSDmAWLGZiDxAUmQOC2MM0jSAgLMPEiCOIH87OZnD95U3LFrbuc9/Nrmz+FogwPMB2/zMNjl4dJicOeYueGcM8+SG5v50qxz2w7nbjjAlmzNw5BcjFOLRIKZNG/b4WTGZh4z49yG20AtPGbSPAwHEhtwOWxG+jeEFos/IC383/BqYbiRA7bFDqjF+DEDG9gWNrxaDG7klEnOOXM4AWQLY2/b/9yZh9mMLecYJONz2DaJNxWH7Q37zxh/+PEnLbfvePPDG28q7HA7DAoSNzYwsEGigxlsOwH1QGAvD1T7gbC6UTAKRsEoGIkAAI6fWgxJrTtxAAAAAElFTkSuQmCC","orcid":"","institution":"Southwest Medical University","correspondingAuthor":true,"prefix":"","firstName":"Dongjing","middleName":"","lastName":"Shan","suffix":""},{"id":376010927,"identity":"bef39c25-1db7-4a01-950b-72a8436b594c","order_by":1,"name":"Jin Li","email":"","orcid":"","institution":"Southwest Medical University","correspondingAuthor":false,"prefix":"","firstName":"Jin","middleName":"","lastName":"Li","suffix":""},{"id":376010928,"identity":"28951521-23b5-4cec-9cad-27b12906d0dd","order_by":2,"name":"Lisha Zhong","email":"","orcid":"","institution":"Southwest Medical University","correspondingAuthor":false,"prefix":"","firstName":"Lisha","middleName":"","lastName":"Zhong","suffix":""},{"id":376010929,"identity":"efcf11a5-fcb2-431d-8baa-6ef307f8b8e7","order_by":3,"name":"Biao Qu","email":"","orcid":"","institution":"Southwest Medical University","correspondingAuthor":false,"prefix":"","firstName":"Biao","middleName":"","lastName":"Qu","suffix":""},{"id":376010930,"identity":"c32cb27e-eff0-4216-9ce5-1429b4bd444b","order_by":4,"name":"Qi Han","email":"","orcid":"","institution":"Nuclear Medicine Department, General Hospital of Tibet Military Area Command","correspondingAuthor":false,"prefix":"","firstName":"Qi","middleName":"","lastName":"Han","suffix":""}],"badges":[],"createdAt":"2024-10-31 02:38:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5364284/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5364284/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":69042633,"identity":"96c9a93c-0ec6-4501-b3c3-7b667f1f7b2d","added_by":"auto","created_at":"2024-11-15 02:08:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":565516,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptLatex2024.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5364284/v1_covered_1b132359-61c5-4e43-ba08-489d80d15caf.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Vision Transformer, graph convolution, selfattention, graph-pooling, image classification","lastPublishedDoi":"10.21203/rs.3.rs-5364284/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5364284/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eVision Transformers (ViTs) have achieved impressive results in large-scale image classification. However, when training from scratch on small datasets, there is still a significant performance gap between ViTs and Convolutional Neural Networks (CNNs), which is attributed to the lack of inductive bias. To address this issue, we propose a Graph-based Vision Transformer (GvT) that utilizes graph convolutional projection and graphpooling. In each block, queries and keys are calculated through graph convolutional projection based on the spatial adjacency matrix, while dot-product attention is used in another graph convolution to generate values. When using more attention heads, the queries and keys become lower-dimensional, making their dot product an uninformative matching function. To overcome this low-rank bottleneck in attention heads, we employ talkingheads technology based on bilinear pooled features and sparse selection of attention tensors. This allows interaction among filtered attention scores and enables each attention mechanism to depend on all queries and keys. Additionally, we apply graphpooling between two intermediate blocks to reduce the number of tokens and aggregate semantic information more effectively. Our experimental results show that GvT produces comparable or superior outcomes to deep convolutional networks and surpasses vision transformers without pre-training on large datasets. The code for our proposed model is publicly available on the website: GitHub/GvT.\u003c/p\u003e","manuscriptTitle":"Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-14 07:47:14","doi":"10.21203/rs.3.rs-5364284/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"01dda1cc-b675-4b03-961f-e8521b669f16","owner":[],"postedDate":"November 14th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-11-15T02:08:31+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-14 07:47:14","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5364284","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5364284","identity":"rs-5364284","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}