Efficient GPT-4V Level Multimodal Large Language Model for Deployment on Edge Devices

Efficient GPT-4V Level Multimodal Large Language Model for Deployment on Edge Devices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Efficient GPT-4V Level Multimodal Large Language Model for Deployment on Edge Devices Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, and 18 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5830327/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 01 Jul, 2025 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Abstract The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone. However, significant challenges remain preventing MLLMs from being practical in real-world applications. The most notable challenge comes from the huge cost of running an MLLM with a massive number of parameters and extensive computation. As a result, most MLLMs need to be deployed on high-performing cloud servers, which greatly limits their application scopes such as mobile, offline, energy-sensitive, and privacy-protective scenarios. In this work, we present MiniCPM-V, a series of efficient MLLMs deployable on edge devices. By integrating the latest MLLM techniques in architecture, pre-training and alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1) Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong OCR capability and 1.8M pixel high-resolution image perception at any aspect ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual support for 30+ languages, and (5) efficient deployment on mobile phones. More importantly, MiniCPM-V can be viewed as a representative example of a promising trend (Fig. 1): The model sizes for achieving usable (e.g., GPT-4V) level performance are rapidly decreasing, along with the fast growth of edge computation capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices are becoming increasingly possible, unlocking a wider spectrum of real-world AI applications in the near future. Physical sciences/Mathematics and computing/Computer science Biological sciences/Computational biology and bioinformatics/Machine learning Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Published Journal Publication published 01 Jul, 2025 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5830327","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":403044965,"identity":"408070e7-7ffb-4da5-a792-22bd1bde9630","order_by":0,"name":"Yuan Yao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA4UlEQVRIiWNgGAWjYDADAwbmA0BKQoZoHRIGDGwJIJqHFC08BiAGYS0Gxw8fk+apuVNnLpHz+dWNGgseBvbDRzfg1XImLU2a59gzCcsZudusc44BHcaTlnYDr5YDOWbSPGyHJQxu5G4zzmEDapHgMcOv5fwboJZ/IC05z4xz/hGj5QbQFt42sBbmx7ltRGiRvPEs2XJu32HJDWeemTHn9knwsBHyC9/55IM33nw7zG9wPPnx55xvdXL87IeP4dWicICBRQLKZgMz2PApBwH5BgbmD1A2nDEKRsEoGAWjAAUAAJJuSSMnNtceAAAAAElFTkSuQmCC","orcid":"","institution":"National University of Singapore","correspondingAuthor":true,"prefix":"","firstName":"Yuan","middleName":"","lastName":"Yao","suffix":""},{"id":403044966,"identity":"69a87aaa-92a3-4638-813b-4bbc9eba1876","order_by":1,"name":"Tianyu Yu","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Tianyu","middleName":"","lastName":"Yu","suffix":""},{"id":403044967,"identity":"047abcd8-86f2-4085-bd55-cd7521b67649","order_by":2,"name":"Ao Zhang","email":"","orcid":"","institution":"National University of Singapore","correspondingAuthor":false,"prefix":"","firstName":"Ao","middleName":"","lastName":"Zhang","suffix":""},{"id":403044968,"identity":"4ef319cf-384d-49aa-8c22-44db35bddcee","order_by":3,"name":"Chongyi Wang","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Chongyi","middleName":"","lastName":"Wang","suffix":""},{"id":403044969,"identity":"5e966630-55f1-444a-8085-406b932477ab","order_by":4,"name":"Junbo Cui","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Junbo","middleName":"","lastName":"Cui","suffix":""},{"id":403044970,"identity":"4ee2bd1e-2bce-4172-ae49-a6fb52a200a9","order_by":5,"name":"Hongji Zhu","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Hongji","middleName":"","lastName":"Zhu","suffix":""},{"id":403044971,"identity":"42383914-7125-4d9b-93ac-e3d8d091a975","order_by":6,"name":"Tianchi Cai","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Tianchi","middleName":"","lastName":"Cai","suffix":""},{"id":403044972,"identity":"09eaf3b9-e8cc-4e85-8e29-68452b3e008b","order_by":7,"name":"Chi Chen","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Chi","middleName":"","lastName":"Chen","suffix":""},{"id":403044973,"identity":"4dbd1435-c371-4414-9ef8-a0893c49a4ff","order_by":8,"name":"Haoyu Li","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Haoyu","middleName":"","lastName":"Li","suffix":""},{"id":403044974,"identity":"f3acd605-8904-49ef-8628-a76e0072280e","order_by":9,"name":"Weilin Zhao","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Weilin","middleName":"","lastName":"Zhao","suffix":""},{"id":403044975,"identity":"9a763e4f-ba43-450c-a468-2f0306636318","order_by":10,"name":"Zhihui He","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Zhihui","middleName":"","lastName":"He","suffix":""},{"id":403044976,"identity":"984285dc-2a4e-46ff-988f-7d1545843d5c","order_by":11,"name":"Qianyu Chen","email":"","orcid":"","institution":"The Chinese University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Qianyu","middleName":"","lastName":"Chen","suffix":""},{"id":403044977,"identity":"80bb292b-ec4b-4d27-a75b-24edc6e7c1c3","order_by":12,"name":"Ronghua Zhou","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Ronghua","middleName":"","lastName":"Zhou","suffix":""},{"id":403044978,"identity":"c9f6651a-909f-4073-9f89-cb0a0edce4a5","order_by":13,"name":"Zhensheng Zou","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Zhensheng","middleName":"","lastName":"Zou","suffix":""},{"id":403044979,"identity":"cd99fdd4-cd05-49a7-bdde-8ff3545b5365","order_by":14,"name":"Haoye Zhang","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Haoye","middleName":"","lastName":"Zhang","suffix":""},{"id":403044980,"identity":"8803090b-34a2-483d-b7e3-c49e9f37c1ee","order_by":15,"name":"Shengding Hu","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Shengding","middleName":"","lastName":"Hu","suffix":""},{"id":403044981,"identity":"d1c5c309-96b3-442d-a30b-5c9e7dd3c827","order_by":16,"name":"Zhi Zheng","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Zhi","middleName":"","lastName":"Zheng","suffix":""},{"id":403044982,"identity":"e36472af-3225-4d25-bb2d-495ceac88d1c","order_by":17,"name":"Jie Zhou","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Jie","middleName":"","lastName":"Zhou","suffix":""},{"id":403044983,"identity":"6dcb487e-e423-4eae-afe1-effc9c72338d","order_by":18,"name":"Jie Cai","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Jie","middleName":"","lastName":"Cai","suffix":""},{"id":403044984,"identity":"d46fb4b6-dadc-4c06-a1aa-2b9ba08835c7","order_by":19,"name":"Xu Han","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Xu","middleName":"","lastName":"Han","suffix":""},{"id":403044985,"identity":"fc595a2b-0c46-4c1f-8f6c-4ee73f14a269","order_by":20,"name":"Guoyang Zeng","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Guoyang","middleName":"","lastName":"Zeng","suffix":""},{"id":403044986,"identity":"63a7a6be-df49-43d8-b184-18a5b5b177e9","order_by":21,"name":"Dahai Li","email":"","orcid":"","institution":"ModelBest Inc.","correspondingAuthor":false,"prefix":"","firstName":"Dahai","middleName":"","lastName":"Li","suffix":""},{"id":403044987,"identity":"f464f851-977a-4a31-a584-3bc5d30dd19d","order_by":22,"name":"Zhiyuan Liu","email":"","orcid":"https://orcid.org/0000-0002-7709-2543","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Zhiyuan","middleName":"","lastName":"Liu","suffix":""},{"id":403044988,"identity":"50edbc39-f54d-4486-b971-0d3bf0f5bfab","order_by":23,"name":"Maosong Sun","email":"","orcid":"https://orcid.org/0000-0002-6011-6115","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Maosong","middleName":"","lastName":"Sun","suffix":""}],"badges":[],"createdAt":"2025-01-15 01:35:40","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5830327/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5830327/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41467-025-61040-5","type":"published","date":"2025-07-01T04:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":85832795,"identity":"e5c010ed-3794-4d28-9851-75ae01b710a4","added_by":"auto","created_at":"2025-07-02 08:05:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3704372,"visible":true,"origin":"","legend":"Article File","description":"","filename":"MiniCPMV1209.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5830327/v1_covered_141d75cb-d43f-4b8e-8f85-4384adb9a769.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Efficient GPT-4V Level Multimodal Large Language Model for Deployment on Edge Devices","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5830327/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5830327/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone. However, significant challenges remain preventing MLLMs from being practical in real-world applications. The most notable challenge comes from the huge cost of running an MLLM with a massive number of parameters and extensive computation. As a result, most MLLMs need to be deployed on high-performing cloud servers, which greatly limits their application scopes such as mobile, offline, energy-sensitive, and privacy-protective scenarios. In this work, we present MiniCPM-V, a series of efficient MLLMs deployable on edge devices. By integrating the latest MLLM techniques in architecture, pre-training and alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1) Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong OCR capability and 1.8M pixel high-resolution image perception at any aspect ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual support for 30+ languages, and (5) efficient deployment on mobile phones. More importantly, MiniCPM-V can be viewed as a representative example of a promising trend (Fig. 1): The model sizes for achieving usable (e.g., GPT-4V) level performance are rapidly decreasing, along with the fast growth of edge computation capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices are becoming increasingly possible, unlocking a wider spectrum of real-world AI applications in the near future.","manuscriptTitle":"Efficient GPT-4V Level Multimodal Large Language Model for Deployment on Edge Devices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-17 03:36:45","doi":"10.21203/rs.3.rs-5830327/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"3e46cb3d-ff8d-4e3b-ac79-443a12a19e70","owner":[],"postedDate":"January 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":42969702,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":42969703,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2025-07-02T08:04:58+00:00","versionOfRecord":{"articleIdentity":"rs-5830327","link":"https://doi.org/10.1038/s41467-025-61040-5","journal":{"identity":"nature-communications","isVorOnly":false,"title":"Nature Communications"},"publishedOn":"2025-07-01 04:00:00","publishedOnDateReadable":"July 1st, 2025"},"versionCreatedAt":"2025-01-17 03:36:45","video":"","vorDoi":"10.1038/s41467-025-61040-5","vorDoiUrl":"https://doi.org/10.1038/s41467-025-61040-5","workflowStages":[]},"version":"v1","identity":"rs-5830327","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5830327","identity":"rs-5830327","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}