A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety

A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety Shuai Liu, Weilin Pu, Chongling Xu, Zishuo Huang, Qian Li, Hang Wang, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5270567/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Recent advancements in MLLM, such as those exemplified by developments like GPT-4o, have positioned them as a significant focus within the research community. MLLMs leverage the general capabilities of Large Language Models (LLMs) to handle tasks across multiple modalities, including text, image, audio, and video. With their unique ability to understand and generate content, such as composing narratives from visual inputs, MLLMs are attracting substantial interest from both academia and industry. However, the great outburst of algorithms and techniques of MLLMs has led to the emergence of new types of architectures, applications and safety issues in MLLMs. We provide this more comprehensive survey aiming to document and analyze the latest advancements in MLLMs. First, we introduce the fundamental concepts of MLLMs, including the development history of multimodal algorithms, the architecture of MLLMs, and their evaluation and benchmarks. We then explore advanced techniques in MLLMs, such as Multimodal In-Context Learning, Multimodal Chain of Thought, and LLM-aided Visual Reasoning. Following this, we examine the safety aspects of MLLMs, focusing on security issues, potential attacks, and model safety assessments. Finally, we discuss the current challenges and identify potential areas for future research. Multimodal large language models language models large models modalities survey Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5270567","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":366739645,"identity":"d5f90e1a-30d1-4f79-a78b-465340f9cef7","order_by":0,"name":"Shuai Liu","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Shuai","middleName":"","lastName":"Liu","suffix":""},{"id":366739646,"identity":"cdf74d84-bb20-4aca-8af9-b972dbfb5a3e","order_by":1,"name":"Weilin Pu","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Weilin","middleName":"","lastName":"Pu","suffix":""},{"id":366739647,"identity":"15d99f3d-b687-4270-a5ff-9ef2b054cfcf","order_by":2,"name":"Chongling Xu","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Chongling","middleName":"","lastName":"Xu","suffix":""},{"id":366739648,"identity":"f8087ef2-e574-4189-82f9-b214091c4918","order_by":3,"name":"Zishuo Huang","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Zishuo","middleName":"","lastName":"Huang","suffix":""},{"id":366739649,"identity":"751c6e29-8a1c-4e3b-9022-0d4a240eee9d","order_by":4,"name":"Qian Li","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Qian","middleName":"","lastName":"Li","suffix":""},{"id":366739650,"identity":"b022c7a1-09f3-4e97-81a0-0f57854fa7cb","order_by":5,"name":"Hang Wang","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Hang","middleName":"","lastName":"Wang","suffix":""},{"id":366739651,"identity":"eda5aeb0-e02c-4b44-bc29-de614ece9f46","order_by":6,"name":"Chenhao Lin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA60lEQVRIiWNgGAWjYBACPmYGBiBiSABSBxgkwGIJ+LWwIbSwJTBIJBCjhQGuhccAqpqQFnbeg58LKu7k8bf3fJOw/HGYgZ89x4Dh5w58DuNLlp5x5lmxxJmzmw0kEg4zSPa8MWDsPYNPC48ZM2/b4cSGG7kbH4C0GNzIMWBmbCOk5d/hxPk3ch4cAGmxJ05Lw+HEDTdyGCG2SBDWYizNc+xw4sYzx4wNJNLSeSTOPCs42ItHCz//GcPPPDWHE+cdb34mLWFjLcffnrzxwU88WlAAMzD2eUCMA0RqYGBg/EC00lEwCkbBKBhJAABMQkpqsVhNYwAAAABJRU5ErkJggg==","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":true,"prefix":"","firstName":"Chenhao","middleName":"","lastName":"Lin","suffix":""},{"id":366739652,"identity":"b8343d5c-e287-47f1-a682-544ad8ae2e00","order_by":7,"name":"Chao Shen","email":"","orcid":"","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Chao","middleName":"","lastName":"Shen","suffix":""}],"badges":[],"createdAt":"2024-10-15 17:23:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5270567/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5270567/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":73593991,"identity":"6727f946-0963-4c13-89b5-759fea3285ab","added_by":"auto","created_at":"2025-01-12 12:01:34","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":6462132,"visible":true,"origin":"","legend":"","description":"","filename":"mllmsurvey.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5270567/v1_covered_9b662874-3022-4829-8d54-451a03a37da3.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multimodal large language models, language models, large models, modalities, survey","lastPublishedDoi":"10.21203/rs.3.rs-5270567/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5270567/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recent advancements in MLLM, such as those exemplified by developments like GPT-4o, have positioned them as a significant focus within the research community. MLLMs leverage the general capabilities of Large Language Models (LLMs) to handle tasks across multiple modalities, including text, image, audio, and video. With their unique ability to understand and generate content, such as composing narratives from visual inputs, MLLMs are attracting substantial interest from both academia and industry. However, the great outburst of algorithms and techniques of MLLMs has led to the emergence of new types of architectures, applications and safety issues in MLLMs. We provide this more comprehensive survey aiming to document and analyze the latest advancements in MLLMs. First, we introduce the fundamental concepts of MLLMs, including the development history of multimodal algorithms, the architecture of MLLMs, and their evaluation and benchmarks. We then explore advanced techniques in MLLMs, such as Multimodal In-Context Learning, Multimodal Chain of Thought, and LLM-aided Visual Reasoning. Following this, we examine the safety aspects of MLLMs, focusing on security issues, potential attacks, and model safety assessments. Finally, we discuss the current challenges and identify potential areas for future research.","manuscriptTitle":"A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-18 02:59:19","doi":"10.21203/rs.3.rs-5270567/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5919a91e-fb83-4fc7-8b89-31615f33991a","owner":[],"postedDate":"October 18th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-01-12T11:53:25+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-18 02:59:19","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5270567","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5270567","identity":"rs-5270567","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}