The News with ChatGPT: An Audit and Survey Experiment on the Effects of GPT-Enabled News Search on User Attitudes

The News with ChatGPT: An Audit and Survey Experiment on the Effects of GPT-Enabled News Search on User Attitudes | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article The News with ChatGPT: An Audit and Survey Experiment on the Effects of GPT-Enabled News Search on User Attitudes Kokil Jaidka, Shaz Furniturewala This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6692200/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Policymakers and researchers are increasingly concerned with how algorithmic systems shape public access to political information. AI-powered chatbots, such as OpenAI’s GPT-4o, are being adopted as search tools, yet their impacts on information quality and user attitudes remain unclear. This study evaluates the information retrieval performance and cognitive consequences of GPT-4o News compared to Google News. In the first part, we audited 592,008 news articles retrieved by GPT-4o News and Google News in response to over 80,000 political queries across eleven countries. Our results show that GPT-4o News retrieved significantly less relevant and more controversial content than Google News, despite slightly lower partisanship scores. In the second part, we conducted a bilingual survey experiment (N = 921) in the U.S. and India, randomly assigning participants to one of four retrieval formats. We found that participants exposed to GPT-4o reported higher knowledge gains but also demonstrated lower misinformation discernment, particularly when interacting with a conversational chatbot. These effects were moderated by gender, with non-male participants gaining more knowledge but showing greater declines in discernment. Our findings highlight trade-offs between perceived learning and evaluative accuracy when relying on AI-driven information systems, raising important implications for the design of trustworthy political information retrieval tools. Humanities/Language and linguistics Social science/Science, technology and society OpenAI Google search ChatGPT audit Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6692200","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":458834503,"identity":"8d68bd8e-7829-4579-90f1-682f5603801f","order_by":0,"name":"Kokil Jaidka","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1klEQVRIiWNgGAWjYBACxgbGBskGBhsIj4cELWkkaAEBoJbDJGhhbj/ceHNGzfk8g/MHGB+8bWOwl3cg5LCexGbLDcduFxvcSGA2nNvGkLjxACEtDYltkg/YbiduuMHAJs3bxpBg2EBIS/9DoJZ/5xI3nD/A/huoxZ6wlhlAWza2HUjccCCBjRmohXE+AR1ALQ+bLWf2JSfOvJHYLDnnnETiBkJaDPvTH97s+WaX2Hf+8MEPb8ps7OUJOQzJ5YwgpgSDwQECWuQxRQjZMgpGwSgYBSMOAADSSkc4y0VomAAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-8127-1157","institution":"National University of Singapore","correspondingAuthor":true,"prefix":"","firstName":"Kokil","middleName":"","lastName":"Jaidka","suffix":""},{"id":458834504,"identity":"bf430d35-4b69-4e2a-be25-870246f25d74","order_by":1,"name":"Shaz Furniturewala","email":"","orcid":"","institution":"National University of Singapore","correspondingAuthor":false,"prefix":"","firstName":"Shaz","middleName":"","lastName":"Furniturewala","suffix":""}],"badges":[],"createdAt":"2025-05-18 14:10:22","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6692200/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6692200/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":84488994,"identity":"a9aeb708-41b2-4bd8-9998-bf86e6f01874","added_by":"auto","created_at":"2025-06-12 14:15:39","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3070266,"visible":true,"origin":"","legend":"Article File #1","description":"","filename":"NHBgptsearchfinal17476591661.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6692200/v1_covered_31ca9680-7e14-48d4-9bb4-7fa90a8f068e.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"The News with ChatGPT: An Audit and Survey Experiment on the Effects of GPT-Enabled News Search on User Attitudes","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"OpenAI, Google, search, ChatGPT, audit","lastPublishedDoi":"10.21203/rs.3.rs-6692200/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6692200/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Policymakers and researchers are increasingly concerned with how algorithmic systems shape public access to political information. AI-powered chatbots, such as OpenAI’s GPT-4o, are being adopted as search tools, yet their impacts on information quality and user attitudes remain unclear. This study evaluates the information retrieval performance and cognitive consequences of GPT-4o News compared to Google News. In the first part, we audited 592,008 news articles retrieved by GPT-4o News and Google News in response to over 80,000 political queries across eleven countries. Our results show that GPT-4o News retrieved significantly less relevant and more controversial content than Google News, despite slightly lower partisanship scores. In the second part, we conducted a bilingual survey experiment (N = 921) in the U.S. and India, randomly assigning participants to one of four retrieval formats. We found that participants exposed to GPT-4o reported higher knowledge gains but also demonstrated lower misinformation discernment, particularly when interacting with a conversational chatbot. These effects were moderated by gender, with non-male participants gaining more knowledge but showing greater declines in discernment. Our findings highlight trade-offs between perceived learning and evaluative accuracy when relying on AI-driven information systems, raising important implications for the design of trustworthy political information retrieval tools.","manuscriptTitle":"The News with ChatGPT: An Audit and Survey Experiment on the Effects of GPT-Enabled News Search on User Attitudes","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-20 11:39:15","doi":"10.21203/rs.3.rs-6692200/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d226ea2e-ca14-4034-9a90-4a597dc3342c","owner":[],"postedDate":"May 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":48743163,"name":"Humanities/Language and linguistics"},{"id":48743164,"name":"Social science/Science, technology and society"}],"tags":[],"updatedAt":"2025-06-12T18:05:42+00:00","versionOfRecord":[],"versionCreatedAt":"2025-05-20 11:39:15","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6692200","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6692200","identity":"rs-6692200","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}