Automated assessment of nocturnal affect using infrared recordings of facial expressions

Automated assessment of nocturnal affect using infrared recordings of facial expressions | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automated assessment of nocturnal affect using infrared recordings of facial expressions Sharon Ernst, Daniel K. Krauss, Moritz Moss, Martin Vossiek, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9418466/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The affective sciences have predominantly investigated affective states at wakefulness. This leaves a significant research gap since affective processes during sleep remain largely unexplored. This research gap is particularly important given that evidence suggests that sleep is one critical timespan for affective processing. To date however, methodological limitations challenge the assessment of affect during sleep. To overcome these challenges, we propose a novel, contactless approach for the automated assessment of nocturnal affect using infrared recordings of facial expressions (ASANA-IF). ASANA-IF consists of three components: (1) recordings of facial expressions during sleep with infrared cameras, (2) extracting affective states from these expressions using machine learning tools for objective emotion assessment (OpenFace, a validated tool based on deep neural networks), (3) aggregating these estimates within domains of emotion (e.g., anger, fear) with an algorithm developed for this purpose. To explore the feasibility and validity of ASANA-IF, we used this approach to assess nocturnal affective states in 4 participants over 21 nights from 3 perspectives. Here, we empirically tested the hypothesis that affective states during sleep (anxious, angry, sad and happy affect) would predict self-reports of respective affective states on the following day. Findings provide evidence for the feasibility of ASANA-IF. Moreover, they confirm the hypothesis that nocturnal affective states predict levels of respective affective states on the following day. Future research should further improve this new approach and use it to address research questions associated with assessing nocturnal affect in a feasible and valid way. Affect Emotion Machine Learning Nocturnal Assessment Sleep Full Text Additional Declarations No competing interests reported. Table 1 to 8 are available in the Supplementary Files section. Supplementary Files Manuscript2026SleepAffectTables.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9418466","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":629531852,"identity":"61cbdf1d-9ff9-4416-bd9a-902268f0a037","order_by":0,"name":"Sharon Ernst","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1ElEQVRIiWNgGAWjYDACCWTOBxsQmUC8FsbGGWmkamnmIUaL/OzmZx8+7mCQM5/d/vyxTYJdYgN78gG8WgzuHDOeOfMMg7HMnTOGzTkJyYkNPM/wW2MgkWDMzNvGkDhDIoexOfcHc2KDRI4BfofNSP8M0lI/QyL9YbNFQj1QS/4H/J65kQO2JUFCIsGwmSHhMMgW/DoMbuQUM85skzAEOsxwZk/CceM2nmcEHbaZ4WObjbyERPqDDz8SqmX72ZMf4LcGApBih40Y9aNgFIyCUTAK8AMADuZCal4X2zEAAAAASUVORK5CYII=","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":true,"prefix":"","firstName":"Sharon","middleName":"","lastName":"Ernst","suffix":""},{"id":629531853,"identity":"9028e089-bba3-4b26-ab0e-aa278ef022c1","order_by":1,"name":"Daniel K. Krauss","email":"","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"K.","lastName":"Krauss","suffix":""},{"id":629531856,"identity":"fd08de4d-e3aa-477f-8c53-5b3ed8bcfe09","order_by":2,"name":"Moritz Moss","email":"","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":false,"prefix":"","firstName":"Moritz","middleName":"","lastName":"Moss","suffix":""},{"id":629531858,"identity":"7e7ae664-b44b-4e19-b7ee-f5641197906e","order_by":3,"name":"Martin Vossiek","email":"","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":false,"prefix":"","firstName":"Martin","middleName":"","lastName":"Vossiek","suffix":""},{"id":629531860,"identity":"28b5ae6b-8c10-4dfa-8f63-31c000dcf618","order_by":4,"name":"Bjoern M. Eskofier","email":"","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":false,"prefix":"","firstName":"Bjoern","middleName":"M.","lastName":"Eskofier","suffix":""},{"id":629531861,"identity":"35f4e740-9b30-43f1-8378-31c36cbb872c","order_by":5,"name":"Matthias Berking","email":"","orcid":"","institution":"Friedrich-Alexander Universität Erlangen-Nürnberg","correspondingAuthor":false,"prefix":"","firstName":"Matthias","middleName":"","lastName":"Berking","suffix":""}],"badges":[],"createdAt":"2026-04-14 17:38:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9418466/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9418466/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108029386,"identity":"a1bc2677-a04f-4980-8817-9e86d7993e7e","added_by":"auto","created_at":"2026-04-28 15:41:08","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":386483,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript2026SleepAffect.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9418466/v1_covered_289ee780-e514-40cd-afe0-4da9f66183bc.pdf"},{"id":108029368,"identity":"30bcab58-69d4-46a5-9986-5aa8e6b6aa25","added_by":"auto","created_at":"2026-04-28 15:40:56","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":32508,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript2026SleepAffectTables.docx","url":"https://assets-eu.researchsquare.com/files/rs-9418466/v1/e14b63c580ce2b82e24fc5cc.docx"}],"financialInterests":"\u003cp\u003eNo competing interests reported.\u003c/p\u003e\n\u003cp\u003eTable 1 to 8 are available in the Supplementary Files section.\u003c/p\u003e","formattedTitle":"Automated assessment of nocturnal affect using infrared recordings of facial expressions","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Affect, Emotion, Machine Learning, Nocturnal Assessment, Sleep","lastPublishedDoi":"10.21203/rs.3.rs-9418466/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9418466/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe affective sciences have predominantly investigated affective states at wakefulness. This leaves a significant research gap since affective processes during sleep remain largely unexplored. This research gap is particularly important given that evidence suggests that sleep is one critical timespan for affective processing. To date however, methodological limitations challenge the assessment of affect during sleep. To overcome these challenges, we propose a novel, contactless approach for the automated assessment of nocturnal affect using infrared recordings of facial expressions (ASANA-IF). ASANA-IF consists of three components: (1) recordings of facial expressions during sleep with infrared cameras, (2) extracting affective states from these expressions using machine learning tools for objective emotion assessment (OpenFace, a validated tool based on deep neural networks), (3) aggregating these estimates within domains of emotion (e.g., anger, fear) with an algorithm developed for this purpose. To explore the feasibility and validity of ASANA-IF, we used this approach to assess nocturnal affective states in 4 participants over 21 nights from 3 perspectives. Here, we empirically tested the hypothesis that affective states during sleep (anxious, angry, sad and happy affect) would predict self-reports of respective affective states on the following day. Findings provide evidence for the feasibility of ASANA-IF. Moreover, they confirm the hypothesis that nocturnal affective states predict levels of respective affective states on the following day. Future research should further improve this new approach and use it to address research questions associated with assessing nocturnal affect in a feasible and valid way.\u003c/p\u003e","manuscriptTitle":"Automated assessment of nocturnal affect using infrared recordings of facial expressions","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-28 15:40:36","doi":"10.21203/rs.3.rs-9418466/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"713742f1-3edd-49e6-b414-112c19dbae62","owner":[],"postedDate":"April 28th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-28T15:40:37+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-28 15:40:36","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9418466","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9418466","identity":"rs-9418466","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}