Comparing remote smartphone-based semen assessments with laboratory evaluations in men unselected for fertility concerns

The RECAP study is a sub-cohort of the Growing-Up Today Study (GUTS)( 10 ), a prospective cohort of 27,805 offspring of participants in the Nurses’ Health Study II. Participants in GUTS were enrolled at age 9–17 years in 1996 or 2004. In the 2019 and 2021 follow-up questionnaires (median age in 2019: 32 years), male GUTS participants were asked whether they were interested in providing semen samples for research studies. Supplemental Figure 1 shows the study participants’ flow. We invited 718 male participants by email to participate in RECAP based on their stated interest in 2019/2021. A total of 243 men responded to the invitation, and 218 men were eligible and completed the electronic consent form. Of these, 169 men arranged a time to begin the study and were sent study kits between June 2019 and December 2022. Study kits contained: 1) a smartphone-based semen analyzer and supplies for semen sample collection, 2) indoor air and chemical exposure monitoring devices, 3) supplemental study questionnaires, and 4) supplies to return semen samples and monitoring devices. Of all men who received a study kit, 150 men successfully completed a remote semen analysis, sent a semen sample for delayed semen analysis, or both. Of these, 122 men (81.3%) returned samples to the lab and completed a remote smartphone-based semen analysis, 21 only sent semen samples (14.0%), and 7 (4.7%) completed only a smartphone assessment ( Supplemental Figure 1 ). Participants who were included in the RECAP study were older, less likely to be white, more likely to have ever used marijuana, more likely to be married, and more likely to have gotten a partner pregnant compared to the rest of the GUTS participants who answered in the 2019 questionnaire ( Supplemental Table 1 ). The Mass General Brigham Human Research Protection Program approved the RECAP study. All participants completed an electronic consent form before beginning the study. Participants received smartphone-based semen analyzers and detailed instructions on producing and collecting semen samples; and on processing the sample for using the smartphone analyzer and returning the remainder of the sample by mail ( Supplemental Figure 2 ). A video guide and contact information were provided for assistance ( 11 ). The semen analyzer, developed at low manufacturing costs ($4.45 material cost), utilizes a combination of hardware optics, a user-friendly microfluidic design, and a rapid image processing algorithm for on-phone analysis of fresh, unwashed, unprocessed semen samples ( 5 ). Participants were instructed to abstain from ejaculation for 3–5 days before providing a sample and reported the most recent ejaculation in a supplemental questionnaire. After 20 minutes of liquefication at room temperature, participants used a disposable pipette to deposit the sample into one chamber of a Leja slide. The slide was inserted into the custom printed smartphone attachment on the study-provided smartphone, and then participants were asked to turn on the integrated light in the attachment. They then initiated the custom smartphone application and were guided to record the semen analysis results. The application automatically logged the results, notified the participants that they had successfully analyzed their sample, and transmitted the results to investigators in real time. Data were classified into 4 categories: 1) No issues: samples in this category exhibited expected visual characteristics, including sperm cells and/or the presence of cell debris, consistent with typical semen samples, 2) Potential user issue: these samples lacked any visible sperm cells or cell debris indicating unusually empty semen sample or patterns, suggesting possible errors such as inadequate liquefaction, improper sample placement on the slide, or imaging of an incorrect slide region, 3) No video transmitted: these samples had no available backup video due to technical failures, preventing any manual confirmation of the analysis, 4) Blue/black screen visible in captured video: this classification was assigned to videos captured with the light-emitting diode (LED) turned off, resulting in either a blue-tinted screen or complete darkness due to insufficient illumination, rendering sperm analysis impossible. Data from categories 1, 2, and 3 were used in the analysis. Participants were only informed whether the test had been completed; they were blinded to the results. After completing at-home semen analysis, participants returned the smartphone analyzer to the investigators with the other study supplies. Semen samples were wrapped in an absorbent pad, placed into a sealed bag, and then put into the cardboard box inside a Styrofoam cooler with six freezer packs (placed on each side and on the top and bottom of the box). The box to ship the semen sample was separate from the one to return the other study materials. The box was shipped overnight for delayed semen analysis. Semen samples were received at the Massachusetts General Hospital Fertility Center and processed for semen analysis. The container was evaluated upon arrival to check if it was broken, leaking, or intact. Ejaculation volume was measured with a graduated serological pipet. A Computer Assisted Sperm Analysis (CASA) system was used to assess sperm concentration and total motility (IVOS II, version 14; Hamilton Thorne, Beverly, Massachusetts). Sperm morphology was evaluated manually according to Kruger’s strict criteria ( 12 ). Total sperm count (TSC) was calculated as sperm concentration multiplied by ejaculate volume measured in the laboratory. We dichotomized semen parameters according to the 2021 WHO reference limits for sperm concentration and TSC: 16 million per ml for sperm concentration and 39 million per ejaculate for TSC ( 13 ). As a part of the GUTS questionnaire, participants reported height, weight, race/ethnicity, educational attainment, employment status, medical history, and reproductive history. In addition, the RECAP supplemental questionnaire asked about sample collection time, last ejaculation date before sample collection, if the entire ejaculate had been captured in the collection jar, and if they had taken a hot bath or been in a hot tub or sauna in the previous 3–5 days. We also collected information on relevant urological medical and surgical history, including sexually transmitted infections, testicular torsion, and inguinal hernia repair. For participants who analyzed the same sample more than once with the smartphone analyzer, we retained the measurements with the highest quality assessment category and used the average of all the retained valid smartphone measures in that category for comparison against delayed CASA measures. We calculated descriptive statistics and the differences between results from the smartphone analyzer and laboratory CASA measurements for participants who had at least one valid smartphone analyzed measure and a delayed CASA measure and whose semen container was intact and not leaking at the time of arrival at the laboratory (n=92) ( Supplemental figure 1 ). Further, we used all valid smartphone-based assessments (n=116 assessments in 92 men) to calculate intraclass correlation coefficients (ICCs), as a measure of reproducibility, for log-transformed sperm concentration, total motility, and log-transformed TSC using mixed-effects regression models. We assessed the classification agreement between the results from the smartphone analyzer and laboratory CASA assessment according to WHO 2021 lower reference limits for concentration and TSC. We calculated sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy with Clopper-Pearson exact confidence interval ( 14 ). Additionally, we adjusted the cutoff values for concentration and TSC for the smartphone analysis to optimize the threshold and compared these five parameters accordingly. Lastly, logistic regression models were used to examine the extent to which smartphone-based motility assessments predicted having a sample with completely immotile sperm (0% motility) in the delayed CASA assessment, using as predictors the WHO reference values (=42%) ( 13 ) and increasing levels of sperm motility in the smartphone analysis (60%) roughly corresponding to quartiles of the observed distribution. This outcome is based on the report by Samplaski et al., demonstrating a linear association between transport time and motility in the mail-in semen analysis system( 15 ). We compared the semen parameters based on participants’ responses to whether the entire ejaculate was captured in the collection jar in the RECAP supplemental questionnaire. In addition, we performed the same assessments for conventional semen parameters and binary indicators for impaired semen quality (concentration and TSC) based on WHO reference value, excluding participants who did not collect the entire ejaculate. All statistical analyses were conducted using SAS 9.4 (SAS Institute. Inc., Cary, North Carolina).

The median (IQR) age of participants at the time of semen sample collection was 36.2 (34.1, 37.5) years ( Table 1 ). Over half had previously fathered a pregnancy, 12 participants had been diagnosed with any sexually transmitted infections, and 12 participants had been diagnosed with infertility ( Table1 ). The median abstinence time was 4 ( 3 , 4 ) days. Of 150 participants who attempted to provide any semen sample in remote and unsupervised conditions, 129 participants (85%) completed a smartphone-based semen analysis, of whom 106 participants (70%) had valid usable data for analysis ( Supplemental Figure 1 ). xAmong the 21 participants who provided laboratory samples but had no smartphone data, 14 (67%) showed login activity without video, suggesting they attempted but did not complete the smartphone analysis. In contrast, of the 7 participants with smartphone data but no laboratory sample, 6 (86%) returned other study materials, such as questionnaires or wristbands, indicating partial compliance with study instructions. Of the 129 participants who submitted semen analyses via the smartphone analyzer, 43 (33.3%) had no issues, 39 (30.2%) had potential user issues, and 24 (18.6%) had no video transmitted—these were considered usable. However, 23 (17.8%) submitted videos with a blue or black screen and were excluded from the analysis ( Supplemental Table 2 ). Among 106 participants who provided usable data, 24 participants (22.6%) completed 2 or more smartphone-based semen analyses using the same semen sample; the median (range) number of analyses per subject was 1 ( 1 – 5 ). Among 143 samples that arrived at the laboratory, 11 containers (7.7%) were broken or leaking, and 1 sample (0.7%) was empty. The median (IQR) interval from sample collection to lab analysis was 29.9 (26.5, 32.3) hours. When comparing semen parameters based on whether participants reported capturing the entire ejaculate in the collection jar, those who answered “no” showed lower concentration in the laboratory measurement ( Supplemental Table 3 ). The smartphone analyzer estimates of sperm concentration and total sperm count were higher than those of delayed CASA assessment in the andrology lab. However, the IQRs for the median difference include zero ( Table 2 ). Bland-Altman plots comparing the smartphone analyzer to delayed CASA assessments for sperm concentration and total sperm count suggest that differences between the two methods were higher as sperm concentration or count increased ( Figure 1 ). Smartphone based assessments were highly reproducible (ICCs 0.90–0.99) for sperm concentration, total sperm count and sperm motility. We then evaluated the ability of the smartphone-based assessment to categorize sperm concentration and TSC based on the WHO lower reference limit using delayed CASA as the standard measure ( Table 3 ). The smartphone-based system’s sensitivity and positive predictive value could not be calculated, as there were no true positive cases, but high specificity (86.2% [95% CI = 77.2, 92.7]) and negative predictive value (93.8 % [95% CI = 93.2, 94.2]) for identifying men with low sperm concentration (<16 ×10 6 /ml) according to 2021 WHO reference criteria were observed. For TSC, the smartphone-based system had moderate sensitivity (62.5% [95% CI = 24.5, 91.5]) and positive predictive value (29.4% [95% CI = 16.5, 46.9]) but high specificity (85.7% [95% CI = 76.4, 92.4]) and negative predictive value (96.0% [95% CI = 90.7, 98.3]) for identifying men with low total sperm count (<39 ×10 6 ) according to 2021 WHO reference criteria. For concentration, lowering the cutoff value in the smartphone assessment for identifying low sperm concentration cases determined by delayed CASA, increased the specificity and accuracy, while NPV remained consistent. ( Supplemental Table 4 ). For TSC, lowering the cutoff value in the smartphone assessment increased specificity and maintained accuracy, but decreased sensitivity, PPV, and NPV. ( Supplemental Table 5 ). As anticipated, the smartphone analyzer’s estimate for motility was also significantly higher than the delayed assessment in the lab ( Table 2 ). Bland-Altman plots comparing the smartphone analyzer to delayed CASA assessments for sperm motility illustrated negative proportionality ( Supplemental Figure 2 ). At delayed laboratory assessment, 19 participants had zero sperm total motility. There was a suggestion that sperm motility based on the smartphone assessment was inversely associated with the probability of having no motile sperm in the delayed laboratory assessment ( Supplemental Table 6 ). However, as this outcome has not been validated for clinical utility, the analysis should be interpreted with caution. Lastly, a sensitivity analysis excluding participants who did not capture the entire ejaculate demonstrated almost identical results ( Supplemental Table 7 , 8 ).

This study found that the at-home smartphone-based semen analyzer identified male partners who are unlikely to have low sperm concentration and TSC. While these findings suggest potential future applications of this technology as a research tool and a clinical screening tool, further studies are needed to establish its effectiveness. At-home smartphones may play an important role in reproductive telemedicine, particularly in resource-poor settings where people have less accessibility to fertility clinics.

This study compared information obtained from a smartphone-based semen parameter analyzer with fresh, unwashed, and unprocessed semen samples to the delayed CASA assessment at an academic fertility clinic laboratory among general reproductive-aged men in the U.S. We found that smartphone-based analyses are highly reproducible. However, the smartphone analyzer may systematically overestimate sperm concentration and total sperm count when used in unsupervised conditions. Nevertheless, smartphone-based semen analyzers had high specificity and negative predictive value for identifying men with low sperm concentration or TSC according to the 2021 WHO reference criteria and may be even better at ruling out severe oligospermia than oligospermia in general. These findings suggest that the smartphone analyzer can provide time-sensitive sperm motility data, reproducible sperm concentration, and total sperm count data from most users for research use and has the potential to be a screening tool in reproductive telemedicine. In previous studies, at-home semen analysis systems have been evaluated among subfertile patients presenting to fertility clinics ( 2 , 6 – 8 , 16 ). This study design offers obvious advantages for studies comparing different semen analysis methods, such as the ability to make synchronous comparisons from the same semen sample, easy access to men who are already producing semen samples, and the ability to address concerns of users in real time. In fact, during the development of the smartphone analyzer, we also utilized semen samples from male partners presenting to an academic fertility center. In this study, we reported 98% diagnostic accuracy when comparing dichotomized results based on WHO guidelines from the smartphone analyzed to clinical evaluations ( 5 ). Convenience notwithstanding, this study design may not fully account for important circumstances that may influence the performance of a semen analyzer when used at home, such as the ability to address questions about using the device in real time. Importantly, the observed distribution of semen parameters may differ systematically between men presenting to fertility centers seeking fertility care and men in the general population or otherwise not selected based on fertility or reproductive history ( 17 ). Since the accuracy of diagnostic or screening tests is influenced by disease prevalence ( 18 ), we were able to examine the performance of the smartphone analyzer among men unselected for fertility status, among whom the prevalence of low sperm concentration or low sperm count may resemble more closely that of the general population. Although the sensitivity and PPV were not assessable, given the very limited number of participants (n=5) with low sperm concentrations determined by delayed CASA assessment, the smartphone analyzer demonstrated moderate specificity and accuracy and a high NPV. We previously investigated the accuracy of a single semen sample in classifying dichotomized semen parameters among those who attended an academic fertility center ( 19 ). The NPV for concentration and TSC from the first semen sample was comparable to those in this study (97.9% for concentration, 96.9% for TSC in the previous study). Together, these findings suggest that when used remotely with minimal training, the smartphone analyzer may have a similar ability to rule out oligospermia as a single in-person assessment performed under standard clinical conditions. Few studies have evaluated the performance of at-home male fertility testing in real world conditions. A previous study evaluated the usability of the Trak At-Home Male Fertility Testing System among participants in the Pregnancy Study Online (PRESTO)( 17 , 20 ). During the development of the Trak system, all 239 participants completed the test and obtained results ( 20 ); nevertheless, approximately 30% of the participants (102 out of 373 consented participants) did not complete the upload of their first results and survey. Although approximately 70% of participants who attempted to provide semen samples provided analyzable data in our study, which was consistent with the PRESTO study, this proportion is relatively low compared to other smartphone analyzer validation studies conducted in fertility clinics, where usability among lay users was reported to be 93.3–100% ( 8 , 20 ). These findings suggest there may be other factors or difficulties in using such devices in a real-world setting that haven’t been well-studied or documented. For one, general research participants may not be as motivated to complete the analysis as individuals undergoing fertility treatment. Additionally, first-time semen sample collection may pose a greater challenge for participants with no prior experience, whereas subfertile individuals often have familiarity with the process due to clinical evaluations. Further research is needed to identify and address these barriers, ensuring the usability and effectiveness of at-home fertility testing in diverse populations. Although sperm motility is an important diagnosis parameter for male infertility ( 21 ), most home semen analysis kits predominantly measure sperm concentration ( 2 , 3 ). Since motility is strongly influenced by room temperature, humidity, and time since ejaculation, the WHO manual also recommends that it be evaluated as soon as possible after liquefaction is complete ( 13 ). However, in settings such as a research study or at-home screening, information on motility could be a useful measure available from home semen analysis systems. While the primary strength of the study stems from its novel study design where the results from at-home smartphones analyzers were compared to delayed CASA assessment using semen samples among men from the general population of the U.S., its robustness is further enhanced by the use of the same semen sample for both assessment methods, thereby minimizing inter-sample variability. Furthermore, the collection of repeated measures allowed for assessments on the reproducibility of the results from the smartphone analyzer. Lastly, our results may be more generalizable to evaluations utilized by the general population than samples from fertility clinic settings since we collected semen samples from general male partners without fertility concerns. Additionally, using the at-home analyzer made it possible to obtain information on motility from this geographically dispersed group of participants. However, it is also important to interpret the results of this study while being aware of its limitations. Firstly, the devices used in this study were lab-based prototypes fabricated primarily to collect data remotely for a larger epidemiological study, and thus, data collection was prioritized even in the event of possible user errors, such as poor lighting conditions or possible incomplete slide insertions. The data used in this study included data with potential user errors (n=39) and thus has a substantial effect on the comparative analysis of the technology itself. Secondly, the rapid 3D-printed lab-based prototypes, while previously evaluated in controlled settings, may have contributed to user errors due to factors such as structural fragility. However, the extent of this influence could not be assessed in this study. As a result, the study’s insights into the technology’s usability are limited, as the usability of commercial product versions is largely dependent on manufacturing robustness and design refinements. Next, we could not calculate sensitivity and PPV because no true positive cases existed in the classification agreement. This limitation is attributed to the low prevalence of low sperm concentration in the general population, as well as a reduction in the number of participants due to potential user errors in smartphone-based assessments and broken containers during sample delivery. This limits our ability to assess the device’s performance in subfertile men, but it does highlight its potential as a screening tool among men without a prior fertility evaluation. Finally, CASA assessments are not a gold standard measure of semen quality, especially post-shipment, as used in this study. However, with the advancement of technology, CASA has been increasingly used in research and clinical practice and can provide reliable results, especially in concentration and motility ( 22 ).

As the awareness of men’s contributions to infertility increases each year ( 1 ), semen self-testing kits are increasingly promoted as a viable alternative solution for those who recognize the importance of semen testing but are hesitant or unable to access clinics. Hence, at-home testing for male fertility has gained popularity in the last decade ( 2 , 3 ), offering the advantages of no-visiting time to clinics and mitigating embarrassment in collecting semen samples at clinics ( 4 ). While different at-home systems are generally tested in controlled conditions against standard clinical measures ( 2 , 5 – 8 ), their real-world performance remains largely unverified. At the same time, most male fertility research targets male partners in couples with fertility concerns or those presenting at fertility clinics ( 9 ). Thus, data on sperm quality and the usability of semen testing kits among men unselected for fertility status or concerns are also quite scarce. We have previously developed an automated smartphone-based microchip technology semen analysis system that can accurately measure sperm concentration and motility, allowing for the remote collection of time-sensitive results, such as sperm motility, while mitigating mental, physical, and logistical barriers to collecting semen samples for clinical or research purposes ( 5 ). The Reproductive Effects of Chemicals and Air Pollution (RECAP) primarily aims to investigate how air pollution and endocrine-disrupting chemicals affect semen quality and sperm DNA methylation in young men( 10 ). To reduce participants’ burden and facilitate remote assessment of fresh sperm motility, the RECAP study utilized the smartphone-based analyzer for semen sample collection at home. Although the RECAP study was not designed initially as a validation study, the current analysis leverages data collected from participants in a prospective cohort study who were unselected for fertility status and concerns, to evaluate the performance of this system under real-world conditions.