#ChronicPain: Automated Building of a Chronic Pain Cohort from Twitter Using Machine Learning.

This study was reviewed by the Institutional Review Board of Emory University and considered to be exempt (category 4; publicly available data). Figure 1 presents our data processing pipeline for this study. The overall study can be divided into 4 steps: (a) data collection, (b) manual annotation, (c) supervised classification, and (d) postclassification analyses. We now describe each of these steps. End-to-end data processing framework involving data collection/storage via the Twitter API, data preparation and preprocessing, training, evaluation, and classification using the final model and postclassification analyses. We collected data through the Twitter academic streaming application programming interface (API) [ 33 ]. The API enables the collection of a sample of data in real time based on keywords. We used the hashtag “#chronicpain” and the phrase “chronic pain ” . For the phrases, we first collected tweets using the terms individually and then applied a regular expression pattern to detect the exact phrase. We excluded retweets and tweets that did not meet this exact expression. We randomly selected 5,000 posts from our entire collected set for manual annotation. The objective of the annotation was to determine whether a post represented a personal experience or not. Thus, the annotation was a binary labeling task indicating if a tweet represented a self-report (label: Y) or something other than a self-report (label: N). Label N represented tweets that were general information about chronic pain or those that could be about raising awareness or promotion. We conducted the annotation in 3 batches. In the first batch, we annotated a small sample of tweets ( n = 200) without an explicit guideline and we discussed the disagreements from this round of annotation to create a simple annotation guideline. We annotated a larger sample ( n = 1,300; batch 2) following this guideline, and we further discussed disagreements and resolved them. We made minor updates to the annotation guideline based on these discussions. Since the disagreements were relatively low after the second round of annotations, we annotated the remaining tweets (batch 3). Randomly selected subsets of the tweets were annotated by 3 annotators. We used these overlapping annotations to compute interannotator agreements (IAA). Since the interpretation of the tweets can be subjective, it was important to capture the extent of human agreement as it represents a potential ceiling for any machine learning algorithm used to automate the process. We used the Cohen’s kappa measure to measure IAA. All disagreements were resolved by the first author of the article (A.S.). The annotation guideline is available as supplementary material. Table 1 presents examples of tweets and their labels. Sample posts representing self-reported chronic pain (Y) and generic chronic pain-mentioning posts (N). We divided the annotated data into 3 sets—60% for training, 20% for validation, and 20% for testing/evaluation. Transformer-based approaches that use large pretrained language models such as BERT are currently state-of-the-art for text classification, both in and outside of the medical domain. We, therefore, fine-tuned and evaluated several transformer-based classifiers. The following is an outline of the classifiers we used: 1. RoBERTa: A transformer-based model popular for its training on big batches and long sequences [ 34 ]. 2. SciBERT: This model is trained on a large corpus of scientific data and with the same model structure as BERT [ 35 ]. 3. BioClinicalBERT: This is extended from BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and is trained on MIMIC III notes [ 36 ]. 4. BERTweet: A large-scale pretrained model for English tweets [ 37 ]. 5. BioBERT: This is specifically trained for biomedical text and is widely used for biomedical text mining [ 38 ]. 6. Clinical_KB_BERT: This is trained over BioBERT using a joint training method that adds knowledge base information into the model from the Unified Medical Language System [ 39 ]. RoBERTa: A transformer-based model popular for its training on big batches and long sequences [ 34 ]. SciBERT: This model is trained on a large corpus of scientific data and with the same model structure as BERT [ 35 ]. BioClinicalBERT: This is extended from BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and is trained on MIMIC III notes [ 36 ]. BERTweet: A large-scale pretrained model for English tweets [ 37 ]. BioBERT: This is specifically trained for biomedical text and is widely used for biomedical text mining [ 38 ]. Clinical_KB_BERT: This is trained over BioBERT using a joint training method that adds knowledge base information into the model from the Unified Medical Language System [ 39 ]. We compared the performances of the classifiers based on the F 1 score for the Y class. The F 1 score is the harmonic mean of precision and recall. We focused our evaluation on the Y class since that is our class of interest. We also report overall classifier accuracy, but this metric is primarily driven by the majority (N) class. We computed the 95% confidence intervals for the Y class F 1 scores using the bootstrap resampling technique [ 40 ]. In this method, the F 1 score is calculated 1,000 times for randomly selected (with replacement) test set predictions. Out of these, 25th and 975th values are considered as lower bound and upper bound values at 95% confidence interval. We used the best-performing classifier (in terms of F 1 score) from the previous subsection to classify unlabeled posts collected from January 2021 to March 2022. Then, for a sample of subscribers whose posts were classified to be self-reports of chronic pain (Y), we collected all their past posts available via the API. The Twitter API allows the collection of approximately 3,200 past public posts by each subscriber. Thus, for many subscribers—members of our chronic pain cohort—this enabled us to obtain multiple years of posts. We were only able to collect this for a subset of the subscribers due to the API limitation of 10 million posts per month. Finally, we semiautomatically analyzed samples of the cohort posts to assess the presence and type of chronic pain-related information. To detect therapy-related information, we first created a lexicon of therapies, including medication and other therapies. Then, we applied a fuzzy matching approach [ 41 ] to detect potential therapies mentioned, including pharmaceuticals such as opioid medications (e.g., oxycodone and hydrocodone) and nonstandard ones such as behavioral therapies. Following these automatic steps, we performed a number of manual analyses involving the posts that mentioned specific therapies and also the past posts collected from the subscribers. We outline these below. We extracted posts that were detected to mention at least 1 therapy and compared their distributions. We also drew a small sample of posts mentioning therapies and conducted a sentiment analysis to obtain an estimate of how subscriber sentiments were associated with each therapy, if at all, and if the data held potentially differentiable therapy-specific sentiments. Two coauthors (A.L. and O.O.) manually reviewed each post and determined whether the sentiment expressed in relation to the therapy was (a) positive, (b) neutral, or (c) negative. We chose a manual approach for the sentiment analysis, rather than an automatic one, because there is no resource (e.g., lexicon) that specifically caters to this task. Many keywords associated with chronic pain posts have negative sentiments (e.g., the term “pain” has a negative sentiment score in the VADER [ 42 ] lexicon), leading to sentiment estimates being biased. We compared the overall distributions of the mentioned therapies in our cohort data and also the distributions of sentiment associated with each therapy. For Twitter subscribers who mentioned more than 3 chronic pain- or treatment-related posts, we also reviewed samples of their unlabeled tweets to identify other information relevant to chronic pain research and the topics of discussion. In-depth analyses of the types of information posted by the cohort and their quantification were considered to be outside the scope of this study. Instead, we simply focused on characterizing the types of relevant information present and verifying their contents for future analyses.

Of 5,000 posts selected for annotation, 2 were excluded due to issues with text encoding, leading to the annotation of 4,998 posts in total. A total of 550 posts were at least double annotated. Pair-wise IAA among 3 annotators was k = 0.82 (Cohen’s kappa [ 43 ]), which can be considered to be almost-perfect agreement [ 44 ] . A total of 719 (14.4%) were self-reports of chronic pain while the rest (85.6%) were not. The class distribution was thus slightly imbalanced. This was unsurprising since our past studies have found similar imbalances [ 11 , 45 ] . Table 2 presents the results of our automatic classification experiments. The table shows the overall accuracy of the classifiers, the F 1 scores for the positive class and the 95% confidence intervals. The RoBERTa classifier achieved the highest F 1 score among all the classifiers. For the best-performing classifier, we conducted ablation experiments using 10% subsets of the original training data. Figure 2 shows, and as is typically seen from such experiments, that the performance of the classifier increases approximately logarithmically as more training data is added. The best-fit logarithmic trendline suggests that F 1 scores of 0.87 would require us to increase the annotated dataset by 3 times. Accuracy scores are relatively unchanged with training data size since this value is primarily driven by the majority negative class. Classifiers, their overall accuracies, F 1 scores, and 95% confidence intervals (CIs) for the F 1 scores. Classifier F 1 scores or the positive (Y) class at different training set sizes and projected scores based on a logarithmic trendline for the RoBERTa model. The application of our classifier on the unlabeled posts resulted in the identification of 41,262 self-reports of chronic pain from 22,795 subscribers. Collecting their past data resulted in 3,461,619 posts (~151 posts per user on average). We performed NLP-driven and manual postclassification analyses on these posts, and we describe the findings in the following subsections. Table 3 presents the therapies we discovered from the cohort-posted data and their distributions. The therapies we discovered include prescription medications such as opioids, nonprescription pharmacological substances such as cannabidiol (CBD) and cannabis, physical therapies such as massage and chiropractic, and behavioral therapies such as meditation and yoga. We even discovered many examples of relatively unexplored therapies such as music therapy, aromatherapy, and guided imagery. Cannabis was the most commonly mentioned substance in the cohort timelines, although our manual review suggested that many of the posts were about recreational use rather than their use for treating chronic pain. A substantial number of tweets, however, did describe the use of cannabis-related products for treating chronic pain. Chiropractic was the most commonly mentioned physical/manual therapy. Table 4 presents examples of tweets mentioning therapies and their categories. As illustrated in the table, posts often describe self-management strategies for chronic pain, pharmacological substances, and their efficacies, comparisons between different pain management strategies, and adverse effects of therapies (e.g., opioid pain relievers) including long-term impacts (e.g., addiction). Therapies mentioned by our chronic pain cohort and their distributions in the longitudinal cohort data. Sample posts relevant to chronic pain detected within the public timelines of our automatically built cohort. The posts present a plethora of information including the types and descriptions of chronic pain, therapies, social support, questions about therapy and chronic pain, self-management strategies, mental health, and treatment access (including health insurance coverage). Sentiment analysis is performed to estimate the polarity distribution of feelings associated with the chronic pain experienced by the cohort members who also mention specific treatment keywords. Six hundred posts were manually annotated in total with an IAA of 0.88 (Cohen’s kappa). Figure 3 presents the sentiment distributions for different types of therapies. The figure shows that most of the users who mentioned “meditation”, “Guided imagery”, and “Tai Chi” in their posts, also showed more positive emotions, while users who mentioned “hydrocodone” and “tramadol” show negative emotions compared to others. Distribution of manually annotated sentiments (positive, negative, and neutral) for different chronic pain therapies, as discussed by the cohort members. Table 5 presents examples of posts from the longitudinal cohort data that present chronic pain-related information. The types of relevant information posted by the cohort include but are not limited to: descriptions of pain, therapies, impacts of therapies, self-management strategies, social support and its impact, mental health, treatment access-related information, and questions about chronic pain. The presence of such wide-ranging information suggests that publicly available data from this cohort may be invaluable in conducting long-term chronic pain-related studies. We further discuss the utilities of this data source and our data curation strategy in the following section. Sample posts relevant to chronic pain detected within the public timelines of our automatically built cohort. The posts present a plethora of information including the types and descriptions of chronic pain, therapies, social support, questions about therapy and chronic pain, self-management strategies, mental health, and treatment access (including health insurance coverage).

Our study verifies that social networks such as Twitter contain valuable information about chronic pain, posted by sufferers themselves. However, separating such information from the massive volumes of constantly generated data on Twitter is a challenging task, not manually achievable. Our methods automated the process of (a) detecting chronic pain-related posts automatically in real time, and (b) identifying posts from patients describing their own experiences. This strategy of automatic data collection and cohort curation will be able to establish a massive cohort over time and will enable us to study multiple years of data, including longitudinal data from the entire cohort and its subsets. This innovative strategy has additional advantages: (a) it allows us to include people who may not be reachable via traditional settings, such as hospital-based settings (e.g., because of lack of health insurance); (b) it is very cost-effective and is unobtrusive; and (c) it is able to continuously grow the cohort, thus enabling us to gather big data for long-term research. We now outline some related work in order to put our contributions into context. We then verify the utility of automatic classification approaches for monitoring and studying chronic pain. Finally, we present a brief discussion of errors to identify essential future improvements and we discuss some of the limitations of this study. Our findings illustrate that social media is a rich source of information for studying chronic pain. Chronic pain-related discussions are common on Twitter, but this resource has thus far been underutilized in both epidemiological and interventional research. A prime reason for this is the difficulty of handling such massive data and processing complex language. Our proposed machine learning and NLP methods have the potential of overcoming the barriers leading to the underutilization of such an effective source of information. Successful deployment of this pipeline is likely to increase the utility of social media in chronic pain research. Our study also found that the Twitter chronic pain cohort discusses a variety of pain-related topics including but not limited to therapies, addiction, mental health, and support. Conducting more detailed analyses of texts associated with these topics may lead to important breakthroughs in chronic pain research. For example, analyses of the opioid alternatives discussed by subscribers can generate hypotheses about their efficacies, which can be tested in the future through more traditional research mechanisms. Studying a social media-based cohort may also reveal information about factors impacting the sufferers of chronic pain that may not be available from any other source. These may include, for example, their social support (e.g., number of followers, number of interactions with a given post) and its influence (if any) on the quality of life of chronic pain patients. Data gathered by our methods may also enable researchers to connect alternative therapies and treatment strategies to social aspects of living with pain, ensuring their better efficacy. Monitoring the large cohort over time may lead to the discovery of social/therapeutic and efficacy-related aspects of alternative therapies that are not visible in traditional settings. The scale of knowledge that can be mined from this cohort is unparalleled, and the longitudinal nature of the data will enable the analyses of patient experiences and satisfaction in the short and long run. There is also the potential of going beyond epidemiological studies over social media and conducting interventional studies (e.g., reaching out to cohort members to suggest therapies, recruit for clinical studies, or to warn about potential adverse impacts of self-management strategies). We conducted a manual analysis of the misclassified tweets to identify patterns of errors. We focused on both false positives (classified erroneously as self-reports) and false negatives (classified as not self-reports). Under the former category, we found that most posts were misclassified when users intended to share the experiences of the people around them, not their personal experiences. One such post is, “today is mother's day and it's not an easy day. although my mom is still biologically alive, it feels like i've lost her to her chronic pain years ago. they say to treasure every moment but it's not always easy”. Under the latter category where model fails to classify as self-reports, we found that one common reason for misclassification was due to the expressions being implicit, making it difficult for the machine learning algorithms to capture their true meanings. One such example is, “ music therapy for chronicpain. thinking of ways to make movement enjoyable might involve music dance. sometimes I'll just chuck on some of my favorites and dance, mindfully. i always feel a million bucks after as well as more connected with my body.” Another reason for misclassification in this category was determined to be incomplete information/partial information. For instance, “ finally starting to ease, been trying to focus my mind on other things to get my brain to turn down the chronic pain it's at out of ” . While for a human reviewer the post has enough context, it could not be picked up by the machine learning classifier. The most important limitation of the current method is perhaps the performance of the machine learning classifier. The F 1 score for the positive class is not perfect (0.84), and there is room for improvement. Since most of the posts are not self-reports of chronic pain, the dataset is highly imbalanced—the negative class comprises most of the dataset. Improving performance over the minority class is a common challenge in text classification. The most common and effective strategy is perhaps to annotate more data manually so that the system can better generalize the features of the minority class instances. Our postclassification analyses show that the performance of the best classifier does not fully plateau with the data that is currently available for training, meaning that further annotation of data is likely to improve performance albeit only slightly. Another strategy that can be employed to improve classification performance is ensembling. In recent research, we have obtained performance improvements in social media text classification by applying fusion-based classification strategies [ 25 ]. We will attempt to apply similar strategies for this task, although, due to the high computational cost associated with running multiple classifiers in an ensemble, improving classification via further annotation is a more sustainable long-term approach. Another major limitation is associated with the platform—social media subscribers tend to be younger and often more tech-savvy compared to the general population. Thus, Twitter is not fully reflective of the U.S. or global population. It is likely that our cohort is skewed toward younger people and has an underrepresentation of older people. Recent surveys have shown that there is substantial variance in demographic characteristics across social networks [ 22 ], and thus, combining data or building a cohort from multiple social networks could lead to the creation of a better representative cohort. Collecting, processing, and aggregating data across multiple social networks is, however, a substantially more challenging task compared to the task described in this paper, and we leave such efforts for future work. Furthermore, only information publicly shared by our cohort is available for analysis. Self-reported information may be biased, and information never shared by a subscriber will thus be missing. The former limitations are not unique to Twitter or social media—no data source is bias-free. Social media, however, is perhaps the platform with the best reach of any. It is also possible that certain population groups will share more information over social media compared to others (e.g., women have been found to share more chronic pain-related information compared to men), and we did not adjust for that. While it is not possible for us to influence the demographic distribution of Twitter, in the future we will use automatic cohort characterization strategies for estimating the distributions of race, gender identity, and age-group of the cohort [ 46 ]. A careful study of the biases present in our social media cohort data will also be important future work [ 47 ]. Importantly, future studies relying on or utilizing our outputs/findings should be mindful of the many and evolving biases in social media data and the potential harms of propagating inequity. Our work is the first that builds an elaborate NLP and machine learning pipeline to curate knowledge about chronic pain from patient-generated social media data. The collection of cohort members via classification and their posts over time is fully automated, and so the cohort and the data will continue to grow over time. This will lead to the creation of an unprecedented resource for conducting long-term studies on the topic. With the detrimental public health impact of opioid pain relievers now well known, our social media mining infrastructure will be of particularly high utility for generating hypotheses about opioid alternatives from the Twitter cohort. Public release of our annotated data will enable the reproduction of this work and the deployment of this pipeline by other researchers. All pretrained models used in this study are publicly available.

Between 5.5% and 33% of the world’s and between 11% and 30% of the United States (U.S.) adult population are estimated to suffer from chronic pain [ 1 – 4 ]. The total financial burden of chronic pain in the U.S. is estimated to be between $560 and $635 billion per year [ 5 ]. Opioids have remained the go-to treatment for years [ 3 ], but they are addictive and they have been credited with heavily contributing to the epidemic of drug overdose-related deaths in the U.S., particularly in the early years. Due to the enormous financial and nonfinancial cost of the use of opioids as first-line therapy for chronic pain, recent research efforts have focused on identifying alternative treatments, including pharmacological, natural, physical, and behavioral therapies [ 6 ]. Currently, evidence is limited regarding the efficacy of many alternative therapies [ 6 – 10 ], the associations between types of pain and effective alternative management strategies, and the risks involved with the different strategies (e.g., long- and short-term side effects). It is difficult to establish the evidence and risk profiles of alternative therapies as they require studies in clinical settings, particularly trials. However, many alternative therapies have widespread use, and curating the knowledge from peoples’ experiences in a systematic manner may lead to the discovery of useful alternative therapies for targeted chronic pain types. Many recent studies have leveraged social media big data for obtaining insights directly from patients about targeted health-related topics including, but not limited to, substance use [ 11 – 14 ], adverse drug reactions [ 15 – 17 ], and mental health [ 18 – 20 ]. Over 220 million Americans (~70%) use social media, and there is an abundance of health-related information on such platforms [ 21 , 22 ]. Recent advances in data science and machine learning have enabled researchers to extract complex information expressed in natural language. These advances have made it possible to curate specialized cohorts from social media for conducting longitudinal studies based on data posted by the cohort members, such as breast cancer patients [ 23 ], pregnant people [ 24 ], and people who use substances nonmedically [ 25 ]. Social media thus present an attractive and underexplored opportunity for studying people suffering from chronic pain at low cost, unobtrusively, and at scale, provided that a chronic pain cohort can be built automatically from such sources, and longitudinal data from the cohort members can be collected and analyzed. Several past studies have demonstrated the utility of social media data for chronic pain-related research. Some early studies focused on studying the impacts of social media on chronic disease management, rather than the use of it to study chronic pain specifically [ 26 , 27 ]. Through a global online survey conducted partially via social media, Merolli et al. [ 27 ] concluded that areas of research that warranted attention included the ability to (a) filter information and guide people to pertinent information, (b) connect sufferers of chronic pain (i.e., cohort members), and (c) explore relationships between the therapeutic affordances of social media and health outcomes. In a separate, questionnaire-based study, Ressler et al. [ 28 ] found that posting about chronic pain online may decrease the sense of isolation and increase a sense of purpose. Similar findings were reported by Tsai et al. [ 29 ] in a more recent study. Gonzalez-Polledo et al. [ 30 ] studied social media (Tumblr and Flickr) and chronic pain from the perspective of digital anthropology, characterized chronic pain narratives over these platforms, and presented a typology of chronic pain expressions. Works most closely related to ours are those by Sendra and Farre, and Mullins et al. [ 31 , 32 ]. In the former study, the authors analyzed a small sample of data ( n = 350) and concluded that social media is changing the way patients live with their chronic pain and care providers could benefit from paying attention to self-reported information by these individuals. Mullins et al. collected a small sample of data from Twitter and performed natural language processing (NLP)-driven analysis of discussion topics, sentiments, and advice provided. The study concluded that the pain-related discussions on the platform can enrich our understanding of the chronic pain experience. In this paper, we take the first steps toward building a system that automatically detects self-reports of chronic pain from social media, specifically Twitter. Our study (a) verifies that reports of chronic pain are frequently posted on Twitter by sufferers themselves, (b) such reports can be automatically detected via NLP and supervised machine learning, and (c) longitudinal data posted by self-reported sufferers of chronic pain contain a trove of knowledge about chronic pain therapies, including nontraditional therapies and self-management strategies, their efficacies, and risks. We focus this paper on the chronic pain cohort-building process, leaving the tasks of mining longitudinal data and deriving potential causal associations as future work. The automatically detected cohort will continue to grow over time and will enable us to study a large group of people over many years. Our annotated dataset is made publicly available with this publication (see supplementary material) in order to enable interested researchers to reproduce our work and deploy the pipeline for their own analyses. The rest of the paper is organized as follows: in Methods, we describe the data collection, manual annotation, supervised classification, and postclassification analyses (sentiment and content analyses) approaches; the outputs from these approaches are presented in Results; in Discussion, we outline the utility of our approaches for creating and data from a chronic pain cohort from social media, discuss errors made by the supervised classification system and limitations of the current work, propose future work, and present some concluding remarks.