The Validity of Content Mapping: Let’s Call a Spade a Spade

The Validity of Content Mapping: Let’s Call a Spade a Spade | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Validity of Content Mapping: Let’s Call a Spade a Spade Jarl K. Kampen, Hilde Tobi, Jos Hagenaars, Marian Breuer This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4353956/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Concept Mapping (CM) is promoted as a research method suitable for interdisciplinary and international research, “… best suited to applications where diverse or wide-ranging opinions need to be gathered and made sense of." Our study does not support these claims. We created a situation where 54 statements were sorted on just 2 different attributes. Neither the conventional analytic approach in CM nor alternative clustering algorithms appear able to reveal these attributes. CM may be appropriate to use when groups can be assumed to be homogeneous, but CM cannot test this assumption. Discovering which attributes are used by sorters requires thematic analysis or focus groups. concept mapping multidimensional scaling cluster analysis inter-disciplinary research Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction Content Mapping is an exploratory research method that is “inherently integrative” in its use of both qualitative and quantitative procedures in a structured conceptualization process (see Dixon, 2009 , p. 87; Burke et al., 2005 ). The method was originally developed and introduced by Trochim ( 1989a ) and was designed to yield a representation of reality or an interesting suggestive map meant for planning and evaluation purposes. By the year 1989, projects in which Trochim had used concept mapping ranged from the identification of four staff members’ multicultural awareness goals for a day camp, to the production of a map as an organizing device for long-range planning efforts of the University Health Services at Cornell University (with between 50 and 75 participants), to the development of a framework for designing a training program for volunteers to work with mental patients (number of participants not given) (Trochim, 1989b ). Since the introduction of concept mapping (CM) in the 1980s, it’s participatory character has been recognized as an important and attractive characteristic (Burke et al. 2005 ) and useful in the development of a community based participatory research program (Windsor, 2013 ). CM has not only made use of written statements but also of photo-elicitation (Shannon et al., 2020). CM has been suggested as an alternative approach to the analysis of open ended items in questionnaires (Jackson et al., 2002). Concept mapping appears to have been mainly used in public health oriented research, human services, biomedical research, social science research, and business or human resources research (see Rosas & Kane, 2012 ). Because of the use of concept mapping in a wide range of academic disciplines, we were eager to learn more about the usefulness of concept mapping in interdisciplinary settings. After all, CM is being promoted as suitable for interdisciplinary and international research. Trochim & Kane (2005) state Concept Mapping is “purposefully designed to integrate input from multiple sources with differing content expertise or interest.” The aim of our study is to investigate the validity of concept mapping by means of a critical review of the procedure, and a series of experiments simulating an interdisciplinary research setting. In the following sections, we first present the CM procedure based on Kane & Trochim ( 2007 ). The described CM procedure raised a number of questions on how technical procedures are done exactly, why it is done that way and what the consequences of these choices are. In Section 3 we present a series of five computer experiments to investigate the validity of CM in simulated interdisciplinary research. For each of the experiments the methods are described, followed by the results and the discussion of these results as input for further experimentation and future research. We conclude with a discussion of the validity of CM in interdisciplinary setting and possible alterations. Our study did not require an ethical board approval because it did not directly involve humans or animals. 2. A short review of the procedure of concept mapping 2.1. Phase 1: Preparation The first phase in CM covers preparation. In this stage, the issue to be examined, the goals and desired outcomes are discussed and defined, based on which the facilitator and participants are selected and invited to form a panel. Research questions Kane & Trochim ( 2007 , p. 4) deem appropriate for addressing by CM include, among others, “What are the issues in a planning or evaluation project;” “Do the stakeholders have a common vision of what they are trying to achieve that enables them to stay on track throughout the life cycle of a project;” and “Can stakeholders link program outcomes to original expectations or intentions to see if they are achieving what they set out to achieve?” 2.2. Phase 2: Producing statements The panel is asked a question and participants formulate answers to that question in an individual or group ‘brainstorm session’. The resulting primary statement set is reduced and edited to ensure uniqueness, relevance, clarity and comprehension of the final statement set (Kane & Trochim 2007 , Chap. 3). The question posed to the participants is crucial for the statements that will be generated and, thereby, for the rest of the procedure. The editing by the researchers to reach the final statement set, gives plenty of room for interpretation, as is commonly the case in qualitative data reduction. 2.3. Phase 3: Sorting statements Next, each participant is asked to sort the Q statements in the final statement set in piles, on the basis of similarity between the statements. The sorting task gives no guidance to the panel members other than that piles must have a minimum of two statements and may not form a single pile. No information is supplied about the attributes to sort on, nor on the number of piles to aim at. After the sorting tasks, participants rate the Q statements on importance or priority on an ordinal scale (Kane & Trochim, Chap. 4). As the sorting task does not steer in any way, each participant can make a different number of piles based on a different set of attributes and (perceived) meaningful commonalities and differences on these attributes. Therefore, one would expect the data to contain plenty of variation. Intuitively we would assume that two statements j and k are more similar (i.e. conceptually close) to one another than to statement h when more panel members put statements j and k on the same pile, than either statements j and h or statements k and h. This principle brings us to the core of CM. 2.4 Phase 4: The core of the analysis First, a Q x Q similarity matrix is constructed on the basis of how frequent a statement ended up in the same pile as another statement. The similarity matrix is a symmetrical co-occurrence matrix (Leyesdorff & Vaughan, 2006 ; with citation from book Kruskal & Wish, 1978). With Q the number of statements and N the number of participants, the cells of the Q x Q similarity matrix can take on values between 0 and N. The more frequently statements j and k are put onto the same pile, the larger their perceived similarity and the smaller the distance between the statements. More formally, for each participant we may define: $${x}_{ijk}=\left\{ \begin{array}{c}1 \text{i}\text{f} {s}_{j} \text{a}\text{n}\text{d} {s}_{k} \text{w}\text{e}\text{r}\text{e} \text{p}\text{u}\text{t} \text{i}\text{n} \text{t}\text{h}\text{e} \text{s}\text{a}\text{m}\text{e} \text{p}\text{i}\text{l}\text{e} \text{b}\text{y} \text{p}\text{a}\text{r}\text{t}\text{i}\text{c}\text{i}\text{p}\text{a}\text{n}\text{t} i \\ 0 \text{i}\text{f} {s}_{j} \text{a}\text{n}\text{d} {s}_{k}\text{w}\text{e}\text{r}\text{e} \text{p}\text{u}\text{t} \text{i}\text{n} \text{d}\text{i}\text{f}\text{f}\text{e}\text{r}\text{e}\text{n}\text{t} \text{p}\text{i}\text{l}\text{e}\text{s} \text{b}\text{y} \text{p}\text{a}\text{r}\text{t}\text{i}\text{c}\text{i}\text{p}\text{a}\text{n}\text{t} i\end{array}\right.$$ 1 Not all participants are assumed to have sorted each statement. When in effect n jk participants have sorted both statements j and k , summing over all participants who sorted both statements gives the elements of the Q × Q similarity or co-occurrence matrix X (Leydesdorff & Vaughan, 2006): $${x}_{jk}=\sum _{i=1}^{{n}_{jk}}{x}_{ijk}$$ 2 . These similarities (or co-occurences) are transformed into the Q × Q Euclidian distance matrix D with $${d}_{jk}=\sqrt{\sum _{h=1}^{Q}{({x}_{jh}-{x}_{kh})}^{2}}$$ 3 . The distance matrix is input for multidimensional scaling (MDS), a data reduction technique in which the dimensionality is reduced based on distances (dissimilarities) or closeness (similarities) of the Q statements (ref; Lattin, Carroll & Green, 2003). In CM, MDS is used to reduce the dimensionality of the data from Q to, usually, 2. The choice to scale down to just 2 dimensions is motivated by Kruskal &Wish’s (1978) observation that “it is generally easier to work with two-dimensional configurations than with those involving more dimensions.” The coordinates of each statement in the resulting two-dimensional plane are used to identify meaningful clusters of statements by means of K-means cluster analysis using Ward’s method, as it ”…. generally gave more reasonable and interpretable solutions than other approaches such as single linkage or centroid methods” (Trochim, 1989: 8). The number of clusters in the initial solution is somewhere between 3–20 ( Trochim, 1989), while Rosas & Kanes’ ( 2012 ) pooled study analysis showed that final solutions on average present 9 clusters and range between 6–14 clusters. The solution, displayed in the so-called point cluster map, depicts the statements in the two-dimensional plane coloured by cluster membership. This attractive visualisation of clustered statements is subjected to interpretation of attributes in Phase 5. Some researchers have used hierarchical cluster analysis (HCA) instead of K-means clustering (e.g. Shannon et al., 2020). The order of first MDS and then cluster analysis (either K-means or HCA) has been debated by Péladeau et al. ( 2017 ) who claimed that the order is best changed into first cluster analysis, then MDS. Also, while it is true that the human mind can comprehend two dimensions better than higher amounts of dimensions, the two-dimensional solution may not adequately reflect distances between statements and may therefore distort interpretation. Given that producing a two-dimensional plot is the goal of these analytic steps, below we inquire if skipping MDS altogether and turning directly to the dendrogram of the HCA may produce an equivalent, if not better, overview of clusters of statements. 2.5 Phase 5: Interpretation of the clusters In the fifth and last stage, the point cluster maps of the statements are examined either by the participants or the researchers, and are given a name and meaning often based on so-called anchor statements. Sometimes, the K-means cluster analysis is repeated until an interpretable set of clusters is identified by the examiners. In the interpretation session, also consensus across groups or the consistency of results may be discussed (Kane & Trochim, 2007 , Chap. 6). 3. A series of simulation experiments 3.1 Aim The overall aim of our series of computer experiments was to discover whether the CM procedure described in the previous section is able to identify meaningful clusters in collections of sorted statements when the underlying cluster structure is known in a setting where the panel members come from different disciplines and therefore use different attributes for their classification. 3.2 Methods The series of simulation studies was designed on the model of sorting one standard deck of Q = 54 cards with numbers 2 through 10, Jack, Queen, King and Ace of Spades, Clubs, Hearts and Diamonds, and 2 Wildcards. The cards allow for (at least) 3 attributes for classification: Suits , resulting in 5 piles: Spades, Clubs, Hearts, Diamonds. and Wildcards Ranks , resulting in 11 piles: with piles Ace, 2, 3, ..., 10, and Picture cards (i.e. Jack, Queen, King, and Wildcard) Odd-Even-Other , resulting in 3 piles: with piles odd numbers (including ace), even numbers, and Picture cards We assumed that each participant classified cards according to 1 (and just 1) attribute. In order to introduce some random noise in the data, we further assumed that within a selected attribute for classification, respondents had a high probability to correctly classify (95%), and any wrongly classified card had an equal probability to end up in one of the other piles. Note that a high probability of correct classification prevents the need for replication. Finally, we assume that all participants sorted all statements (so that n jk = n for all j , k ). The simulations produced 54×54 co-occurrence matrices that were input for further analysis i.e. the MDS as prescribed in CM with two dimensions followed by a K-means clustering using Ward’s method (see Everitt, 1980 , p. 65). We will use K = 10. Five experiments were executed. The first two experiments supply a proof of principle in which the code is checked, and results from MDS2 and HCA are compared for n = 40 respondents. In these simulations, all respondents used the same attribute for classification (either suits, or ranks). The third and further experiments compare the results of MDS2 with HCA when half of the respondents used one, and half of the respondents used another attribute for classification (simulating a multi-disciplinary background of the participants). 3.3 Results Figures 1a and 1b provide the point cluster map and the dendrogram using n = 40 sorters all using Suits as the attribute for classification. Both approaches, CM and HCA, successfully revealed 5 clusters. However, in the second experiment using Ranks as attribute, the Content Mapping based on 2 MDS dimensions failed to identify the 11 cluster structure (see Fig. 2a; note that the configuration suggests a 6 cluster solution rather than the 10 cluster solution forced on the data), which can clearly be discerned in the dendrogram produced by hierarchical cluster analysis using Ward’s method directly on the distance matrix (Fig. 2b). Figures 3a and 3b depict the results for a simulation where half of the sample ( n = 20) used Suit, and half of the sample ( n = 20) used Ranks for sorting. Inspection of the point cluster map and the dendrogram supports the interpretation that cards are ranked by suit, and that number cards receive different treatment than picture cards. At no point would one conclude based on the point cluster map, that Rank played a role in sorting. In order to eliminate the possibility that this is just a small sample problem (small n relative to the number of statements), we increased the sample size by a factor 10 to 400 participants and replicated the last experiment (figures not printed to save space). Again, either clustering technique failed to identify Ranks as a used attribute. Further trials using variations of the proportion of participants using either attribute, revealed that when a clear majority of 70% or more used Rank the cluster solutions reveal Rank as sorting criterion, but this solution then failed to identify Suit as a used attribute. The fifth and final experiment aimed to find out if the number of piles relative to the number of cards may play a role in the outcome of CM. In this experiment we combined 20 respondents sorting on the basis of Suits (5 piles) with 20 respondents sorting on the basis of Odd-Even (3 piles). The results are in Figs. 4a and 4b. Focusing on the dendrogram, we find clusters of e.g., even and uneven hearts, even and uneven spades, picture diamonds, etc., and we would conclude that (un)evenness by suit would be the hybrid attribute for sorting. The dendrogram does not reflect two different groups sorting on Suits or on odd-Even exclusively. 3.4 Discussion The two different visualizations of the five classification problems suggest that dendrograms may be more useful to interpret the clusters than point cluster maps (particularly when the number of clusters is large and the names of the statements are complex). Skipping MDS did not cost: dendrograms are easier to interpret than point cluster maps and one does not have to make the assumption that the dimension Q can be reduced to 2. However, neither method is capable to reveal mixtures of attributes. Kane and Trochim consider so-called bridging values important in determining the contents of clusters, which in the first experiment could potentially lead to choosing 4 clusters instead of 5 because the Wildcards are close to another cluster and can be conceived as forming one single cluster. Using bridging values will not help in unmixing mixed sets of used attributes. In these situations, the methods allow either for identification of the attribute with the lowest number of piles, or of some (non-existing) hybrid of attributes, but is unable to separate the two different attributes for sorting. Note that in the highly simplified situation of our experiments, there was no missing data: all participants sorted all statements. There is far more complexity in reality than that: people from the same discipline may actually use different attributes to sort into clusters, and individuals may change or mix attributes during the sorting task (e.g., from red-black to suits) or change the number of values (and thus the number of clusters) they are sorting on (e.g., first sorting numbers 1, 2, ... 10, then sorting odd/even numbers and face cards). 4. Conclusions and outlook We have created a highly simplified situation where the same statements were sorted on 2 different attributes and both CM and HCA failed to reveal these attributes. The results do not support the claims on the suitability of CM for interdisciplinary research. In applications of CM, the attributes for sorting statements are unknown and the analysis aims to identify the attributes. In the face of failure of clustering algorithms to identify these attributes when they are diverse (as should be expected in the case in interdisciplinary research) a qualitative analysis of the statements may be more fruitful. For example, generated statements may be subjected to content analysis, or insight in the possible attributes that were used for sorting of the statements can be acquired by a thematic analysis of the obtained piles. Also, focus groups discussing the meaning of each statement or the relevance of prior identified possibly relevant attributes may reveal dominant attributes within particular groups as well as differences of attribute selection between groups. Declarations Author Contribution JK and HT wrote the main manuscript. JH and JK wrote the script for R. All authors reviewed the manuscript. Data Availability Data and analysis code is provided within supplementary information files. References Burke, J. G., O’Campo, P., Peak, G. L., Gielen, A. C., McDonnell, K. A., & Trochim, W. M. (2005). An introduction to concept mapping as a participatory public health research method. Qualitative health research, 15(10) , 1392-1410. DOI: 10.1177/1049732305278876. Dixon, J. K. (2009). Media Review: Kane, M., & Trochim, WMK (2007). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage. Journal of Mixed Methods Research , 3 (1), 87-89. DOI: 10.1177/1558689808326121 Everitt, B. (1980). Cluster analysis. 2 nd Edition. New York: Halsted Press. Kane M & Trochim WM (2007). Concept mapping for planning and evaluation. London: SAGE. DOI: 10.4135/9781412983730 Lattin J, Carrol JD, Green PE (2003). Analyzing multivariate data. Brooks/Cole Thomson Learning: Pacific Grove CA, USA. Leyesdorff L. & Vaughan (2006) Co-occurrence matrices and their applications in information science: Extending ACA to the Web environment. Journal of the American Society for Information Science and Technology 57 , 1616-1628. DOI: 10.1002/asi.20335 Péladeau N, Dagenais C, Ridde V (2017). Concept mapping internal validity: A case of misconceived mapping? Evaluation and Program Planning, 62, 56–63. http://dx.doi.org/10.1016/j.evalprogplan.2017.02.005 Rosas, S. R., & Kane, M. (2012). Quality and rigor of the concept mapping methodology: a pooled study analysis. Evaluation and program planning, 35(2) , 236-245. http://dx.doi.org/10.1016/j.evalprogplan.2011.10.003 Shannon, J., Borron, A., Kurtz, H., & Weaver, A. (2021). Re-envisioning emergency food systems using photovoice and concept mapping. Journal of Mixed Methods Research , 15 (1), 114-137. https://doi.org/10.1177/1558689820933778 Trochim WMR (1989a). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12 , 1-16. https://doi.org/10.1016/0149-7189%2889%2990016-5 Trochim WMR (1989b). Concept mapping: Soft science or hard art? Evaluation and Program Planning, 12, 87-110. https://doi.org/10.1016/0149-7189(89)90027-X Windsor, L. C. (2013). Using concept mapping in community-based participatory research: A mixed methods approach. Journal of mixed methods research , 7 (3), 274-293. http://dx.doi.org/10.1177/1558689813479175 Additional Declarations No competing interests reported. Supplementary Files CMCardsSimulation.r CardDeckWithDimensions.csv Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4353956","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":298170701,"identity":"afb9b4c8-0ffa-43a6-9ecf-90623bd2e8cb","order_by":0,"name":"Jarl K. Kampen","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAz0lEQVRIiWNgGAWjYBACxgYGxgMMByDsB2CKnbAWBpgWZgMIRYRNMC1sEkRpYW4/++AAwxm7aHP23mPVvG33GMwJaWHsSTc4wHAjOXdnz7m027xtxQyWzYS0NKQBnfWBOXfDjRyz27ltCQwGhwlp6X8G0lKfu+H+G7Ni4rTMANly4zDQFh4zZiK1AG1JOHMc6JccY+k/5xJ4CGox7E9jfPDhWHXudvYzhh9nlCXIGRxvIKAFJJ8AxAZQAR4CdjAwyMMYBvhUjYJRMApGwcgGAH+2RtevKPPYAAAAAElFTkSuQmCC","orcid":"","institution":"Wageningen University \u0026 Research","correspondingAuthor":true,"prefix":"","firstName":"Jarl","middleName":"K.","lastName":"Kampen","suffix":""},{"id":298170704,"identity":"8431c536-cbaf-4595-8d43-61e54350ebc0","order_by":1,"name":"Hilde Tobi","email":"","orcid":"","institution":"Wageningen University \u0026 Research","correspondingAuthor":false,"prefix":"","firstName":"Hilde","middleName":"","lastName":"Tobi","suffix":""},{"id":298170705,"identity":"5174948c-e6c7-4611-8235-0f1a3f147f46","order_by":2,"name":"Jos Hagenaars","email":"","orcid":"","institution":"Wageningen University \u0026 Research","correspondingAuthor":false,"prefix":"","firstName":"Jos","middleName":"","lastName":"Hagenaars","suffix":""},{"id":298170706,"identity":"e46007c2-fe98-4cad-9f8d-d62ec7dd5551","order_by":3,"name":"Marian Breuer","email":"","orcid":"","institution":"Radboud University Nijmegen Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Marian","middleName":"","lastName":"Breuer","suffix":""}],"badges":[],"createdAt":"2024-05-01 11:23:45","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4353956/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4353956/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":56143471,"identity":"2aa52a13-61aa-4cc1-853e-1b13660b0d1c","added_by":"auto","created_at":"2024-05-09 05:06:02","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":115565,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ea.\u003c/strong\u003e Point cluster map of 40 respondents using Suits\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eb.\u003c/strong\u003e Dendrogram of 40 respondents using Suits\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/f18d9fe609037904b57f03e6.jpg"},{"id":56143468,"identity":"7cacfd03-ff83-4606-85ec-00f46ae4d8f4","added_by":"auto","created_at":"2024-05-09 05:06:01","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":296603,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ea.\u003c/strong\u003e Point cluster map of 40 respondents using Ranks\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eb.\u003c/strong\u003e Dendrogram of 40 respondents using Ranks\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/41403c1dcb610f4a2c57bb90.jpg"},{"id":56143599,"identity":"4817f071-47d3-4c1f-94f1-f56f17c2309d","added_by":"auto","created_at":"2024-05-09 05:06:36","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":96616,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ea.\u003c/strong\u003e Point cluster map of n=20 using Ranks and n=20 using Suits\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eb.\u003c/strong\u003e Dendrogram of n=20 using Ranks and n=20 using Suits\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/c2037e124a4b5c53332d4ed6.jpg"},{"id":56143267,"identity":"ad19b6f9-3f10-406f-b4fb-8506bbebcf17","added_by":"auto","created_at":"2024-05-09 05:05:38","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":299958,"visible":true,"origin":"","legend":"\u003cp\u003ea. Point cluster map of n=20 using Suits and n=20 using Odd-Even\u003c/p\u003e\n\u003cp\u003eb. Dendrogram of n=20 using Suits and n=20 using Odd-Even\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/e1d6281aab6e1110100e3e0f.jpg"},{"id":56319874,"identity":"e44c8714-2d3f-40e5-b40a-d855cfbd7ea0","added_by":"auto","created_at":"2024-05-12 03:16:33","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1159274,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/1a47aed2-fdfa-40c6-85e9-7bfae23cf11b.pdf"},{"id":56143324,"identity":"f5449c3e-f164-44a6-91da-23d1cfc0ae8b","added_by":"auto","created_at":"2024-05-09 05:05:43","extension":"r","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":4038,"visible":true,"origin":"","legend":"","description":"","filename":"CMCardsSimulation.r","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/4b98834f3ab9b05f2fb281e6.r"},{"id":56143262,"identity":"d8421951-b87e-477c-ac46-2b7c2ea98292","added_by":"auto","created_at":"2024-05-09 05:05:31","extension":"csv","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":1176,"visible":true,"origin":"","legend":"","description":"","filename":"CardDeckWithDimensions.csv","url":"https://assets-eu.researchsquare.com/files/rs-4353956/v1/f4abda062b4165a59899ac22.csv"}],"financialInterests":"No competing interests reported.","formattedTitle":"The Validity of Content Mapping: Let’s Call a Spade a Spade","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eContent Mapping is an exploratory research method that is \u0026ldquo;inherently integrative\u0026rdquo; in its use of both qualitative and quantitative procedures in a structured conceptualization process (see Dixon, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2009\u003c/span\u003e, p. 87; Burke et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2005\u003c/span\u003e). The method was originally developed and introduced by Trochim (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e1989a\u003c/span\u003e) and was designed to yield a representation of reality or an interesting suggestive map meant for planning and evaluation purposes. By the year 1989, projects in which Trochim had used concept mapping ranged from the identification of four staff members\u0026rsquo; multicultural awareness goals for a day camp, to the production of a map as an organizing device for long-range planning efforts of the University Health Services at Cornell University (with between 50 and 75 participants), to the development of a framework for designing a training program for volunteers to work with mental patients (number of participants not given) (Trochim, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e1989b\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSince the introduction of concept mapping (CM) in the 1980s, it\u0026rsquo;s participatory character has been recognized as an important and attractive characteristic (Burke et al. \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2005\u003c/span\u003e) and useful in the development of a community based participatory research program (Windsor, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). CM has not only made use of written statements but also of photo-elicitation (Shannon et al., 2020). CM has been suggested as an alternative approach to the analysis of open ended items in questionnaires (Jackson et al., 2002).\u003c/p\u003e \u003cp\u003eConcept mapping appears to have been mainly used in public health oriented research, human services, biomedical research, social science research, and business or human resources research (see Rosas \u0026amp; Kane, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2012\u003c/span\u003e). Because of the use of concept mapping in a wide range of academic disciplines, we were eager to learn more about the usefulness of concept mapping in interdisciplinary settings. After all, CM is being promoted as suitable for interdisciplinary and international research. Trochim \u0026amp; Kane (2005) state Concept Mapping is \u0026ldquo;purposefully designed to integrate input from multiple sources with differing content expertise or interest.\u0026rdquo;\u003c/p\u003e \u003cp\u003eThe aim of our study is to investigate the validity of concept mapping by means of a critical review of the procedure, and a series of experiments simulating an interdisciplinary research setting. In the following sections, we first present the CM procedure based on Kane \u0026amp; Trochim (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). The described CM procedure raised a number of questions on how technical procedures are done exactly, why it is done that way and what the consequences of these choices are. In Section \u003cspan refid=\"Sec8\" class=\"InternalRef\"\u003e3\u003c/span\u003e we present a series of five computer experiments to investigate the validity of CM in simulated interdisciplinary research. For each of the experiments the methods are described, followed by the results and the discussion of these results as input for further experimentation and future research. We conclude with a discussion of the validity of CM in interdisciplinary setting and possible alterations. Our study did not require an ethical board approval because it did not directly involve humans or animals.\u003c/p\u003e"},{"header":"2. A short review of the procedure of concept mapping","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Phase 1: Preparation\u003c/h2\u003e \u003cp\u003eThe first phase in CM covers preparation. In this stage, the issue to be examined, the goals and desired outcomes are discussed and defined, based on which the facilitator and participants are selected and invited to form a panel. Research questions Kane \u0026amp; Trochim (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2007\u003c/span\u003e, p. 4) deem appropriate for addressing by CM include, among others, \u0026ldquo;What are the issues in a planning or evaluation project;\u0026rdquo; \u0026ldquo;Do the stakeholders have a common vision of what they are trying to achieve that enables them to stay on track throughout the life cycle of a project;\u0026rdquo; and \u0026ldquo;Can stakeholders link program outcomes to original expectations or intentions to see if they are achieving what they set out to achieve?\u0026rdquo;\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Phase 2: Producing statements\u003c/h2\u003e \u003cp\u003eThe panel is asked a question and participants formulate answers to that question in an individual or group \u0026lsquo;brainstorm session\u0026rsquo;. The resulting primary statement set is reduced and edited to ensure uniqueness, relevance, clarity and comprehension of the final statement set (Kane \u0026amp; Trochim \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2007\u003c/span\u003e, Chap.\u0026nbsp;3). The question posed to the participants is crucial for the statements that will be generated and, thereby, for the rest of the procedure. The editing by the researchers to reach the final statement set, gives plenty of room for interpretation, as is commonly the case in qualitative data reduction.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3. Phase 3: Sorting statements\u003c/h2\u003e \u003cp\u003eNext, each participant is asked to sort the Q statements in the final statement set in piles, on the basis of similarity between the statements. The sorting task gives no guidance to the panel members other than that piles must have a minimum of two statements and may not form a single pile. No information is supplied about the attributes to sort on, nor on the number of piles to aim at. After the sorting tasks, participants rate the Q statements on importance or priority on an ordinal scale (Kane \u0026amp; Trochim, Chap.\u0026nbsp;4). As the sorting task does not steer in any way, each participant can make a different number of piles based on a different set of attributes and (perceived) meaningful commonalities and differences on these attributes. Therefore, one would expect the data to contain plenty of variation. Intuitively we would assume that two statements j and k are more similar (i.e. conceptually close) to one another than to statement h when more panel members put statements j and k on the same pile, than either statements j and h or statements k and h. This principle brings us to the core of CM.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Phase 4: The core of the analysis\u003c/h2\u003e \u003cp\u003eFirst, a Q x Q similarity matrix is constructed on the basis of how frequent a statement ended up in the same pile as another statement. The similarity matrix is a symmetrical co-occurrence matrix (Leyesdorff \u0026amp; Vaughan, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2006\u003c/span\u003e; with citation from book Kruskal \u0026amp; Wish, 1978). With Q the number of statements and N the number of participants, the cells of the Q x Q similarity matrix can take on values between 0 and N. The more frequently statements j and k are put onto the same pile, the larger their perceived similarity and the smaller the distance between the statements. More formally, for each participant we may define:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$${x}_{ijk}=\\left\\{ \\begin{array}{c}1 \\text{i}\\text{f} {s}_{j} \\text{a}\\text{n}\\text{d} {s}_{k} \\text{w}\\text{e}\\text{r}\\text{e} \\text{p}\\text{u}\\text{t} \\text{i}\\text{n} \\text{t}\\text{h}\\text{e} \\text{s}\\text{a}\\text{m}\\text{e} \\text{p}\\text{i}\\text{l}\\text{e} \\text{b}\\text{y} \\text{p}\\text{a}\\text{r}\\text{t}\\text{i}\\text{c}\\text{i}\\text{p}\\text{a}\\text{n}\\text{t} i \\\\ 0 \\text{i}\\text{f} {s}_{j} \\text{a}\\text{n}\\text{d} {s}_{k}\\text{w}\\text{e}\\text{r}\\text{e} \\text{p}\\text{u}\\text{t} \\text{i}\\text{n} \\text{d}\\text{i}\\text{f}\\text{f}\\text{e}\\text{r}\\text{e}\\text{n}\\text{t} \\text{p}\\text{i}\\text{l}\\text{e}\\text{s} \\text{b}\\text{y} \\text{p}\\text{a}\\text{r}\\text{t}\\text{i}\\text{c}\\text{i}\\text{p}\\text{a}\\text{n}\\text{t} i\\end{array}\\right.$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eNot all participants are assumed to have sorted each statement. When in effect \u003cem\u003en\u003c/em\u003e\u003csub\u003e\u003cem\u003ejk\u003c/em\u003e\u003c/sub\u003e participants have sorted both statements \u003cem\u003ej\u003c/em\u003e and \u003cem\u003ek\u003c/em\u003e, summing over all participants who sorted both statements gives the elements of the \u003cem\u003eQ \u0026times; Q\u003c/em\u003e similarity or co-occurrence matrix \u003cem\u003eX\u003c/em\u003e (Leydesdorff \u0026amp; Vaughan, 2006):\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$${x}_{jk}=\\sum _{i=1}^{{n}_{jk}}{x}_{ijk}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e.\u003c/p\u003e \u003cp\u003eThese similarities (or co-occurences) are transformed into the \u003cem\u003eQ \u0026times; Q\u003c/em\u003e Euclidian distance matrix \u003cem\u003eD\u003c/em\u003e with\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e\n$${d}_{jk}=\\sqrt{\\sum _{h=1}^{Q}{({x}_{jh}-{x}_{kh})}^{2}}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e.\u003c/p\u003e \u003cp\u003eThe distance matrix is input for multidimensional scaling (MDS), a data reduction technique in which the dimensionality is reduced based on distances (dissimilarities) or closeness (similarities) of the \u003cem\u003eQ\u003c/em\u003e statements (ref; Lattin, Carroll \u0026amp; Green, 2003). In CM, MDS is used to reduce the dimensionality of the data from \u003cem\u003eQ\u003c/em\u003e to, usually, 2. The choice to scale down to just 2 dimensions is motivated by Kruskal \u0026amp;Wish\u0026rsquo;s (1978) observation that \u0026ldquo;it is generally easier to work with two-dimensional configurations than with those involving more dimensions.\u0026rdquo; The coordinates of each statement in the resulting two-dimensional plane are used to identify meaningful clusters of statements by means of K-means cluster analysis using Ward\u0026rsquo;s method, as it \u0026rdquo;\u0026hellip;. generally gave more reasonable and interpretable solutions than other approaches such as single linkage or centroid methods\u0026rdquo; (Trochim, 1989: 8). The number of clusters in the initial solution is somewhere between 3\u0026ndash;20 ( Trochim, 1989), while Rosas \u0026amp; Kanes\u0026rsquo; (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2012\u003c/span\u003e) pooled study analysis showed that final solutions on average present 9 clusters and range between 6\u0026ndash;14 clusters. The solution, displayed in the so-called point cluster map, depicts the statements in the two-dimensional plane coloured by cluster membership. This attractive visualisation of clustered statements is subjected to interpretation of attributes in Phase 5.\u003c/p\u003e \u003cp\u003eSome researchers have used hierarchical cluster analysis (HCA) instead of K-means clustering (e.g. Shannon et al., 2020). The order of first MDS and then cluster analysis (either K-means or HCA) has been debated by P\u0026eacute;ladeau et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) who claimed that the order is best changed into first cluster analysis, then MDS. Also, while it is true that the human mind can comprehend two dimensions better than higher amounts of dimensions, the two-dimensional solution may not adequately reflect distances between statements and may therefore distort interpretation. Given that producing a two-dimensional plot is the goal of these analytic steps, below we inquire if skipping MDS altogether and turning directly to the dendrogram of the HCA may produce an equivalent, if not better, overview of clusters of statements.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 Phase 5: Interpretation of the clusters\u003c/h2\u003e \u003cp\u003eIn the fifth and last stage, the point cluster maps of the statements are examined either by the participants or the researchers, and are given a name and meaning often based on so-called anchor statements. Sometimes, the K-means cluster analysis is repeated until an interpretable set of clusters is identified by the examiners. In the interpretation session, also consensus across groups or the consistency of results may be discussed (Kane \u0026amp; Trochim, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2007\u003c/span\u003e, Chap.\u0026nbsp;6).\u003c/p\u003e \u003c/div\u003e"},{"header":"3. A series of simulation experiments","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Aim\u003c/h2\u003e \u003cp\u003eThe overall aim of our series of computer experiments was to discover whether the CM procedure described in the previous section is able to identify meaningful clusters in collections of sorted statements when the underlying cluster structure is known in a setting where the panel members come from different disciplines and therefore use different attributes for their classification.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Methods\u003c/h2\u003e \u003cp\u003eThe series of simulation studies was designed on the model of sorting one standard deck of \u003cem\u003eQ\u003c/em\u003e\u0026thinsp;=\u0026thinsp;54 cards with numbers 2 through 10, Jack, Queen, King and Ace of Spades, Clubs, Hearts and Diamonds, and 2 Wildcards. The cards allow for (at least) 3 attributes for classification:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eSuits\u003c/em\u003e, resulting in 5 piles: Spades, Clubs, Hearts, Diamonds. and Wildcards\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eRanks\u003c/em\u003e, resulting in 11 piles: with piles Ace, 2, 3, ..., 10, and Picture cards (i.e. Jack, Queen, King, and Wildcard)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eOdd-Even-Other\u003c/em\u003e, resulting in 3 piles: with piles odd numbers (including ace), even numbers, and Picture cards\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eWe assumed that each participant classified cards according to 1 (and just 1) attribute. In order to introduce some random noise in the data, we further assumed that within a selected attribute for classification, respondents had a high probability to correctly classify (95%), and any wrongly classified card had an equal probability to end up in one of the other piles. Note that a high probability of correct classification prevents the need for replication. Finally, we assume that all participants sorted all statements (so that \u003cem\u003en\u003c/em\u003e\u003csub\u003e\u003cem\u003ejk\u003c/em\u003e\u003c/sub\u003e = \u003cem\u003en\u003c/em\u003e for all \u003cem\u003ej\u003c/em\u003e, \u003cem\u003ek\u003c/em\u003e). The simulations produced 54\u0026times;54 co-occurrence matrices that were input for further analysis i.e. the MDS as prescribed in CM with two dimensions followed by a K-means clustering using Ward\u0026rsquo;s method (see Everitt, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e1980\u003c/span\u003e, p. 65). We will use \u003cem\u003eK\u003c/em\u003e\u0026thinsp;=\u0026thinsp;10.\u003c/p\u003e \u003cp\u003eFive experiments were executed. The first two experiments supply a proof of principle in which the code is checked, and results from MDS2 and HCA are compared for \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;40 respondents. In these simulations, all respondents used the same attribute for classification (either suits, or ranks). The third and further experiments compare the results of MDS2 with HCA when half of the respondents used one, and half of the respondents used another attribute for classification (simulating a multi-disciplinary background of the participants).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Results\u003c/h2\u003e \u003cp\u003eFigures 1a and 1b provide the point cluster map and the dendrogram using \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;40 sorters all using Suits as the attribute for classification. Both approaches, CM and HCA, successfully revealed 5 clusters. However, in the second experiment using Ranks as attribute, the Content Mapping based on 2 MDS dimensions failed to identify the 11 cluster structure (see Fig.\u0026nbsp;2a; note that the configuration suggests a 6 cluster solution rather than the 10 cluster solution forced on the data), which can clearly be discerned in the dendrogram produced by hierarchical cluster analysis using Ward\u0026rsquo;s method directly on the distance matrix (Fig.\u0026nbsp;2b).\u003c/p\u003e \u003cp\u003eFigures 3a and 3b depict the results for a simulation where half of the sample (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;20) used Suit, and half of the sample (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;20) used Ranks for sorting. Inspection of the point cluster map and the dendrogram supports the interpretation that cards are ranked by suit, and that number cards receive different treatment than picture cards. At no point would one conclude based on the point cluster map, that Rank played a role in sorting.\u003c/p\u003e \u003cp\u003eIn order to eliminate the possibility that this is just a small sample problem (small n relative to the number of statements), we increased the sample size by a factor 10 to 400 participants and replicated the last experiment (figures not printed to save space). Again, either clustering technique failed to identify Ranks as a used attribute. Further trials using variations of the proportion of participants using either attribute, revealed that when a clear majority of 70% or more used Rank the cluster solutions reveal Rank as sorting criterion, but this solution then failed to identify Suit as a used attribute.\u003c/p\u003e \u003cp\u003eThe fifth and final experiment aimed to find out if the number of piles relative to the number of cards may play a role in the outcome of CM. In this experiment we combined 20 respondents sorting on the basis of Suits (5 piles) with 20 respondents sorting on the basis of Odd-Even (3 piles). The results are in Figs.\u0026nbsp;4a and 4b. Focusing on the dendrogram, we find clusters of e.g., even and uneven hearts, even and uneven spades, picture diamonds, etc., and we would conclude that (un)evenness by suit would be the hybrid attribute for sorting. The dendrogram does not reflect two different groups sorting on Suits or on odd-Even exclusively.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Discussion\u003c/h2\u003e \u003cp\u003eThe two different visualizations of the five classification problems suggest that dendrograms may be more useful to interpret the clusters than point cluster maps (particularly when the number of clusters is large and the names of the statements are complex). Skipping MDS did not cost: dendrograms are easier to interpret than point cluster maps and one does not have to make the assumption that the dimension Q can be reduced to 2. However, neither method is capable to reveal mixtures of attributes. Kane and Trochim consider so-called bridging values important in determining the contents of clusters, which in the first experiment could potentially lead to choosing 4 clusters instead of 5 because the Wildcards are close to another cluster and can be conceived as forming one single cluster. Using bridging values will not help in unmixing mixed sets of used attributes. In these situations, the methods allow either for identification of the attribute with the lowest number of piles, or of some (non-existing) hybrid of attributes, but is unable to separate the two different attributes for sorting. Note that in the highly simplified situation of our experiments, there was no missing data: all participants sorted all statements. There is far more complexity in reality than that: people from the same discipline may actually use different attributes to sort into clusters, and individuals may change or mix attributes during the sorting task (e.g., from red-black to suits) or change the number of values (and thus the number of clusters) they are sorting on (e.g., first sorting numbers 1, 2, ... 10, then sorting odd/even numbers and face cards).\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Conclusions and outlook","content":"\u003cp\u003eWe have created a highly simplified situation where the same statements were sorted on 2 different attributes and both CM and HCA failed to reveal these attributes. The results do not support the claims on the suitability of CM for interdisciplinary research. In applications of CM, the attributes for sorting statements are unknown and the analysis aims to identify the attributes. In the face of failure of clustering algorithms to identify these attributes when they are diverse (as should be expected in the case in interdisciplinary research) a qualitative analysis of the statements may be more fruitful. For example, generated statements may be subjected to content analysis, or insight in the possible attributes that were used for sorting of the statements can be acquired by a thematic analysis of the obtained piles. Also, focus groups discussing the meaning of each statement or the relevance of prior identified possibly relevant attributes may reveal dominant attributes within particular groups as well as differences of attribute selection between groups.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eJK and HT wrote the main manuscript. JH and JK wrote the script for R. All authors reviewed the manuscript.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData and analysis code is provided within supplementary information files.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBurke, J. G., O\u0026rsquo;Campo, P., Peak, G. L., Gielen, A. C., McDonnell, K. A., \u0026amp; Trochim, W. M. (2005). An introduction to concept mapping as a participatory public health research method. \u003cem\u003eQualitative health research, 15(10)\u003c/em\u003e, 1392-1410. DOI: 10.1177/1049732305278876.\u003c/li\u003e\n\u003cli\u003eDixon, J. K. (2009). Media Review: Kane, M., \u0026amp; Trochim, WMK (2007). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage. \u003cem\u003eJournal of Mixed Methods Research\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(1), 87-89. DOI: 10.1177/1558689808326121\u003c/li\u003e\n\u003cli\u003eEveritt, B. (1980). \u003cem\u003eCluster analysis. \u003c/em\u003e2\u003csup\u003end\u003c/sup\u003e Edition. New York: Halsted Press.\u003c/li\u003e\n\u003cli\u003eKane M \u0026amp; Trochim WM (2007). \u003cem\u003eConcept mapping for planning and evaluation.\u003c/em\u003e London: SAGE. DOI: 10.4135/9781412983730\u003c/li\u003e\n\u003cli\u003eLattin J, Carrol JD, Green PE (2003).\u003cem\u003e Analyzing multivariate data.\u003c/em\u003e Brooks/Cole Thomson Learning: Pacific Grove CA, USA. \u003c/li\u003e\n\u003cli\u003eLeyesdorff L. \u0026amp; Vaughan (2006) Co-occurrence matrices and their applications in information science: Extending ACA to the Web environment. \u003cem\u003eJournal of the American Society for Information Science and Technology 57\u003c/em\u003e, 1616-1628. DOI: 10.1002/asi.20335\u003c/li\u003e\n\u003cli\u003eP\u0026eacute;ladeau N, Dagenais C, Ridde V (2017). Concept mapping internal validity: A case of misconceived mapping? \u003cem\u003eEvaluation and Program Planning, 62,\u003c/em\u003e 56\u0026ndash;63. http://dx.doi.org/10.1016/j.evalprogplan.2017.02.005\u003c/li\u003e\n\u003cli\u003eRosas, S. R., \u0026amp; Kane, M. (2012). Quality and rigor of the concept mapping methodology: a pooled study analysis. \u003cem\u003eEvaluation and program planning, 35(2)\u003c/em\u003e, 236-245. http://dx.doi.org/10.1016/j.evalprogplan.2011.10.003\u003c/li\u003e\n\u003cli\u003eShannon, J., Borron, A., Kurtz, H., \u0026amp; Weaver, A. (2021). Re-envisioning emergency food systems using photovoice and concept mapping. \u003cem\u003eJournal of Mixed Methods Research\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(1), 114-137. https://doi.org/10.1177/1558689820933778\u003c/li\u003e\n\u003cli\u003eTrochim WMR (1989a). An introduction to concept mapping for planning and evaluation. \u003cem\u003eEvaluation and Program Planning, 12\u003c/em\u003e, 1-16. https://doi.org/10.1016/0149-7189%2889%2990016-5\u003c/li\u003e\n\u003cli\u003eTrochim WMR (1989b). Concept mapping: Soft science or hard art? \u003cem\u003eEvaluation and Program Planning, 12,\u003c/em\u003e 87-110. https://doi.org/10.1016/0149-7189(89)90027-X\u003c/li\u003e\n\u003cli\u003eWindsor, L. C. (2013). Using concept mapping in community-based participatory research: A mixed methods approach. \u003cem\u003eJournal of mixed methods research\u003c/em\u003e, \u003cem\u003e7\u003c/em\u003e(3), 274-293. http://dx.doi.org/10.1177/1558689813479175\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"concept mapping, multidimensional scaling, cluster analysis, inter-disciplinary research","lastPublishedDoi":"10.21203/rs.3.rs-4353956/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4353956/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cdiv language=\"En\" class=\"ArticleSubTitle\"\u003eConcept Mapping (CM) is promoted as a research method suitable for interdisciplinary and international research, \u0026ldquo;\u0026hellip; best suited to applications where diverse or wide-ranging opinions need to be gathered and made sense of.\" Our study does not support these claims. We created a situation where 54 statements were sorted on just 2 different attributes. Neither the conventional analytic approach in CM nor alternative clustering algorithms appear able to reveal these attributes. CM may be appropriate to use when groups can be assumed to be homogeneous, but CM cannot test this assumption. Discovering which attributes are used by sorters requires thematic analysis or focus groups.\u003c/div\u003e","manuscriptTitle":"The Validity of Content Mapping: Let’s Call a Spade a Spade","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-09 04:43:45","doi":"10.21203/rs.3.rs-4353956/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e2e7d516-d50e-4365-b83f-f6ef5eb3a38e","owner":[],"postedDate":"May 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-05-12T03:08:26+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-09 04:43:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4353956","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4353956","identity":"rs-4353956","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}