Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals

Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals Mohammed Abdulaziz Aljallal, Qasim Chaudhry, Nicholas R. Price This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3969238/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 07 Aug, 2024 Read the published version in Scientific Reports → Version 1 posted 11 You are reading this latest preprint version Abstract Growing restrictions and bans on animal testing for chemical safety assessment under different regulations have led to an increasing use of alternative methods. Read-across is one of the major approaches used for this purpose, which relies on the identification of toxicological hazards of a data-poor or untested (target) chemical from data on other already-tested (source) similar chemicals. This requires the target substance to be first assigned to a group or category of ‘similar’ chemicals. The ‘similarity’ may be in terms of structural features alone, or in combination with certain rules that are based on mechanistic and/or toxicological aspects. In this regard, the OECD QSAR Toolbox - a major free-access in silico platform - is widely used to derive toxicity predictions for a range of (eco) toxicological endpoints. The Toolbox allows the user to identify a set of similar chemicals (analogues) by computational ‘profilers’ that incorporate different structural alerts, or a combination of structural alerts and physicochemical and/or toxicokinetic rules relevant to a specific toxicological endpoint. The overall aim of this study was to assess the performance of the in silico profilers provided in the OECD QSAR Toolbox for reliability for identifying chemical analogues for category formation in a number of high-quality databases on mutagenicity, carcinogenicity, and skin sensitisation. The study also aimed to identify the reasons for any limitations in the performance of the profilers, and propose ways to improve their overall accuracy. The results showed that whilst some structural alerts are fit-for-purpose as such within the acceptable limits, others need refinement or a consideration for their possible exclusion from the profiler. Such refinements are imperative for a reliable use of the profilers in read-across and grouping/categorisation for classification, labelling and risk assessment of chemicals. Biological sciences/Computational biology and bioinformatics Biological sciences/Computational biology and bioinformatics/Computational models Figures Figure 1 Figure 2 Figure 3 Introduction Chemical grouping based on category formation allows risk assessment of a data-poor substance through inference from the data on other members of the same category. The concept behind the approach is based on the notion that similar substances generally have similar properties. At the physicochemical level, the factors considered for grouping can be a similarity in chemical structures, functional group(s); metabolic/ degradation profiles, or additional parameters, such as log P, protein binding, etc. Where sufficient 'similarity' criteria are met in a set of chemical substances that follow a regular pattern, it can be considered a 'category'. According to the OECD grouping guidance (OECD, 2014): ‘A chemical category is a group of chemicals whose physicochemical and human health and/or ecotoxicological properties and/or environmental fate properties are likely to be similar or follow a regular pattern as a result of structural similarity. The similarities may be based on the following: a common functional grouping common constituents or chemical classes an incremental and constant changes across the category The likelihood of common precursors and/or breakdown products, via physical or biological processes, which result in structurally similar chemicals.’ The European Chemicals Regulation “REACH” encourages the use of grouping and categorisation of chemicals for classification, risk assessment and labelling purposes. Some of the data gaps in this regard can be filled using read-across within a given category, which is based on interpolation of experimental data from the tests conducted on 'similar' (source) substances to the untested (target) substance. The data for the endpoint in question for the target substance are predicted using the experimental data for the same endpoint of the source substance(s). Since each endpoint has a different set of complexities, e.g. in regard to mode of action against a biological target site, a read-across needs to be considered on an endpoint-by-endpoint basis (EChA, 2017). There has been an increasing emphasis on applying the 3Rs principles to refine, reduce and replace the use of animals in chemical safety testing under most regulatory frameworks in Europe. A complete ban on animal testing has been implemented under the EU Cosmetic Regulation since March 2013. Such developments have made the use of read-across in risk assessments ever more important, with emphasis on the outcomes to be transparent and reliable. In this regard, the definition of 'similarity' between the target and source substances has also evolved to include similarity not only in the structural features, but also in other physicochemical, mechanistic and/or metabolic aspects (Schultz and Cronin, 2017 ). For read-across to be valid, a robust category of analogues must be derived from high quality datasets. A valuable tool for achieving this is the OECD QSAR Toolbox, which is a freely available multifunctional in silico platform that allows the users to make informed decisions about toxicity predictions for a range of (eco)toxicological endpoints (Schultz et al, 2018 ). As part of the process of identifying a set of analogues for read-across, the Toolbox allows the user to apply structural alerts in the form of computational “profilers”. One or more profilers are available for many of the toxicological endpoints that are based around chemistry; e.g. covalent binding to proteins or DNA, and/or other mechanistic or toxicological aspects. These profilers incorporate structural alerts, and in some cases a combination of structural alerts and physicochemical rules, that are relevant to a specific toxicological endpoint. The target compound is first subjected to profiling, and then the profile is used to screen for compounds in the databases with the same or similar structural, mechanistic, and/or toxicological profiles. The analogues found this way are reduced to those that have measured values for the specific endpoint(s) of interest and therefore provide a basis for prediction of the endpoint value of the target compound. The OECD QSAR Toolbox (referred to herein as the “Toolbox”) incorporates databases on chemical (e.g. properties), toxicological and ecotoxicological data, as well as estimated values from in silico QSAR models. The system also incorporates QSAR models built within an informatics chassis, that are designed for in silico prediction of toxicological hazard. The Toolbox therefore allows the user to perform a number of functions (OECD 2008 ): Identification of analogues for a chemical, retrieval if experimental results available for those analogues and data gap filling by read-across or trend analysis; Categorisation of large inventories of chemicals according to mechanisms or modes of action; Filling of data gaps for a chemical by using appropriate model(s) from the collection of QSAR models; Evaluation of the robustness of a potential analogue for read-across; Evaluation of the appropriateness of a (Q)SAR model for filling a data gap for a particular target chemical; and The capability of building QSAR models. Over the years, the Toolbox has become a widely used platform for chemical grouping/categorisation and estimation of chemical toxicity by in silico methods and read-across. Despite the increasing reliance of risk assessors on the Toolbox, attempts have only recently started to assess the reliability and limitations of the profilers provided in the system (Devillers et al., 2011 ; Mombelli, 2012 ; Yordanova et al., 2019 ). As part of the current study, it was felt that an in-depth assessment of the profilers was necessary to determine the usefulness and limitations for their use in screening databases for analogous compounds for further use in read-across or development of (Q)SAR models. This study, therefore, investigated the performance of the OECD QSAR Toolbox profilers for mutagenicity, carcinogenicity and skin sensitisation potential of chemical substances, to assess how reliably they identified both positive and negative compounds contained within the databases. The analysis also aimed at investigating any factors that would limit the usefulness of a profiler, and to propose ways to improving their performance. Materials and Methods This study used version 3.1 of the Toolbox. As part of the appraisal of the available profilers, a number of databases integral to the Toolbox as well as a few additional databases from publicly available sources were examined. These are listed in Table 1 . Table 1 Toxicological databases used in this study (details given for each database are as retrieved at the time of this study in 2020) Database/ source Description Bacterial mutagenicity ISSSTY (donated to the Toolbox by the Istituto Superiore di Sanità (ISS), Rome, Italy) 41,634 Ames test data points for 7,367 compounds. The overall endpoint value (positive, negative, equivocal, inconclusive) outcome is determined as: - Positive: at least one strain is positive (with or without Metabolic activation); - Equivocal: no strain is positive, and at least one equivocal result is present in one of the following strains (with or without Metabolic activation): TA1535, TA100, TA98, TA1538, TA1535, TA97; - Negative: no positive or equivocal results are present in any strain, and negative outcomes exist for: a) at least one strain from among TA1535 or TA100 or TA97 (with and without Metabolic activation); and b) at least one strain from among TA1538 or TA98 or TA1537 (with and without Metabolic activation); - Inconclusive: If none of the above criteria are fulfilled. When more than one experiment in one strain was available, the number of reported positive and negative studies was counted, and the strain overall outcome was determined as follows: if the percentage of Positive studies is lower than 40%, then outcome = Negative; if the percentage of Positive studies is between 40 − 60%, then outcome = Equivocal; if the percentage of Positive studies is higher than 60%, then outcome = Positive (Aljallal,2020). Carcinogenicity and mutagenicity (ISSCAN) (donated to the Toolbox by the Istituto Superiore di Sanità (ISS), Rome, Italy) 6,979 data points for 1,150 compounds. There are three endpoints for which data are presented; gene mutation, summary carcinogenicity, and TD50. The TD50 data are not used in this current study. Gene mutation data in Ames reported in the same way as in the previous database, and a single datum point is available for each of the 832 compounds. Summary carcinogenicity data are represented as “positive”, “negative” or “equivocal”. Positives are carcinogenic in at least one experimental group; equivocal results are given to chemicals with equivocal results in at least one experimental group, together with negative results in the other experimental groups, and negatives are non-carcinogenic in all tests (Aljallal,2020). Genotoxicity OASIS (donated to the Toolbox by the Laboratory of Mathematical Chemistry, Bourgas, Bulgaria) 7,500 compounds collected from seven sources. It contains data for mutagenic determined by the Ames test with and without metabolic activation. The database also includes chromosomal aberrations determined by in vitro tests using Chinese hamster lung cells (CHL, with and without S9). Micronucleus (MN) and mouse lymphoma gene mutation assay (MLA) are evaluated by Chinese hamster lung cells (CHL / IU) and by in vitro T-lymphoma cell lines, respectively. All endpoints were evaluated on a dichotomic scale (YES/NO). Data used in this study were the Ames test data, and chromosomal aberration as the Toolbox has profilers for these endpoints (Aljallal,2020). CRD-AGES (Data from the UK Chemicals Regulatory Directorate and the Austrian Agency for Health and Food Safety) 179 pesticides with mutagenicity data, and 100 with carcinogenicity data (Worth et al 2010 ). Mutagenicity data are binary active/inactive from Ames tests and carcinogenicity also binary active/inactive from a range of tests DSS Pesticide carcinogenicity (the US EPA National Center for Computational Toxicology, www.epa.gov/ncct/dsstox/index.html ) Summary carcinogenicity data for 1,282 pesticide compounds SAR carcinogenicity and genotoxicity databases These are high-quality data databases used by Leadscope® program to build in silico predictive models. Both databases contain summarised in vitro and in vivo cancer and mutagenicity endpoints along with chemical structures. To ensure the high quality data for SAR, structure form, analysis of various salt forms of chemical compounds and their respective toxicity data have also been carried out to derive an overall endpoint for the active portion of the chemical. Several sources of experimental test results have been included in these databases, such as from FDA, NTP, CCRIS, CPDB and other primary sources. All chemical structures have been provided in SAR, neutral and tested form, and confirmed for accuracy. The SAR carcinogenicity database includes 3,598 compounds with 11,538 test results and provides carcinogenicity study endpoint for male and female rat (1,774, 1,725 compounds respectively) and male and female mouse (1,640, 1,675 compounds respectively). The SAR genotoxicity database provides compound-level calls for 46 genetic toxicity endpoints for 10,534 compounds. These include 32 bacterial mutagenicity endpoints, four in vitro mammalian, five in vitro chromosomal aberration and six in vivo micronucleus results (Aljallal,2020). EFSA Pesticides Mutagenicity Ames test active/inactive classification for 741 pesticides, compiled by, and downloaded from the European Food Safety Authority. NISS mutagenicity database A database of 1,863 compounds with binary active/inactive Ames test data. Downloaded from the US National Institute of Statistical Sciences, ( www.niss.org/ ) Inchemicotox Skin Sensitisation A version of the Cronin and Basketter dataset (Cronin and Basketter 1994) from the Inchemicotox project ( www.inchemicotox.org/ ), comprising 322 compounds, with results taken from guinea pig maximisation tests. The classification is derived from the percentage of animals sensitised in the test: non-sensitiser = 0–9%, weak sensitiser = 10–29%, moderate sensitiser = 30–79%, strong sensitiser = 80–100%. CAESAR Skin Sensitisation 209 compounds from the EU CAESAR project ( www.caesar-project.eu ). For developing classification models, this data set was subdivided in two classes, sensitiser (S) and non-sensitisers (N), which gave a good distribution of the numbers of compounds in each class. The class S merges the first four ranges established by ECETOC: Extreme (EC3 < 0.1%), Strong (0.1%<EC3 < 1%), and Moderate (1%<EC3 10%) ranges; the class N regroups all compounds belonging to the non sensitisers ECETOX Skin sensitisation 39 compounds with experimental results on skin and respiratory sensitisation. The compounds were selected as known sensitisers and non-sensitisers for the assessment of novel test techniques. ( www.ecetoc.org/technical-reports ) OECD Skin Sensitisation 1,036 compounds from 2 databases and includes chemicals tested by Local Lymph Node Assay,(LLNA) or Guinea Pig Maximization Test, (GPMT). Based on the observed skin sensitisation effect the chemicals are classified in three classes: - strong sensitisers, weak sensitisers or non-sensitisers Lazar Opentox Rat Carcinogenicity Lazy Structure-Activity Relationships (LAZAR) is an open-source tool for the prediction of complex toxicological endpoints such as carcinogenicity (female/male, hamster/ mouse/rat/rodent) and Salmonella mutagenicity. The compounds were selected from database of experimental toxicity data. Carcinogenicity models are based on CPDB, while the Salmonella mutagenicity model uses a dataset of 3895 compounds determined in vitro. ( https://lazar.in-silico.de/predict ) VEGA carcinogenicity The VEGA platform serves to access a number of QSAR models for predicting mutagenicity and carcinogenicity. The compounds were selected from a set of 4225 molecules tested with the Ames bacterial test. And for carcinogenicity were selected from set of 805 chemicals from the Carcinogenic Potency Database (CPDB) The Carcinogenic Potency Database (CPDB) Data relating to cancer causing chemicals were compiled from the Carcinogenic Potency Database (CPDB), which is freely available at http://toxnet.nlm.nih.gov/cpdb/cpdb.html . This database is a widely used and unique international resource comprising the results of 6,540 chronic, long-term animal carcinogenicity tests on 1,547 chemicals in rats, mice, dogs, hamsters and non-human primates (Aljallal,2020). The profilers assessed in this study are listed in Table 2 . These were used to assess the compounds that had experimental values against specific endpoints from the databases listed in Table 1 . Table 2 The OECD QSAR Toolbox profilers assessed in this study Profiler/ toxicological endpoint Description Mutagenicity/ genotoxicity DNA binding by OASIS v1.1 This profiler is a mechanistic profiler developed from an analysis of Ames mutagenicity data. It contains a number of structural alerts that have been shown to be related to established electrophilic reaction chemistry known to be important in covalent DNA binding (Mekenyan et al 2004 ; Serafimova et al. 2007 ) DNA binding by OECD. This profiler is based on structural alerts for the electrophilic reaction chemistry associated with covalent DNA binding (Enoch and Cronin 2010). The profiler returns a range of structural alerts that contain electrophilic centres or those that can be metabolically activated to electrophiles. Carcinogenicity (genotox and nongenotox) alerts by ISS. This profiler is based on a list of 55 structural alerts from the software Toxtree ( http://toxtree.sourceforge.net/ ). About 20 of the alerts are for non-genotoxic carcinogenicity, and the remainder for genotoxic carcinogenicity (mutagenicity) DNA alerts for AMES, MN and CA by OASIS v.1.1 a refinement of above mutagenicity (Ames test) alerts by ISS. The present list of SAs is a subset of the original Toxtree list, obtained by eliminating the SAs for nongenotoxic carcinogenicity and is a refinement of above In vivo mutagenicity (Micronucleus) alerts by ISS . This profiler is based on the ToxMic rule-base of the software Toxtree. This rule-base provides a list of 35 structural alerts (SAs) for a preliminary screening of potentially in vivo mutagens. These SAs are molecular functional groups or substructures that are known to be linked to the induction of effects in the in vivo micronucleus assay Carcinogenicity DNA binding by OASIS v1.1 As above DNA binding by OECD As above Carcinogenicity (genotox and nongenotox) alerts by ISS The SAs for carcinogenicity are molecular functional groups or substructures known to be linked to the carcinogenic activity of chemicals. As one or more SAs embedded in a molecular structure are recognised, the system flags the potential carcinogenicity of the chemical (Aljallal,2020). OncoLogic Primary Classifier This profiler consists of molecular definitions derived by the Toolbox developers to mimic the structural criteria of chemical classes of potential carcinogens covered by the U.S. Environmental Protection Agency’s OncoLogic™ Cancer Expert System for Predicting the Carcinogenicity Potential ( www.epa.gov/oppt/sf/pubs/oncologic.htm ). Skin Sensitisation Protein binding by OASIS These profilers have been developed to indicative of skin sensitisation potential and consist of 85 structural alerts relating to 11 reactions, or chemical interactions, which are known to be associated with skin sensitisers Protein Binding by OECD The protein binding by OECD profiler contains 16 mechanistic alerts covering 52 structural alerts. These data are supported by mechanistic chemistry and references to the scientific literature (the meta data). They represent a parallel approach to those of the OASIS profiler and capture mechanistic features of target compounds Protein binding potency This profiler is developed on the basis of empirical data for thiol reactivity expressed by the in chemico RC50 value. All the chemicals have two common electrophilic mechanisms of interaction with GSH – interaction via SN2 and interaction via Michael addition (MA) mechanism. The profiler contains 49 MA and 46 SN2 categories Keratinocyte gene expression This profile is built in relation to the database derived from the KeratinoSens assay, which examines the potential for chemicals to induce the expression of a luciferase reporter gene under control of a single copy of the ARE element of the human AKR1C2 gene stably inserted into immortalised human keratinocytes. Relevance to skin sensitisation is inferred from the relationship of Keap1-Nrf2-ARE regulatory pathway and its detection of electrophilic chemicals to sensitisation. The profiler contains 22 categories Protein binding alerts for skin sensitisation by OASIS This profiler seems to be much the same as the one at 2.2.3.1 above though there are some minor differences DPRA Lysine peptide depletion This profile is built on the basis of data derived from Direct Peptide Reactivity Assay (DPRA). The DPRA is a reactivity assay which evaluates the ability of chemicals to react with proteins. Model synthetic peptides containing either lysine or cysteine are used. The remaining concentration of cysteine- or lysine-containing peptide is measure after 24 hours incubation with the test chemical at 25 ± 2.5ºC. The peptide reactivity is reported as percent peptide depletion. The relevance with skin sensitisation is the presence of cysteine and lysine residuals in the skin proteins. The profiler contains 24 structural alerts extracted from about 110 chemicals with experimentally measured lysine depletion values DPRA Cysteine peptide depletion As described above, this profiler contains 32 categories of alerts Further information is provided in the supplementary file on the total number of substances identified with experimental data against each endpoint in different databases, and that used in the assessment of the profilers performance. Data analysis The compounds identified in the databases for the specific endpoint were profiled using appropriate profilers. For each of the compounds, if an alert was triggered, the compound was allocated a score of 1, if no alerts were triggered, a score of 0 was allocated. The results were compared with the assigned binary activities from the original database (positive = 1; negative = 0). Cooper statistics (Cooper et al , 1979) were used to assess the results against the experimental values given in the databases, by calculating the sensitivity, specificity and accuracy of the alert triggers as follows: Sensitivity (True positive rate) = TP/ TP + FN Specificity (True negative rate) = TN/TN + FP Accuracy = (TN + TP)/(TN + FP + FN + TP) PPV (Positive predictive value) or (precision) = TP/ TP + FP MCC = (TPxTN)-(FPxFN)/√(TP + FN)(TP + FP)(TN + FN)(TN + FP) Where TP = True positive, TN = True negative, FP = False positive, FN = False negative Sensitivity is defined as the percentage of correctly classified positive predictions among the total number of positive instances. Specificity is the percentage of correct negative predictions compared to the total number of negatives. Accuracy (concordance or “Q”) is defined as the total number both positive and negatives correctly predicted among the total number of compounds. PPV (positive predictive value) is defined as the total number of correctly classified positive predictions among the total number of both negative and positive instance. MCC (Matthews correlation coefficient) is a weighted value that overcomes any imbalance in the data classes which might lead to over optimistic values of Q (Matthews, 1975). An MCC value of 1 indicates that the model can predict the data classes of unknown compounds perfectly, whilst a MCC value of 0 indicates that the predictions are no better than random guessing, and a MCC value of -1 indicates total disagreement between the predicted data and the actual data. Results The results of the assessment of the profilers against the experimental data for mutagenicity, carcinogenicity and skin sensitisation are shown in Tables 3 , 4 and 5 respectively. Further detailed analysis was undertaken to identify over-predicting structural alerts in the carcinogenicity profilers i.e. with the Precision or PPV (positive predictive value) lower than 0.5. This analysis was performed to determine structural alerts with little information or predictive capability to increase the sensitivity and overall accuracy of the profiler. Detailed analysis of 13 non-genotoxic carcinogenicity structural alerts was conducted and results are presented in Table 6.1 and 6.2. An additional analysis for the Oncologic Primary Classification carcinogenicity profiler was performed for 30 structural alerts incorporated in the profiler. The purpose of this analysis was to assess which of the structural alert(s) had a precision PPV (positive predictive value) lower than 0.5. This analysis is presented in Table 6.3 . The cutoff value was set to be 0.5 to ensure that none of these profilers had a lower predictive power compared to bacterial Ames test, which has also been applied to predict carcinogenicity of genotoxic substances in rodents. The high predictive power of positive Ames, which ranges from 77–90% depending on various factors, makes it superior to any other in vitro genotoxicity assay, all of which have a relatively lower performance in terms of predicting genotoxicity (Kazius et al ., 2006). Mutagenicity profilers The results shown in Tables 3 indicated that the accuracy (percentage of positives and negatives correctly predicted) of the mutagenicity profilers varies across the datasets from 51–76%. Clearly, whilst 76% can be accepted because it is in line with the level of error generally seen in the experimental data in most databases, 51% is barely better than chance. The micronucleus alerts appear to be general and, as such, significantly over-predict mutagenicity. The most common alert triggered by this profiler is “Hacceptor-path3-Hacceptor”. This alert indicates the non-covalent binding of the target chemical to DNA via two bonded atoms connecting two H bond acceptors (Snyder et al . 2006). However, it appears that such a functional grouping is common in both mutagens and non-mutagens. It is likely that the performance of this profiler would improve if this specific alert was omitted. As expected, both DNA binding profilers work best with the data obtained from Ames type tests (due to the availability of large databases for this test), but do not perform well for chromosome aberration or micronucleus data. The genotoxicity and non-genotoxicity alerts (ISS) have acceptable true positive results but fail to distinguish the negatives. Overall, these alerts perform best with Ames type data. The OASIS DNA alerts for Ames, micronucleus and chromosomal aberration predict the results in Ames datasets fairly well but for both micronucleus data and chromosomal aberration data, these profilers underpredict positive compounds with sensitivity rates ranging from 36 to 44%. The ISS Ames test alerts have accuracies over 70% for Ames datasets, which together with MCC values greater than 0.5, indicate that the performance is independent of skewed sample categories. It, however, needs to be noted that the micronucleus/ CA alerts may not be suitable predictors of Ames. They need to be considered separately and used, along with Ames, to develop the overall weight of evidence. The receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as the y coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the quality or performance of diagnostic tests which in this case is an in silico profiler. As it is shown in Fig. 1 , the ROC curve analysis showed that Ames ISS profiler achieved the highest balanced accuracy for both true positive rate and low false positive rate values. Carcinogenicity profilers Both DNA binding profilers performed equally poor with carcinogens and non carcinogens from all datasets, with accuracy values rarely above 60% and, MCC values indicating a performance barely better than chance. The ISS carcinogenicity alerts fared a little better in predicting carcinogens, but showed a poor segregation of non-carcinogens reduced the overall effectiveness of this profiler with accuracy levels between 57% and 68% for the sample datasets, and modest to poor performance on skewed datasets as indicated by the MCC values of 0.17 to 0.36. The ROC analysis, shown in Fig. 2 , indicate that ISS carcinogenicity profiler was the highest quality performance profiler compared to the other 3 profilers in terms of both true positive rate and false positive rate. 13 Non-genotoxic carcinogenicity (NGC) structural alerts among ISS carcinogenicity profiler were analysed individually to test their performance by PPV (positive predictive value). As shown in Table 6.1 , the overall PPV of ISS non-genotoxic carcinogen structural alerts was 57%, where 326 substances are truly predicted as nongenotoxic carcinogen out of total 570 substances that had been identified to contain one of the 13 NGC structural alerts. The precision value (PPV) for non-genotoxic carcinogenicity structural alert ranged from 0.92 for Trichloro (or fluoro) ethylene and Tetrachloro (or fluoro) ethylene as a highest PPV to 0.39 for Quercetin type flavonoids. Four out of 13 NGC structural alerts show over-prediction. These are thiocarbonyl, substituted n-alkylcarboxylic acids, quercetin type flavonoids, and phtalate (or butyl) diesters and monoesters. All of these four structural alerts predict non carcinogenic substances as carcinogens in more than 50% of the total substances that contain this structural alert, which lowers the total accuracy of the ISS carcinogenicity profiler. Thiocarbonyl NGC structural alert was flagged in 106 substances as the only structural alert. Any substance that contained more than one NGC structural alert was not counted in this analysis to avoid any interference. Sixty two non-carcinogenic substances were falsely predicted as carcinogenic substances by thiocarbonyl structural alerts with sensitivity rate of 0.42. Due to this over-prediction of thiocarbonyl alert, it can be suggested that taking this alert out could increase sensitivity of the total NGC structural alerts and the ISS carcinogenicity profiler. This would, however, not be the ideal solution, as any thiocarbonyl NGCs would then be completely out of the scope of the profiler. Instead, it is proposed that further research be carried out to see whether performance of this alert can be improved using a larger database of thiocarbonyl substances. Likewise, the other three NGC structural alerts; i.e. substituted n-alkylcarboxylic acids, quercetin type flavonoids and phthalate (or butyl) diesters and monoesters also showed a precision value lower than 0.5 with a 0.42, 0.39 and 0.38 respectively. Again ignoring these four structural alerts increased the total precision value of NGC structural alerts and consequently of the performance of the ISS carcinogenicity profiler. The results (Table 6.2 ) showed that the precision value of ISS non-genotoxic structural alert was improved from 0.57 to 0.64 by ignoring the 4 structural alerts. However, for the reasons mentioned before, it is proposed that further research should be carried out to improve the performance of these alerts within the profiler. The Oncologic primary classification alert over-predicted carcinogens with sensitivity rates of 66–73% at the expense of poor prediction of non-carcinogens (30–46%), resulting in overall performance, which is barely better than chance for most of the datasets. All 30 structural alerts in Oncologic primary classification profiler were individually analysed for their precision as shown in Table 6.3 . The four structural alerts showed over-prediction of non-carcinogenic substances as carcinogenic in more than 50% of the total substances containing this structural alert. These four alerts were carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone reactive functional groups. Carbamate type compounds structural alert for carcinogenicity was triggered in 63 substances with only 22 true positive carcinogenic substances. The other 43 substances that were flagged by this alert to be carcinogenic were non-carcinogenic in real experimental tests. This gave a low precision value for carbamate type compounds of 0.35. The second structural alert in oncologic primary classification profiler with low precision value (lower than 0.36) was organophosphorus type compounds structural alert, where 80 substances out of 126 flagged by this alert were wrongly predicted as carcinogenic substances. Peroxide type compounds structural alert showed only 0.27 precision (positive predictive value) rate with only 3 correctly predicted carcinogenic substances out of 11 substances flagged by the alert. The lowest structural alert in precision within oncologic primary classification profiler was for reactive ketone functional groups with only 0.1 where there was an over-prediction for 19 out of total 21 flagged substances. This mean that only 2 substances that were flagged by this alert were correctly predicted as carcinogenic substances out of the total 21 substances. It can therefore be suggested that all four of these structural alerts could be ignored from the oncologic primary classification profiler to increase the total sensitivity and accuracy of the profiler. Indeed, as shown in Table 6.3 , the total precision of the profiler was improved by 0.023 which is nearly 3% improvement in the overall performance of the profiler. This would, however, not be the ideal solution, as this will miss carcinogenic compounds and would be completely out of the scope of the profiler. Therefore, it is proposed that further research be carried out to see whether performance of these four alerts can be improved using a larger database of thiocarbonyl substances. Skin sensitisation profilers The performance of the protein binding profilers was not found to be consistent across the sampled datasets. For the CAESAR dataset, these profilers tend to have a low predictivity for non-sensitisers, whilst for the other datasets it is the sensitisers that are not well predicted. Overall, the performance of these predictors is moderate to poor for all the datasets. The DPRA lysine peptide depletion profilers showed a similar pattern, with performance being uniformly not better than chance for the CAESAR dataset but highly under-predictive for sensitisers in the other datasets. The protein binding potency profiler is also uniformly poor across all datasets, failing to detect the majority of sensitisers, with sensitivity rates between 12% and 29%. A similar pattern is seen with the keratinocyte gene expression profiler, where sensitivity rates were 19–43%. The overall ROC analysis for all seven profiler shown in Fig. 3 indicated that protein binding OASIS has a relatively better performance compared to the other skin sensitization profilers in OECD Toolbox. The study also carried out further analysis to investigate the contribution of different alerts within a given profiler on its overall performance. For this purpose, non-genotoxic carcinogenicity profilers were investigated as an example. This was because, under most EU regulatory frameworks, tests for carcinogenicity are only required when there is either a positive in vitro mutagenicity/genotoxicity test, or there are indications of carcinogenic effects from long term in vivo studies. This means that, whilst the current risk assessments framework would identify genotoxic carcinogens, it is possible that the carcinogenic effects of a nongenotoxic mechanism will not be identified without in vivo tests that are increasingly restricted or banned under different regulatory frameworks. Discussion & Conclusions This study has assessed and evaluated the structural alerts and profilers provided in the OECD QSAR Toolbox for mutagenicity, carcinogenicity, and skin sensitisation. These profilers are provided for the purpose of constructing categories of mechanistic and identifying structural analogues of a target compound. Apart from the OECD QSAR Toolbox, profilers are also used by a number of other applications for indicating certain toxicity endpoints, such as Toxtree ( http://toxtree.sourceforge.net/ ), and Oncologic ( www.epa.gov/oppt/sf/pubs/oncologic.htm ). Although not intended for making toxicological predictions in themselves, these profilers provide a useful means for read-across from experimental data on analogous compounds to estimate a property or biological activity of an untested compound. The accuracy and reliability of a profiler in terms of predicting a target compound is, therefore, important for defining the structural and functional features so that it is placed in the correct category/group of analogous substances for the purpose of read-across. This is also important because, in practice, safety of chemical substances is often based on (Q)SAR Toolbox profilers alone in the dossiers submitted for regulatory risk assessment, where the absence of an alert alone is (wrongly) argued as an evidence for the absence of toxicity. In this regard, those profilers that are equally likely to select endpoint-positive and endpoint-negative compounds into a grouping of analogues to be used in a read across, will by definition give rise to equivocal predictions for the target compound. This makes it essential to know how accurately different profilers perform in terms of sensitivity, specificity and accuracy, and to investigate possibilities for their improvement. This also requires an understanding of the role and merits of each individual alert within a profiler so that only the most relevant and reliable ones can be relied upon as indicators of the particular endpoint. The results of this study have shown that many structural alerts within each profiler have unacceptably low predictivity, which has a bearing on the performance of the relevant profiler. As an example, this study has found that the alert “Hacceptor-path3-Hacceptor” in the micronucleus profiler is too ubiquitous to be a useful indicator of mutagenicity. Similarly, in three structural alerts for non-genotoxic carcinogenicity within the ISS carcinogenicity profiler (thiocarbonyl, sub alkyl carboxylic acid, quercetin and phthalate), and the four structural alerts in oncologic primary classification profiler (carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone functional groups) were identified to have extremely low precision values. As shown in the examples investigated in this study, the omission or substitution of those alerts that unduly draw predictions towards equivocal outcomes within a profiler, can improve the overall performance of the profilers. This, however, would bring the drawback of making some of the toxicity alerts out of the scope of the profiler, and therefore further research is needed to find out whether they can be substituted with other more efficient alerts, or their performance can be enhanced by other means, e.g. by exploring larger datasets of relevant substances. Another factor influencing the segregation of compounds by the alerts could be the way in which the categorical data found in the 3 endpoint datasets studied here is derived. Very often the binary categorisation of data is achieved by manipulation of the continuous data in some way to provide “cut off” points for positive or negative assignment. The way in which this is done may affect the flagging of an alert in an essentially “negative” compound, or vice-versa. By definition, most alerts have been derived from datasets that are rich in endpoint-positive compounds, because deriving “negative alerts”, like proving a negative hypothesis, is generally not feasible, and negative results are not published so often. This study, therefore, has also given an insight into those alerts that may be found equally in endpoint-positive or negative compounds, and those which may be more effectively utilised to form groups of analogues for read-across predictions. Further research in this area is necessary to study the suitability and merits of each of the alerts within the profilers, both within the OECD Toolbox and other in silico toxicity platforms, to identify the root causes of the inadequacies and to investigate the possibilities for improvement in the profilers’ performance. This will, by implication, also improve the reliability of chemical read-across and grouping/categorisation for use in classification, labelling and risk assessment. The main goal of the study was to inform the scientific community about any limitations of the Toolkit profiles, and the structural alerts contained within the profilers, that are otherwise useful in understanding the underlying mechanisms of toxicity. The in-depth analysis has indicated that many of the profilers have quite poor performance when tested against a large number of compounds with experimental data. This casts doubts over their usefulness in category formation for read-across for which they are commonly employed for subsequent use in regulatory risk assessments. There is therefore a need for caution when using the profilers for chemical category formation and in read-across, because reliability of the latter would be dependent on the accuracy and specificity of grouping/category formation in the first place. It has also highlighted that toxicity predictions should not be performed blindly by relying on structural alerts alone, and that other in silico tools, such as QSAR models, should be employed to develop a more reliable weight of evidence for that purpose. Another important point is that the influence of any part of a compound on its biological effect(s) is not constant and strongly depends on its structural environment. Thus, any alert, even when derived by mechanistic interpretation of statistically significant QSAR models does not have automatic predictive power. Alerts should therefore be viewed as structural hypotheses of chemical action only, and their true predictive power should be confirmed by QSAR predictions and, if possible, through experimental validation. In this context, this study is positioned as a pragmatic statement that could change the thinking of both regulators and researchers. Declarations Funding statement The authors did not receive support from any organization for the submitted work. No funding was received to assist with the preparation of this manuscript. No funding was received for conducting this study. No funds, grants, or other support was received. Author Contribution M.J., Q.S. and N.P designed the study,M.J filtered and did the calculation for the profilers. M.J wrote the manuscript with support from Q.S and N.P All authors discussed the results and contributed to the final manuscript. All authors provided critical feedback and helped shape the research, analysis and manuscript. References Aljallal, M (2020) Investigation of in Silico Modelling to Predict the Human Health Effects of Cosmetics Ingredients. Doctoral thesis, Liverpool John Moores University. Balls, M. (2019). Chapter 1.1 - The Introduction and Influence of the Concept of Humane Experimental Technique. In: M. Balls, R. Combes and A. Worth, ed., The History of Alternative Test Methods in Toxicology . pp.3-6. Benigni, R. (2008). The Benigni/Bossa rulebase for mutagenicity and carcinogenicity–a module of Toxtree . EUR, pp.23241:1–70. Cronin, M., Madden, J., Enoch, S. and Roberts, D. (2013). Chemical Toxicity Prediction: Category Formation and Read-Across . Royal society of chemistry (RSC), pp.1-29. Devillers, J., Mombelli, E. and Samserà, R. (2011). Structural alerts for estimating the carcinogenicity of pesticides and biocides. SAR and QSAR in Environmental Research , 22(1-2), pp.89-106. European chemical agency (Echa) (2017). Grouping of substances and read-across - ECHA . [online] Echa.europa.eu. Available at: https://echa.europa.eu/documents/10162/13628/read_across_introductory_note_en.pdf [Accessed 18 Sep. 2018]. Enoch, S.J. et al. (2011) ‘A review of the electrophilic reaction chemistry involved in covalent protein binding relevant to toxicity’, Critical Reviews in Toxicology, 41(9), pp. 783–802. doi:10.3109/10408444.2011.598141. Mekenyan, O., Dimitrov, S., Serafimova, R., Thompson, E., Kotov, S., Dimitrova, N. and Walker, J. (2004). Identification of the Structural Requirements for Mutagenicity by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals I: TA100 Model. Chemical Research in Toxicology , 17(6), pp.753-766. Mombelli, E. (2012). Evaluation of the OECD (Q)SAR Application Toolbox for the profiling of estrogen receptor binding affinities. SAR and QSAR in Environmental Research , 23(1-2), pp.37-57. OECD (2008) (Q)SAR Application Toolbox version 1.1: Getting Started , Organisation for Economic Cooperation and Development, Paris, France. Schultz, T. and Cronin, M. (2017). Lessons learned from read-across case studies for repeated-dose toxicity. Regulatory Toxicology and Pharmacology , 88, pp.185-191. Schultz, T., Diderich, R., Kuseva, C. and Mekenyan, O. (2018). The OECD QSAR Toolbox Starts Its Second Decade. In: O. Nicolotti, ed., Computational Toxicology . New York, NY: Humana Press, pp.55-77. Serafimova, R., Todorov, M., Pavlov, T., Kotov, S., Jacob, E., Aptula, A. and Mekenyan, O. (2007). Identification of the Structural Requirements for Mutagencitiy, by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals. II. General Ames Mutagenicity Model. Chemical Research in Toxicology , 20(4), pp.662-676. Worth A., Lapenna S., Lo Piparo E., Mostrag-Szlichty A., Serafimova R. (2010) The Applicability of Software Tools for Genotoxicity and Carcinogenicity Prediction: Case Studies relevant to the Assessment of Pesticides JRC Scientific and Technical Report. 2010 EUR 24640. Yordanova, D., Schultz, T., Kuseva, C., Ivanova, H., Pavlov, T., Chankov, G., Karakolev, Y., Gissi, A., Sobanski, T. and Mekenyan, O. (2019). Alert performance: A new functionality in the OECD QSAR Toolbox. Computational Toxicology , 10, pp.26-37. Tables Tables 3 to 6 are available in the Supplementary Files section. Additional Declarations No competing interests reported. Supplementary Files SupplementarydataAljallaletal.docx Supportingdatacarcinogenicity.xlsx Supportingdatamutagenicity.xlsx Supportingdataskinsensitisation.xlsx Tables36.docx Cite Share Download PDF Status: Published Journal Publication published 07 Aug, 2024 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 12 Jun, 2024 Reviews received at journal 07 Jun, 2024 Reviewers agreed at journal 29 May, 2024 Reviews received at journal 02 May, 2024 Reviewers agreed at journal 30 Apr, 2024 Reviewers agreed at journal 25 Apr, 2024 Reviewers invited by journal 15 Mar, 2024 Editor assigned by journal 11 Mar, 2024 Editor invited by journal 11 Mar, 2024 Submission checks completed at journal 11 Mar, 2024 First submitted to journal 19 Feb, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3969238","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":278955019,"identity":"2017b9d7-b674-46c9-9f64-3e2135ae108f","order_by":0,"name":"Mohammed Abdulaziz Aljallal","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIie2PsWoCQRCG/+Pgrhm49kS5NwgMLByKkGcZWbBKkVJIonY2wtW+zcKCNgHbtTuxTXHlFSmyWFltrgxkv+4f5uOfASKRv4mgfcWkQNKWwxVh0Gib8mAFd4XNUOVpZ66d8DOps01cj4+KTX7tQkr9KboU1lQ7nc72OCk2pIJ1tRHxSuqV4lgSjovDlhBWzq3uhTekGpuNvu9KfuuDipOlb7HE0NmY8L5o/DDc4trlVPhEpf9lPmGjCn/kNHzYi3bd6q0qGptcvlbrKst3NxdSAJKHwBZIw/ue3Dym9a/7kUgk8v/4AcZsRd0fYQORAAAAAElFTkSuQmCC","orcid":"","institution":"Saudi Food and Drug Authority","correspondingAuthor":true,"prefix":"","firstName":"Mohammed","middleName":"Abdulaziz","lastName":"Aljallal","suffix":""},{"id":278955020,"identity":"6639cf9f-bce5-4900-a7c7-25c7fb18d064","order_by":1,"name":"Qasim Chaudhry","email":"","orcid":"","institution":"University of Chester","correspondingAuthor":false,"prefix":"","firstName":"Qasim","middleName":"","lastName":"Chaudhry","suffix":""},{"id":278955021,"identity":"69a361f0-0a36-4657-8629-d858bdfed877","order_by":2,"name":"Nicholas R. Price","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Nicholas","middleName":"R.","lastName":"Price","suffix":""}],"badges":[],"createdAt":"2024-02-19 07:14:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3969238/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3969238/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-024-69157-1","type":"published","date":"2024-08-07T15:58:17+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":52595479,"identity":"426399a4-a385-4905-b9fc-00de49490fe9","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":32034,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver operating characteristic (ROC) curve analysis for Mutagenicity profilers in OECD Toolbox\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/c72dae8fdbc0a3c2e394bcf7.png"},{"id":52595777,"identity":"2c505171-a12b-42a1-bd2c-79b39e099fef","added_by":"auto","created_at":"2024-03-13 11:52:01","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":59652,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver operating characteristic (ROC) curve analysis for carcinogenicity profilers in OECD Toolbox\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/c8550dad72ce608d93b74d3b.png"},{"id":52595480,"identity":"6a7dab71-887d-489f-b91e-c82138802cab","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":24747,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver operating characteristic (ROC) curve analysis for Skin sensitization profilers in OECD Toolbox\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/370c85651e7d18c30b343db8.png"},{"id":62298980,"identity":"c2fe877a-fc28-45f2-b5e1-7f5489899b57","added_by":"auto","created_at":"2024-08-12 16:18:09","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":591525,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/e7901d23-f56d-4620-b196-b6a94fd23683.pdf"},{"id":52595778,"identity":"954857b6-6f73-4ca0-9004-8a64351e6cc2","added_by":"auto","created_at":"2024-03-13 11:52:01","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":21495,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementarydataAljallaletal.docx","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/2405634b603eca56c1a9060c.docx"},{"id":52595483,"identity":"1b35574c-069f-4c7c-a604-e3d03c347afc","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":1208914,"visible":true,"origin":"","legend":"","description":"","filename":"Supportingdatacarcinogenicity.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/65ade5a723241de0ddba8136.xlsx"},{"id":52595486,"identity":"26cf541b-427e-4fb4-9e34-f412890a72fe","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"xlsx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":4417897,"visible":true,"origin":"","legend":"","description":"","filename":"Supportingdatamutagenicity.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/8aa03c2c0f973fb7406d4539.xlsx"},{"id":52595484,"identity":"28374137-21a8-4afe-a88a-a5c88d7e61f8","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"xlsx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":457106,"visible":true,"origin":"","legend":"","description":"","filename":"Supportingdataskinsensitisation.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/7770abddd7d3223f78791fd2.xlsx"},{"id":52595482,"identity":"eb8a0e28-efc1-4e78-a886-3dc368ee0312","added_by":"auto","created_at":"2024-03-13 11:44:01","extension":"docx","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":47638,"visible":true,"origin":"","legend":"","description":"","filename":"Tables36.docx","url":"https://assets-eu.researchsquare.com/files/rs-3969238/v1/cf14dde68646a289de21a24e.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals","fulltext":[{"header":"Introduction","content":"\u003cp\u003eChemical grouping based on category formation allows risk assessment of a data-poor substance through inference from the data on other members of the same category. The concept behind the approach is based on the notion that similar substances generally have similar properties. At the physicochemical level, the factors considered for grouping can be a similarity in chemical structures, functional group(s); metabolic/ degradation profiles, or additional parameters, such as log P, protein binding, etc. Where sufficient 'similarity' criteria are met in a set of chemical substances that follow a regular pattern, it can be considered a 'category'. According to the OECD grouping guidance (OECD, 2014): \u0026lsquo;A chemical category is a group of chemicals whose physicochemical and human health and/or ecotoxicological properties and/or environmental fate properties are likely to be similar or follow a regular pattern as a result of structural similarity. The similarities may be based on the following:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003ea common functional grouping\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ecommon constituents or chemical classes\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ean incremental and constant changes across the category\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe likelihood of common precursors and/or breakdown products, via physical or biological processes, which result in structurally similar chemicals.\u0026rsquo;\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe European Chemicals Regulation \u0026ldquo;REACH\u0026rdquo; encourages the use of grouping and categorisation of chemicals for classification, risk assessment and labelling purposes. Some of the data gaps in this regard can be filled using read-across within a given category, which is based on interpolation of experimental data from the tests conducted on 'similar' (source) substances to the untested (target) substance. The data for the endpoint in question for the target substance are predicted using the experimental data for the same endpoint of the source substance(s). Since each endpoint has a different set of complexities, e.g. in regard to mode of action against a biological target site, a read-across needs to be considered on an endpoint-by-endpoint basis (EChA, 2017).\u003c/p\u003e \u003cp\u003eThere has been an increasing emphasis on applying the 3Rs principles to refine, reduce and replace the use of animals in chemical safety testing under most regulatory frameworks in Europe. A complete ban on animal testing has been implemented under the EU Cosmetic Regulation since March 2013. Such developments have made the use of read-across in risk assessments ever more important, with emphasis on the outcomes to be transparent and reliable. In this regard, the definition of 'similarity' between the target and source substances has also evolved to include similarity not only in the structural features, but also in other physicochemical, mechanistic and/or metabolic aspects (Schultz and Cronin, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2017\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eFor read-across to be valid, a robust category of analogues must be derived from high quality datasets. A valuable tool for achieving this is the OECD QSAR Toolbox, which is a freely available multifunctional in silico platform that allows the users to make informed decisions about toxicity predictions for a range of (eco)toxicological endpoints (Schultz et al, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). As part of the process of identifying a set of analogues for read-across, the Toolbox allows the user to apply structural alerts in the form of computational \u0026ldquo;profilers\u0026rdquo;. One or more profilers are available for many of the toxicological endpoints that are based around chemistry; e.g. covalent binding to proteins or DNA, and/or other mechanistic or toxicological aspects. These profilers incorporate structural alerts, and in some cases a combination of structural alerts and physicochemical rules, that are relevant to a specific toxicological endpoint. The target compound is first subjected to profiling, and then the profile is used to screen for compounds in the databases with the same or similar structural, mechanistic, and/or toxicological profiles. The analogues found this way are reduced to those that have measured values for the specific endpoint(s) of interest and therefore provide a basis for prediction of the endpoint value of the target compound.\u003c/p\u003e \u003cp\u003eThe OECD QSAR Toolbox (referred to herein as the \u0026ldquo;Toolbox\u0026rdquo;) incorporates databases on chemical (e.g. properties), toxicological and ecotoxicological data, as well as estimated values from in silico QSAR models. The system also incorporates QSAR models built within an informatics chassis, that are designed for in silico prediction of toxicological hazard. The Toolbox therefore allows the user to perform a number of functions (OECD \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2008\u003c/span\u003e):\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eIdentification of analogues for a chemical, retrieval if experimental results available for those analogues and data gap filling by read-across or trend analysis;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCategorisation of large inventories of chemicals according to mechanisms or modes of action;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eFilling of data gaps for a chemical by using appropriate model(s) from the collection of QSAR models;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eEvaluation of the robustness of a potential analogue for read-across;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eEvaluation of the appropriateness of a (Q)SAR model for filling a data gap for a particular target chemical; and\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe capability of building QSAR models.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eOver the years, the Toolbox has become a widely used platform for chemical grouping/categorisation and estimation of chemical toxicity by \u003cem\u003ein silico\u003c/em\u003e methods and read-across. Despite the increasing reliance of risk assessors on the Toolbox, attempts have only recently started to assess the reliability and limitations of the profilers provided in the system (Devillers et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2011\u003c/span\u003e; Mombelli, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2012\u003c/span\u003e; Yordanova et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). As part of the current study, it was felt that an in-depth assessment of the profilers was necessary to determine the usefulness and limitations for their use in screening databases for analogous compounds for further use in read-across or development of (Q)SAR models. This study, therefore, investigated the performance of the OECD QSAR Toolbox profilers for mutagenicity, carcinogenicity and skin sensitisation potential of chemical substances, to assess how reliably they identified both positive and negative compounds contained within the databases. The analysis also aimed at investigating any factors that would limit the usefulness of a profiler, and to propose ways to improving their performance.\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cp\u003eThis study used version 3.1 of the Toolbox. As part of the appraisal of the available profilers, a number of databases integral to the Toolbox as well as a few additional databases from publicly available sources were examined. These are listed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eToxicological databases used in this study (details given for each database are as retrieved at the time of this study in 2020)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDatabase/ source\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBacterial mutagenicity ISSSTY\u003c/p\u003e \u003cp\u003e(donated to the Toolbox by the Istituto Superiore di Sanit\u0026agrave; (ISS), Rome, Italy)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e41,634 Ames test data points for 7,367 compounds. The overall endpoint value (positive, negative, equivocal, inconclusive) outcome is determined as:\u003c/p\u003e \u003cp\u003e- Positive: at least one strain is positive (with or without Metabolic activation);\u003c/p\u003e \u003cp\u003e- Equivocal: no strain is positive, and at least one equivocal result is present in one of the following strains (with or without Metabolic activation): TA1535, TA100, TA98, TA1538, TA1535, TA97;\u003c/p\u003e \u003cp\u003e- Negative: no positive or equivocal results are present in any strain, and negative outcomes exist for: a) at least one strain from among TA1535 or TA100 or TA97 (with and without Metabolic activation); and b) at least one strain from among TA1538 or TA98 or TA1537 (with and without Metabolic activation);\u003c/p\u003e \u003cp\u003e- Inconclusive: If none of the above criteria are fulfilled. When more than one experiment in one strain was available, the number of reported positive and negative studies was counted, and the strain overall outcome was determined as follows: if the percentage of Positive studies is lower than 40%, then outcome = Negative; if the percentage of Positive studies is between 40 \u0026minus;\u0026thinsp;60%, then outcome = Equivocal; if the percentage of Positive studies is higher than 60%, then outcome = Positive (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCarcinogenicity and mutagenicity (ISSCAN)\u003c/p\u003e \u003cp\u003e(donated to the Toolbox by the Istituto Superiore di Sanit\u0026agrave; (ISS), Rome, Italy)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6,979 data points for 1,150 compounds. There are three endpoints for which data are presented; gene mutation, summary carcinogenicity, and TD50. The TD50 data are not used in this current study. Gene mutation data in Ames reported in the same way as in the previous database, and a single datum point is available for each of the 832 compounds. Summary carcinogenicity data are represented as \u0026ldquo;positive\u0026rdquo;, \u0026ldquo;negative\u0026rdquo; or \u0026ldquo;equivocal\u0026rdquo;. Positives are carcinogenic in at least one experimental group; equivocal results are given to chemicals with equivocal results in at least one experimental group, together with negative results in the other experimental groups, and negatives are non-carcinogenic in all tests (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGenotoxicity OASIS\u003c/p\u003e \u003cp\u003e(donated to the Toolbox by the Laboratory of Mathematical Chemistry, Bourgas, Bulgaria)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e7,500 compounds collected from seven sources. It contains data for mutagenic determined by the Ames test with and without metabolic activation. The database also includes chromosomal aberrations determined by in vitro tests using Chinese hamster lung cells (CHL, with and without S9). Micronucleus (MN) and mouse lymphoma gene mutation assay (MLA) are evaluated by Chinese hamster lung cells (CHL / IU) and by in vitro T-lymphoma cell lines, respectively. All endpoints were evaluated on a dichotomic scale (YES/NO). Data used in this study were the Ames test data, and chromosomal aberration as the Toolbox has profilers for these endpoints (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCRD-AGES\u003c/p\u003e \u003cp\u003e(Data from the UK Chemicals Regulatory Directorate and the Austrian Agency for Health and Food Safety)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e179 pesticides with mutagenicity data, and 100 with carcinogenicity data (Worth et al \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2010\u003c/span\u003e). Mutagenicity data are binary active/inactive from Ames tests and carcinogenicity also binary active/inactive from a range of tests\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDSS Pesticide carcinogenicity\u003c/p\u003e \u003cp\u003e(the US EPA National Center for Computational Toxicology, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.epa.gov/ncct/dsstox/index.html\u003c/span\u003e\u003cspan address=\"http://www.epa.gov/ncct/dsstox/index.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSummary carcinogenicity data for 1,282 pesticide compounds\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSAR carcinogenicity and genotoxicity databases\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThese are high-quality data databases used by Leadscope\u0026reg; program to build in silico predictive models. Both databases contain summarised in vitro and in vivo cancer and mutagenicity endpoints along with chemical structures. To ensure the high quality data for SAR, structure form, analysis of various salt forms of chemical compounds and their respective toxicity data have also been carried out to derive an overall endpoint for the active portion of the chemical. Several sources of experimental test results have been included in these databases, such as from FDA, NTP, CCRIS, CPDB and other primary sources. All chemical structures have been provided in SAR, neutral and tested form, and confirmed for accuracy.\u003c/p\u003e \u003cp\u003eThe SAR carcinogenicity database includes 3,598 compounds with 11,538 test results and provides carcinogenicity study endpoint for male and female rat (1,774, 1,725 compounds respectively) and male and female mouse (1,640, 1,675 compounds respectively). The SAR genotoxicity database provides compound-level calls for 46 genetic toxicity endpoints for 10,534 compounds. These include 32 bacterial mutagenicity endpoints, four in vitro mammalian, five in vitro chromosomal aberration and six in vivo micronucleus results (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEFSA Pesticides Mutagenicity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAmes test active/inactive classification for 741 pesticides, compiled by, and downloaded from the European Food Safety Authority.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNISS mutagenicity database\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA database of 1,863 compounds with binary active/inactive Ames test data. Downloaded from the US National Institute of Statistical Sciences, (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.niss.org/\u003c/span\u003e\u003cspan address=\"http://www.niss.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInchemicotox Skin Sensitisation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA version of the Cronin and Basketter dataset (Cronin and Basketter 1994) from the Inchemicotox project (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.inchemicotox.org/\u003c/span\u003e\u003cspan address=\"http://www.inchemicotox.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), comprising 322 compounds, with results taken from guinea pig maximisation tests. The classification is derived from the percentage of animals sensitised in the test: non-sensitiser\u0026thinsp;=\u0026thinsp;0\u0026ndash;9%, weak sensitiser\u0026thinsp;=\u0026thinsp;10\u0026ndash;29%, moderate sensitiser\u0026thinsp;=\u0026thinsp;30\u0026ndash;79%, strong sensitiser\u0026thinsp;=\u0026thinsp;80\u0026ndash;100%.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCAESAR Skin Sensitisation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e209 compounds from the EU CAESAR project (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.caesar-project.eu\u003c/span\u003e\u003cspan address=\"http://www.caesar-project.eu\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). For developing classification models, this data set was subdivided in two classes, sensitiser (S) and non-sensitisers (N), which gave a good distribution of the numbers of compounds in each class. The class S merges the first four ranges established by ECETOC: Extreme (EC3\u0026thinsp;\u0026lt;\u0026thinsp;0.1%), Strong (0.1%\u0026lt;EC3\u0026thinsp;\u0026lt;\u0026thinsp;1%), and Moderate (1%\u0026lt;EC3\u0026thinsp;\u0026lt;\u0026thinsp;10%) and Weak (EC3\u0026thinsp;\u0026gt;\u0026thinsp;10%) ranges; the class N regroups all compounds belonging to the non sensitisers\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eECETOX Skin sensitisation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e39 compounds with experimental results on skin and respiratory sensitisation. The compounds were selected as known sensitisers and non-sensitisers for the assessment of novel test techniques. (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.ecetoc.org/technical-reports\u003c/span\u003e\u003cspan address=\"http://www.ecetoc.org/technical-reports\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOECD Skin Sensitisation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1,036 compounds from 2 databases and includes chemicals tested by Local Lymph Node Assay,(LLNA) or Guinea Pig Maximization Test, (GPMT). Based on the observed skin sensitisation effect the chemicals are classified in three classes: - strong sensitisers, weak sensitisers or non-sensitisers\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLazar Opentox Rat Carcinogenicity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLazy Structure-Activity Relationships (LAZAR) is an open-source tool for the prediction of complex toxicological endpoints such as carcinogenicity (female/male, hamster/ mouse/rat/rodent) and Salmonella mutagenicity. The compounds were selected from database of experimental toxicity data. Carcinogenicity models are based on CPDB, while the Salmonella mutagenicity model uses a dataset of 3895 compounds determined in vitro. (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://lazar.in-silico.de/predict\u003c/span\u003e\u003cspan address=\"https://lazar.in-silico.de/predict\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVEGA carcinogenicity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe VEGA platform serves to access a number of QSAR models for predicting mutagenicity and carcinogenicity. The compounds were selected from a set of 4225 molecules tested with the Ames bacterial test. And for carcinogenicity were selected from set of 805 chemicals from the Carcinogenic Potency Database (CPDB)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eThe Carcinogenic Potency Database (CPDB)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eData relating to cancer causing chemicals were compiled from the Carcinogenic Potency Database (CPDB), which is freely available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://toxnet.nlm.nih.gov/cpdb/cpdb.html\u003c/span\u003e\u003cspan address=\"http://toxnet.nlm.nih.gov/cpdb/cpdb.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. This database is a widely used and unique international resource comprising the results of 6,540 chronic, long-term animal carcinogenicity tests on 1,547 chemicals in rats, mice, dogs, hamsters and non-human primates (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe profilers assessed in this study are listed in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. These were used to assess the compounds that had experimental values against specific endpoints from the databases listed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe OECD QSAR Toolbox profilers assessed in this study\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProfiler/ toxicological endpoint\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMutagenicity/ genotoxicity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNA binding by OASIS v1.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler is a mechanistic profiler developed from an analysis of Ames mutagenicity data. It contains a number of structural alerts that have been shown to be related to established electrophilic reaction chemistry known to be important in covalent DNA binding (Mekenyan et al \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2004\u003c/span\u003e; Serafimova et al. \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2007\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNA binding by OECD.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler is based on structural alerts for the electrophilic reaction chemistry associated with covalent DNA binding (Enoch and Cronin 2010). The profiler returns a range of structural alerts that contain electrophilic centres or those that can be metabolically activated to electrophiles.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCarcinogenicity (genotox and nongenotox) alerts by ISS.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler is based on a list of 55 structural alerts from the software Toxtree (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://toxtree.sourceforge.net/\u003c/span\u003e\u003cspan address=\"http://toxtree.sourceforge.net/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). About 20 of the alerts are for non-genotoxic carcinogenicity, and the remainder for genotoxic carcinogenicity (mutagenicity)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNA alerts for AMES, MN and CA by OASIS v.1.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ea refinement of above\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003emutagenicity (Ames test) alerts by ISS.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe present list of SAs is a subset of the original Toxtree list, obtained by eliminating the SAs for nongenotoxic carcinogenicity and is a refinement of above\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eIn vivo\u003c/em\u003e mutagenicity (Micronucleus) alerts by ISS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e. This profiler is based on the ToxMic rule-base of the software Toxtree. This rule-base provides a list of 35 structural alerts (SAs) for a preliminary screening of potentially \u003cem\u003ein vivo\u003c/em\u003e mutagens. These SAs are molecular functional groups or substructures that are known to be linked to the induction of effects in the \u003cem\u003ein vivo\u003c/em\u003e micronucleus assay\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCarcinogenicity\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNA binding by OASIS v1.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAs above\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNA binding by OECD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAs above\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCarcinogenicity (genotox and nongenotox) alerts by ISS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe SAs for carcinogenicity are molecular functional groups or substructures known to be linked to the carcinogenic activity of chemicals. As one or more SAs embedded in a molecular structure are recognised, the system flags the potential carcinogenicity of the chemical (Aljallal,2020).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOncoLogic Primary Classifier\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler consists of molecular definitions derived by the Toolbox developers to mimic the structural criteria of chemical classes of potential carcinogens covered by the U.S. Environmental Protection Agency\u0026rsquo;s OncoLogic\u0026trade; Cancer Expert System for Predicting the Carcinogenicity Potential (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.epa.gov/oppt/sf/pubs/oncologic.htm\u003c/span\u003e\u003cspan address=\"http://www.epa.gov/oppt/sf/pubs/oncologic.htm\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSkin Sensitisation\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProtein binding by OASIS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThese profilers have been developed to indicative of skin sensitisation potential and consist of 85 structural alerts relating to 11 reactions, or chemical interactions, which are known to be associated with skin sensitisers\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProtein Binding by OECD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe protein binding by OECD profiler contains 16 mechanistic alerts covering 52 structural alerts. These data are supported by mechanistic chemistry and references to the scientific literature (the meta data). They represent a parallel approach to those of the OASIS profiler and capture mechanistic features of target compounds\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProtein binding potency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler is developed on the basis of empirical data for thiol reactivity expressed by the \u003cem\u003ein chemico\u003c/em\u003e RC50 value. All the chemicals have two common electrophilic mechanisms of interaction with GSH \u0026ndash; interaction via SN2 and interaction via Michael addition (MA) mechanism. The profiler contains 49 MA and 46 SN2 categories\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKeratinocyte gene expression\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profile is built in relation to the database derived from the KeratinoSens assay, which examines the potential for chemicals to induce the expression of a luciferase reporter gene under control of a single copy of the ARE element of the human AKR1C2 gene stably inserted into immortalised human keratinocytes. Relevance to skin sensitisation is inferred from the relationship of Keap1-Nrf2-ARE regulatory pathway and its detection of electrophilic chemicals to sensitisation. The profiler contains 22 categories\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProtein binding alerts for skin sensitisation by OASIS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profiler seems to be much the same as the one at 2.2.3.1 above though there are some minor differences\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDPRA Lysine peptide depletion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis profile is built on the basis of data derived from Direct Peptide Reactivity Assay (DPRA). The DPRA is a reactivity assay which evaluates the ability of chemicals to react with proteins. Model synthetic peptides containing either lysine or cysteine are used. The remaining concentration of cysteine- or lysine-containing peptide is measure after 24 hours incubation with the test chemical at 25\u0026thinsp;\u0026plusmn;\u0026thinsp;2.5\u0026ordm;C. The peptide reactivity is reported as percent peptide depletion. The relevance with skin sensitisation is the presence of cysteine and lysine residuals in the skin proteins.\u003c/p\u003e \u003cp\u003eThe profiler contains 24 structural alerts extracted from about 110 chemicals with experimentally measured lysine depletion values\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDPRA Cysteine peptide depletion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAs described above, this profiler contains 32 categories of alerts\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFurther information is provided in the supplementary file on the total number of substances identified with experimental data against each endpoint in different databases, and that used in the assessment of the profilers performance.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eData analysis\u003c/h2\u003e \u003cp\u003eThe compounds identified in the databases for the specific endpoint were profiled using appropriate profilers. For each of the compounds, if an alert was triggered, the compound was allocated a score of 1, if no alerts were triggered, a score of 0 was allocated. The results were compared with the assigned binary activities from the original database (positive\u0026thinsp;=\u0026thinsp;1; negative\u0026thinsp;=\u0026thinsp;0).\u003c/p\u003e \u003cp\u003eCooper statistics (Cooper \u003cem\u003eet al\u003c/em\u003e, 1979) were used to assess the results against the experimental values given in the databases, by calculating the sensitivity, specificity and accuracy of the alert triggers as follows:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eSensitivity (True positive rate)\u0026thinsp;=\u0026thinsp;TP/ TP\u0026thinsp;+\u0026thinsp;FN\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSpecificity (True negative rate)\u0026thinsp;=\u0026thinsp;TN/TN\u0026thinsp;+\u0026thinsp;FP\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAccuracy = (TN\u0026thinsp;+\u0026thinsp;TP)/(TN\u0026thinsp;+\u0026thinsp;FP\u0026thinsp;+\u0026thinsp;FN\u0026thinsp;+\u0026thinsp;TP)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ePPV (Positive predictive value) or (precision)\u0026thinsp;\u003cb\u003e=\u003c/b\u003e\u0026thinsp;TP/ TP\u0026thinsp;+\u0026thinsp;FP\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eMCC = (TPxTN)-(FPxFN)/\u0026radic;(TP\u0026thinsp;+\u0026thinsp;FN)(TP\u0026thinsp;+\u0026thinsp;FP)(TN\u0026thinsp;+\u0026thinsp;FN)(TN\u0026thinsp;+\u0026thinsp;FP)\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eWhere TP\u0026thinsp;=\u0026thinsp;True positive, TN\u0026thinsp;=\u0026thinsp;True negative, FP\u0026thinsp;=\u0026thinsp;False positive, FN\u0026thinsp;=\u0026thinsp;False negative\u003c/p\u003e \u003cp\u003e \u003cb\u003eSensitivity\u003c/b\u003e is defined as the percentage of correctly classified positive predictions among the total number of positive instances.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSpecificity\u003c/b\u003e is the percentage of correct negative predictions compared to the total number of negatives.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAccuracy\u003c/b\u003e (concordance or \u0026ldquo;Q\u0026rdquo;) is defined as the total number both positive and negatives correctly predicted among the total number of compounds.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePPV\u003c/b\u003e (positive predictive value) is defined as the total number of correctly classified positive predictions among the total number of both negative and positive instance.\u003c/p\u003e \u003cp\u003e \u003cb\u003eMCC\u003c/b\u003e (Matthews correlation coefficient) is a weighted value that overcomes any imbalance in the data classes which might lead to over optimistic values of Q (Matthews, 1975). An \u003cem\u003eMCC\u003c/em\u003e value of 1 indicates that the model can predict the data classes of unknown compounds perfectly, whilst a \u003cem\u003eMCC\u003c/em\u003e value of 0 indicates that the predictions are no better than random guessing, and a \u003cem\u003eMCC\u003c/em\u003e value of -1 indicates total disagreement between the predicted data and the actual data.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eThe results of the assessment of the profilers against the experimental data for mutagenicity, carcinogenicity and skin sensitisation are shown in Tables \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e and \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e respectively. Further detailed analysis was undertaken to identify over-predicting structural alerts in the carcinogenicity profilers i.e. with the Precision or PPV (positive predictive value) lower than 0.5. This analysis was performed to determine structural alerts with little information or predictive capability to increase the sensitivity and overall accuracy of the profiler. Detailed analysis of 13 non-genotoxic carcinogenicity structural alerts was conducted and results are presented in Table \u003cspan class=\"InternalRef\"\u003e6.1\u003c/span\u003e and 6.2.\u003c/p\u003e\n\u003cp\u003eAn additional analysis for the Oncologic Primary Classification carcinogenicity profiler was performed for 30 structural alerts incorporated in the profiler. The purpose of this analysis was to assess which of the structural alert(s) had a precision PPV (positive predictive value) lower than 0.5. This analysis is presented in Table \u003cspan class=\"InternalRef\"\u003e6.3\u003c/span\u003e.\u003c/p\u003e\n\u003cp\u003eThe cutoff value was set to be 0.5 to ensure that none of these profilers had a lower predictive power compared to bacterial Ames test, which has also been applied to predict carcinogenicity of genotoxic substances in rodents. The high predictive power of positive Ames, which ranges from 77\u0026ndash;90% depending on various factors, makes it superior to any other \u003cem\u003ein vitro\u003c/em\u003e genotoxicity assay, all of which have a relatively lower performance in terms of predicting genotoxicity (Kazius \u003cem\u003eet al\u003c/em\u003e., 2006).\u003c/p\u003e\n\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\n \u003ch2\u003eMutagenicity profilers\u003c/h2\u003e\n \u003cp\u003eThe results shown in Tables \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e indicated that the accuracy (percentage of positives and negatives correctly predicted) of the mutagenicity profilers varies across the datasets from 51\u0026ndash;76%. Clearly, whilst 76% can be accepted because it is in line with the level of error generally seen in the experimental data in most databases, 51% is barely better than chance. The micronucleus alerts appear to be general and, as such, significantly over-predict mutagenicity. The most common alert triggered by this profiler is \u0026ldquo;Hacceptor-path3-Hacceptor\u0026rdquo;. This alert indicates the non-covalent binding of the target chemical to DNA via two bonded atoms connecting two H bond acceptors (Snyder \u003cem\u003eet al\u003c/em\u003e. 2006). However, it appears that such a functional grouping is common in both mutagens and non-mutagens. It is likely that the performance of this profiler would improve if this specific alert was omitted.\u003c/p\u003e\n \u003cp\u003eAs expected, both DNA binding profilers work best with the data obtained from Ames type tests (due to the availability of large databases for this test), but do not perform well for chromosome aberration or micronucleus data.\u003c/p\u003e\n \u003cp\u003eThe genotoxicity and non-genotoxicity alerts (ISS) have acceptable true positive results but fail to distinguish the negatives. Overall, these alerts perform best with Ames type data.\u003c/p\u003e\n \u003cp\u003eThe OASIS DNA alerts for Ames, micronucleus and chromosomal aberration predict the results in Ames datasets fairly well but for both micronucleus data and chromosomal aberration data, these profilers underpredict positive compounds with sensitivity rates ranging from 36 to 44%. The ISS Ames test alerts have accuracies over 70% for Ames datasets, which together with MCC values greater than 0.5, indicate that the performance is independent of skewed sample categories. It, however, needs to be noted that the micronucleus/ CA alerts may not be suitable predictors of Ames. They need to be considered separately and used, along with Ames, to develop the overall weight of evidence.\u003c/p\u003e\n \u003cp\u003eThe receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as the y coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the quality or performance of diagnostic tests which in this case is an in silico profiler. As it is shown in Fig. \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e, the ROC curve analysis showed that Ames ISS profiler achieved the highest balanced accuracy for both true positive rate and low false positive rate values.\u003c/p\u003e\n \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e\n \u003ch2\u003eCarcinogenicity profilers\u003c/h2\u003e\n \u003cp\u003eBoth DNA binding profilers performed equally poor with carcinogens and non carcinogens from all datasets, with accuracy values rarely above 60% and, MCC values indicating a performance barely better than chance.\u003c/p\u003e\n \u003cp\u003eThe ISS carcinogenicity alerts fared a little better in predicting carcinogens, but showed a poor segregation of non-carcinogens reduced the overall effectiveness of this profiler with accuracy levels between 57% and 68% for the sample datasets, and modest to poor performance on skewed datasets as indicated by the MCC values of 0.17 to 0.36. The ROC analysis, shown in Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, indicate that ISS carcinogenicity profiler was the highest quality performance profiler compared to the other 3 profilers in terms of both true positive rate and false positive rate.\u003c/p\u003e\n \u003cp\u003e13 Non-genotoxic carcinogenicity (NGC) structural alerts among ISS carcinogenicity profiler were analysed individually to test their performance by PPV (positive predictive value). As shown in Table \u003cspan class=\"InternalRef\"\u003e6.1\u003c/span\u003e, the overall PPV of ISS non-genotoxic carcinogen structural alerts was 57%, where 326 substances are truly predicted as nongenotoxic carcinogen out of total 570 substances that had been identified to contain one of the 13 NGC structural alerts.\u003c/p\u003e\n \u003cp\u003eThe precision value (PPV) for non-genotoxic carcinogenicity structural alert ranged from 0.92 for Trichloro (or fluoro) ethylene and Tetrachloro (or fluoro) ethylene as a highest PPV to 0.39 for Quercetin type flavonoids. Four out of 13 NGC structural alerts show over-prediction. These are thiocarbonyl, substituted n-alkylcarboxylic acids, quercetin type flavonoids, and phtalate (or butyl) diesters and monoesters. All of these four structural alerts predict non carcinogenic substances as carcinogens in more than 50% of the total substances that contain this structural alert, which lowers the total accuracy of the ISS carcinogenicity profiler.\u003c/p\u003e\n \u003cp\u003eThiocarbonyl NGC structural alert was flagged in 106 substances as the only structural alert. Any substance that contained more than one NGC structural alert was not counted in this analysis to avoid any interference. Sixty two non-carcinogenic substances were falsely predicted as carcinogenic substances by thiocarbonyl structural alerts with sensitivity rate of 0.42. Due to this over-prediction of thiocarbonyl alert, it can be suggested that taking this alert out could increase sensitivity of the total NGC structural alerts and the ISS carcinogenicity profiler. This would, however, not be the ideal solution, as any thiocarbonyl NGCs would then be completely out of the scope of the profiler. Instead, it is proposed that further research be carried out to see whether performance of this alert can be improved using a larger database of thiocarbonyl substances.\u003c/p\u003e\n \u003cp\u003eLikewise, the other three NGC structural alerts; i.e. substituted n-alkylcarboxylic acids, quercetin type flavonoids and phthalate (or butyl) diesters and monoesters also showed a precision value lower than 0.5 with a 0.42, 0.39 and 0.38 respectively. Again ignoring these four structural alerts increased the total precision value of NGC structural alerts and consequently of the performance of the ISS carcinogenicity profiler. The results (Table \u003cspan class=\"InternalRef\"\u003e6.2\u003c/span\u003e) showed that the precision value of ISS non-genotoxic structural alert was improved from 0.57 to 0.64 by ignoring the 4 structural alerts. However, for the reasons mentioned before, it is proposed that further research should be carried out to improve the performance of these alerts within the profiler.\u003c/p\u003e\n \u003cp\u003eThe Oncologic primary classification alert over-predicted carcinogens with sensitivity rates of 66\u0026ndash;73% at the expense of poor prediction of non-carcinogens (30\u0026ndash;46%), resulting in overall performance, which is barely better than chance for most of the datasets. All 30 structural alerts in Oncologic primary classification profiler were individually analysed for their precision as shown in Table \u003cspan class=\"InternalRef\"\u003e6.3\u003c/span\u003e. The four structural alerts showed over-prediction of non-carcinogenic substances as carcinogenic in more than 50% of the total substances containing this structural alert. These four alerts were carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone reactive functional groups. Carbamate type compounds structural alert for carcinogenicity was triggered in 63 substances with only 22 true positive carcinogenic substances. The other 43 substances that were flagged by this alert to be carcinogenic were non-carcinogenic in real experimental tests. This gave a low precision value for carbamate type compounds of 0.35. The second structural alert in oncologic primary classification profiler with low precision value (lower than 0.36) was organophosphorus type compounds structural alert, where 80 substances out of 126 flagged by this alert were wrongly predicted as carcinogenic substances. Peroxide type compounds structural alert showed only 0.27 precision (positive predictive value) rate with only 3 correctly predicted carcinogenic substances out of 11 substances flagged by the alert. The lowest structural alert in precision within oncologic primary classification profiler was for reactive ketone functional groups with only 0.1 where there was an over-prediction for 19 out of total 21 flagged substances. This mean that only 2 substances that were flagged by this alert were correctly predicted as carcinogenic substances out of the total 21 substances.\u003c/p\u003e\n \u003cp\u003eIt can therefore be suggested that all four of these structural alerts could be ignored from the oncologic primary classification profiler to increase the total sensitivity and accuracy of the profiler. Indeed, as shown in Table \u003cspan class=\"InternalRef\"\u003e6.3\u003c/span\u003e, the total precision of the profiler was improved by 0.023 which is nearly 3% improvement in the overall performance of the profiler.\u003c/p\u003e\n \u003cp\u003eThis would, however, not be the ideal solution, as this will miss carcinogenic compounds and would be completely out of the scope of the profiler. Therefore, it is proposed that further research be carried out to see whether performance of these four alerts can be improved using a larger database of thiocarbonyl substances.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec7\" class=\"Section3\"\u003e\n \u003ch2\u003eSkin sensitisation profilers\u003c/h2\u003e\n \u003cp\u003eThe performance of the protein binding profilers was not found to be consistent across the sampled datasets. For the CAESAR dataset, these profilers tend to have a low predictivity for non-sensitisers, whilst for the other datasets it is the sensitisers that are not well predicted. Overall, the performance of these predictors is moderate to poor for all the datasets.\u003c/p\u003e\n \u003cp\u003eThe DPRA lysine peptide depletion profilers showed a similar pattern, with performance being uniformly not better than chance for the CAESAR dataset but highly under-predictive for sensitisers in the other datasets. The protein binding potency profiler is also uniformly poor across all datasets, failing to detect the majority of sensitisers, with sensitivity rates between 12% and 29%. A similar pattern is seen with the keratinocyte gene expression profiler, where sensitivity rates were 19\u0026ndash;43%.\u003c/p\u003e\n \u003cp\u003eThe overall ROC analysis for all seven profiler shown in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e indicated that protein binding OASIS has a relatively better performance compared to the other skin sensitization profilers in OECD Toolbox.\u003c/p\u003e\n \u003cp\u003eThe study also carried out further analysis to investigate the contribution of different alerts within a given profiler on its overall performance. For this purpose, non-genotoxic carcinogenicity profilers were investigated as an example. This was because, under most EU regulatory frameworks, tests for carcinogenicity are only required when there is either a positive in vitro mutagenicity/genotoxicity test, or there are indications of carcinogenic effects from long term in vivo studies. This means that, whilst the current risk assessments framework would identify genotoxic carcinogens, it is possible that the carcinogenic effects of a nongenotoxic mechanism will not be identified without in vivo tests that are increasingly restricted or banned under different regulatory frameworks.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"Discussion \u0026 Conclusions","content":"\u003cp\u003eThis study has assessed and evaluated the structural alerts and profilers provided in the OECD QSAR Toolbox for mutagenicity, carcinogenicity, and skin sensitisation. These profilers are provided for the purpose of constructing categories of mechanistic and identifying structural analogues of a target compound. Apart from the OECD QSAR Toolbox, profilers are also used by a number of other applications for indicating certain toxicity endpoints, such as Toxtree (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://toxtree.sourceforge.net/\u003c/span\u003e\u003cspan address=\"http://toxtree.sourceforge.net/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), and Oncologic (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ewww.epa.gov/oppt/sf/pubs/oncologic.htm\u003c/span\u003e\u003cspan address=\"http://www.epa.gov/oppt/sf/pubs/oncologic.htm\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). Although not intended for making toxicological predictions in themselves, these profilers provide a useful means for read-across from experimental data on analogous compounds to estimate a property or biological activity of an untested compound. The accuracy and reliability of a profiler in terms of predicting a target compound is, therefore, important for defining the structural and functional features so that it is placed in the correct category/group of analogous substances for the purpose of read-across. This is also important because, in practice, safety of chemical substances is often based on (Q)SAR Toolbox profilers alone in the dossiers submitted for regulatory risk assessment, where the absence of an alert alone is (wrongly) argued as an evidence for the absence of toxicity.\u003c/p\u003e \u003cp\u003eIn this regard, those profilers that are equally likely to select endpoint-positive and endpoint-negative compounds into a grouping of analogues to be used in a read across, will by definition give rise to equivocal predictions for the target compound. This makes it essential to know how accurately different profilers perform in terms of sensitivity, specificity and accuracy, and to investigate possibilities for their improvement.\u003c/p\u003e \u003cp\u003eThis also requires an understanding of the role and merits of each individual alert within a profiler so that only the most relevant and reliable ones can be relied upon as indicators of the particular endpoint. The results of this study have shown that many structural alerts within each profiler have unacceptably low predictivity, which has a bearing on the performance of the relevant profiler.\u003c/p\u003e \u003cp\u003eAs an example, this study has found that the alert \u0026ldquo;Hacceptor-path3-Hacceptor\u0026rdquo; in the micronucleus profiler is too ubiquitous to be a useful indicator of mutagenicity. Similarly, in three structural alerts for non-genotoxic carcinogenicity within the ISS carcinogenicity profiler (thiocarbonyl, sub alkyl carboxylic acid, quercetin and phthalate), and the four structural alerts in oncologic primary classification profiler (carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone functional groups) were identified to have extremely low precision values. As shown in the examples investigated in this study, the omission or substitution of those alerts that unduly draw predictions towards equivocal outcomes within a profiler, can improve the overall performance of the profilers. This, however, would bring the drawback of making some of the toxicity alerts out of the scope of the profiler, and therefore further research is needed to find out whether they can be substituted with other more efficient alerts, or their performance can be enhanced by other means, e.g. by exploring larger datasets of relevant substances.\u003c/p\u003e \u003cp\u003eAnother factor influencing the segregation of compounds by the alerts could be the way in which the categorical data found in the 3 endpoint datasets studied here is derived. Very often the binary categorisation of data is achieved by manipulation of the continuous data in some way to provide \u0026ldquo;cut off\u0026rdquo; points for positive or negative assignment. The way in which this is done may affect the flagging of an alert in an essentially \u0026ldquo;negative\u0026rdquo; compound, or vice-versa. By definition, most alerts have been derived from datasets that are rich in endpoint-positive compounds, because deriving \u0026ldquo;negative alerts\u0026rdquo;, like proving a negative hypothesis, is generally not feasible, and negative results are not published so often. This study, therefore, has also given an insight into those alerts that may be found equally in endpoint-positive or negative compounds, and those which may be more effectively utilised to form groups of analogues for read-across predictions.\u003c/p\u003e \u003cp\u003eFurther research in this area is necessary to study the suitability and merits of each of the alerts within the profilers, both within the OECD Toolbox and other in silico toxicity platforms, to identify the root causes of the inadequacies and to investigate the possibilities for improvement in the profilers\u0026rsquo; performance. This will, by implication, also improve the reliability of chemical read-across and grouping/categorisation for use in classification, labelling and risk assessment.\u003c/p\u003e \u003cp\u003eThe main goal of the study was to inform the scientific community about any limitations of the Toolkit profiles, and the structural alerts contained within the profilers, that are otherwise useful in understanding the underlying mechanisms of toxicity. The in-depth analysis has indicated that many of the profilers have quite poor performance when tested against a large number of compounds with experimental data. This casts doubts over their usefulness in category formation for read-across for which they are commonly employed for subsequent use in regulatory risk assessments. There is therefore a need for caution when using the profilers for chemical category formation and in read-across, because reliability of the latter would be dependent on the accuracy and specificity of grouping/category formation in the first place. It has also highlighted that toxicity predictions should not be performed blindly by relying on structural alerts alone, and that other in silico tools, such as QSAR models, should be employed to develop a more reliable weight of evidence for that purpose. Another important point is that the influence of any part of a compound on its biological effect(s) is not constant and strongly depends on its structural environment. Thus, any alert, even when derived by mechanistic interpretation of statistically significant QSAR models does not have automatic predictive power. Alerts should therefore be viewed as structural hypotheses of chemical action only, and their true predictive power should be confirmed by QSAR predictions and, if possible, through experimental validation. In this context, this study is positioned as a pragmatic statement that could change the thinking of both regulators and researchers.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003e\u003cu\u003eFunding statement\u003c/u\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors did not receive support from any organization for the submitted work.\u003c/p\u003e\n\u003cp\u003eNo funding was received to assist with the preparation of this manuscript.\u003c/p\u003e\n\u003cp\u003eNo funding was received for conducting this study.\u003c/p\u003e\n\u003cp\u003eNo funds, grants, or other support was received.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eM.J., Q.S. and N.P designed the study,M.J filtered and did the calculation for the profilers. M.J wrote the manuscript with support from Q.S and N.P All authors discussed the results and contributed to the final manuscript. All authors provided critical feedback and helped shape the research, analysis and manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAljallal, M (2020) \u003cem\u003eInvestigation of in Silico Modelling to Predict the Human Health Effects of Cosmetics Ingredients.\u003c/em\u003e Doctoral thesis, Liverpool John Moores University.\u003c/li\u003e\n\u003cli\u003eBalls, M. (2019). Chapter 1.1 - The Introduction and Influence of the Concept of Humane Experimental Technique. In: M. Balls, R. Combes and A. Worth, ed., \u003cem\u003eThe History of Alternative Test Methods in Toxicology\u003c/em\u003e. pp.3-6.\u003c/li\u003e\n\u003cli\u003eBenigni, R. (2008). \u003cem\u003eThe Benigni/Bossa rulebase for mutagenicity and carcinogenicity\u0026ndash;a module of Toxtree\u003c/em\u003e. EUR, pp.23241:1\u0026ndash;70.\u003c/li\u003e\n\u003cli\u003eCronin, M., Madden, J., Enoch, S. and Roberts, D. (2013). \u003cem\u003eChemical Toxicity Prediction: Category Formation and Read-Across\u003c/em\u003e. Royal society of chemistry (RSC), pp.1-29.\u003c/li\u003e\n\u003cli\u003eDevillers, J., Mombelli, E. and Samser\u0026agrave;, R. (2011). Structural alerts for estimating the carcinogenicity of pesticides and biocides. \u003cem\u003eSAR and QSAR in Environmental Research\u003c/em\u003e, 22(1-2), pp.89-106.\u003c/li\u003e\n\u003cli\u003eEuropean chemical agency (Echa) (2017). \u003cem\u003eGrouping of substances and read-across - ECHA\u003c/em\u003e. [online] Echa.europa.eu. Available at: https://echa.europa.eu/documents/10162/13628/read_across_introductory_note_en.pdf [Accessed 18 Sep. 2018].\u003c/li\u003e\n\u003cli\u003eEnoch, S.J. et al. (2011) \u0026lsquo;A review of the electrophilic reaction chemistry involved in covalent protein binding relevant to toxicity\u0026rsquo;, Critical Reviews in Toxicology, 41(9), pp. 783\u0026ndash;802. doi:10.3109/10408444.2011.598141. \u003c/li\u003e\n\u003cli\u003eMekenyan, O., Dimitrov, S., Serafimova, R., Thompson, E., Kotov, S., Dimitrova, N. and Walker, J. (2004). Identification of the Structural Requirements for Mutagenicity by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals I: TA100 Model. \u003cem\u003eChemical Research in Toxicology\u003c/em\u003e, 17(6), pp.753-766. \u003c/li\u003e\n\u003cli\u003eMombelli, E. (2012). Evaluation of the OECD (Q)SAR Application Toolbox for the profiling of estrogen receptor binding affinities. \u003cem\u003eSAR and QSAR in Environmental Research\u003c/em\u003e, 23(1-2), pp.37-57.\u003c/li\u003e\n\u003cli\u003eOECD (2008) \u003cem\u003e(Q)SAR Application Toolbox version 1.1: Getting Started\u003c/em\u003e, Organisation for Economic Cooperation and Development, Paris, France.\u003c/li\u003e\n\u003cli\u003eSchultz, T. and Cronin, M. (2017). Lessons learned from read-across case studies for repeated-dose toxicity. \u003cem\u003eRegulatory Toxicology and Pharmacology\u003c/em\u003e, 88, pp.185-191.\u003c/li\u003e\n\u003cli\u003eSchultz, T., Diderich, R., Kuseva, C. and Mekenyan, O. (2018). The OECD QSAR Toolbox Starts Its Second Decade. In: O. Nicolotti, ed., \u003cem\u003eComputational Toxicology\u003c/em\u003e. New York, NY: Humana Press, pp.55-77.\u003c/li\u003e\n\u003cli\u003eSerafimova, R., Todorov, M., Pavlov, T., Kotov, S., Jacob, E., Aptula, A. and Mekenyan, O. (2007). Identification of the Structural Requirements for Mutagencitiy, by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals. II. General Ames Mutagenicity Model. \u003cem\u003eChemical Research in Toxicology\u003c/em\u003e, 20(4), pp.662-676. \u003c/li\u003e\n\u003cli\u003eWorth A., Lapenna S., Lo Piparo E., Mostrag-Szlichty A., Serafimova R. (2010) The Applicability of Software Tools for Genotoxicity and Carcinogenicity Prediction: Case Studies relevant to the Assessment of Pesticides JRC Scientific and Technical Report. 2010 EUR 24640.\u003c/li\u003e\n\u003cli\u003eYordanova, D., Schultz, T., Kuseva, C., Ivanova, H., Pavlov, T., Chankov, G., Karakolev, Y., Gissi, A., Sobanski, T. and Mekenyan, O. (2019). Alert performance: A new functionality in the OECD QSAR Toolbox. \u003cem\u003eComputational Toxicology\u003c/em\u003e, 10, pp.26-37.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables 3 to 6 are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3969238/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3969238/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eGrowing restrictions and bans on animal testing for chemical safety assessment under different regulations have led to an increasing use of alternative methods. Read-across is one of the major approaches used for this purpose, which relies on the identification of toxicological hazards of a data-poor or untested (target) chemical from data on other already-tested (source) similar chemicals. This requires the target substance to be first assigned to a group or category of \u0026lsquo;similar\u0026rsquo; chemicals. The \u0026lsquo;similarity\u0026rsquo; may be in terms of structural features alone, or in combination with certain rules that are based on mechanistic and/or toxicological aspects. In this regard, the OECD QSAR Toolbox - a major free-access in silico platform - is widely used to derive toxicity predictions for a range of (eco) toxicological endpoints. The Toolbox allows the user to identify a set of similar chemicals (analogues) by computational \u0026lsquo;profilers\u0026rsquo; that incorporate different structural alerts, or a combination of structural alerts and physicochemical and/or toxicokinetic rules relevant to a specific toxicological endpoint.\u003c/p\u003e \u003cp\u003eThe overall aim of this study was to assess the performance of the in silico profilers provided in the OECD QSAR Toolbox for reliability for identifying chemical analogues for category formation in a number of high-quality databases on mutagenicity, carcinogenicity, and skin sensitisation. The study also aimed to identify the reasons for any limitations in the performance of the profilers, and propose ways to improve their overall accuracy. The results showed that whilst some structural alerts are fit-for-purpose as such within the acceptable limits, others need refinement or a consideration for their possible exclusion from the profiler. Such refinements are imperative for a reliable use of the profilers in read-across and grouping/categorisation for classification, labelling and risk assessment of chemicals.\u003c/p\u003e","manuscriptTitle":"Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-13 11:43:56","doi":"10.21203/rs.3.rs-3969238/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-06-12T07:51:58+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-07T20:49:18+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"316632478371930951185773133457233827229","date":"2024-05-29T20:30:06+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-02T17:58:46+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"2260054f-86c6-4ee9-818f-5c3338ba854c","date":"2024-04-30T08:10:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"5cb685eb-75c0-42d1-891b-f3802c77975b","date":"2024-04-25T06:47:50+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-03-15T18:45:54+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-03-12T02:31:11+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-03-12T00:54:41+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-03-12T00:50:35+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-02-19T07:07:44+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e47016e0-cbce-410f-91c7-2a49bacd5975","owner":[],"postedDate":"March 13th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":29378629,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":29378630,"name":"Biological sciences/Computational biology and bioinformatics/Computational models"}],"tags":[],"updatedAt":"2024-08-12T16:10:53+00:00","versionOfRecord":{"articleIdentity":"rs-3969238","link":"https://doi.org/10.1038/s41598-024-69157-1","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2024-08-07 15:58:17","publishedOnDateReadable":"August 7th, 2024"},"versionCreatedAt":"2024-03-13 11:43:56","video":"","vorDoi":"10.1038/s41598-024-69157-1","vorDoiUrl":"https://doi.org/10.1038/s41598-024-69157-1","workflowStages":[]},"version":"v1","identity":"rs-3969238","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3969238","identity":"rs-3969238","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}