A Step Towards Generalisability: Training a Machine Learning Scoring Function for Structure-Based Virtual Screening

BackgroundThe diagnostic value of clinical and laboratory features to differentiate between malignant pleural effusion (MPE) and benign pleural effusion (BPE) has not yet been established.ObjectivesThe present study aimed to develop and validate the diagnostic accuracy of a scoring system based on a nomogram to distinguish MPE from BPE.MethodsA total of 1,239 eligible patients with PE were recruited in this study and randomly divided into a training set and an internal validation set at a ratio of 7:3. Logistic regression analysis was performed in the training set, and a nomogram was developed using selected predictors. The diagnostic accuracy of an innovative scoring system based on the nomogram was established and validated in the training, internal validation, and external validation sets (n = 217). The discriminatory power and the calibration and clinical values of the prediction model were evaluated.ResultsSeven variables [effusion carcinoembryonic antigen (CEA), effusion adenosine deaminase (ADA), erythrocyte sedimentation rate (ESR), PE/serum CEA ratio (CEA ratio), effusion carbohydrate antigen 19-9 (CA19-9), effusion cytokeratin 19 fragment (CYFRA 21-1), and serum lactate dehydrogenase (LDH)/effusion ADA ratio (cancer ratio, CR)] were validated and used to develop a nomogram. The prediction model showed both good discrimination and calibration capabilities for all sets. A scoring system was established based on the nomogram scores to distinguish MPE from BPE. The scoring system showed favorable diagnostic performance in the training set [area under the curve (AUC) = 0.955, 95% confidence interval (CI) = 0.942–0.968], the internal validation set (AUC = 0.952, 95% CI = 0.932–0.973), and the external validation set (AUC = 0.973, 95% CI = 0.956–0.990). In addition, the scoring system achieved satisfactory discriminative abilities at separating lung cancer-associated MPE from tuberculous pleurisy effusion (TPE) in the combined training and validation sets.ConclusionsThe present study developed and validated a scoring system based on seven parameters. The scoring system exhibited a reliable diagnostic performance in distinguishing MPE from BPE and might guide clinical decision-making.

Fourteen popular scoring functions, i.e., X-Score, DrugScore, five scoring functions in the Sybyl software (D-Score, PMF-Score, G-Score, ChemScore, and F-Score), four scoring functions in the Cerius2 software (LigScore, PLP, PMF, and LUDI), two scoring functions in the GOLD program (GoldScore and ChemScore), and HINT, were tested on the refined set of the PDBbind database, a set of 800 diverse protein-ligand complexes with high-resolution crystal structures and experimentally determined Ki or Kd values. The focus of our study was to assess the ability of these scoring functions to predict binding affinities based on the experimentally determined high-resolution crystal structures of proteins in complex with their ligands. The quantitative correlation between the binding scores produced by each scoring function and the known binding constants of the 800 complexes was computed. X-Score, DrugScore, Sybyl::ChemScore, and Cerius2::PLP provided better correlations than the other scoring functions with standard deviations of 1.8-2.0 log units. These four scoring functions were also found to be robust enough to carry out computation directly on unaltered crystal structures. To examine how well scoring functions predict the binding affinities for ligands bound to the same target protein, the performance of these 14 scoring functions were evaluated on three subsets of protein-ligand complexes from the test set: HIV-1 protease complexes (82 entries), trypsin complexes (45 entries), and carbonic anhydrase II complexes (40 entries). Although the results for the HIV-1 protease subset are less than desirable, several scoring functions are able to satisfactorily predict the binding affinities for the trypsin and the carbonic anhydrase II subsets with standard deviation as low as 1.0 log unit (corresponding to 1.3-1.4 kcal/mol at room temperature). Our results demonstrate the strengths as well as the weaknesses of current scoring functions for binding affinity prediction.

Molecular docking, scoring, and virtual screening play an increasingly important role in computer-aided drug discovery. Scoring functions (SFs) are typically employed to predict the binding conformation (docking task), binding affinity (scoring task), and binary activity level (screening task) of ligands against a critical protein target in a disease's pathway. In most molecular docking software packages available today, a generic binding affinity-based (BA-based) SF is invoked for all three tasks to solve three different, but related, prediction problems. The limited predictive accuracies of such SFs in these three tasks has been a major roadblock toward cost-effective drug discovery. Therefore, in this work, we develop BT-Score, an ensemble machine-learning (ML) SF of boosted decision trees and thousands of predictive descriptors to estimate BA. BT-Score reproduced BA of out-of-sample test complexes with correlation of 0.825. Even with this high accuracy in the scoring task, we demonstrate that the docking and screening performance of BT-Score and other BA-based SFs is far from ideal. This has motivated us to build two task-specific ML SFs for the docking and screening problems. We propose BT-Dock, a boosted-tree ensemble model trained on a large number of native and computer-generated ligand conformations and optimized to predict binding poses explicitly. This model has shown an average improvement of 25% over its BA-based counterparts in different ligand pose prediction scenarios. Similar improvement has also been obtained by our screening-based SF, BT-Screen, which directly models the ligand activity labeling task as a classification problem. BT-Screen is trained on thousands of active and inactive protein-ligand complexes to optimize it for finding real actives from databases of ligands not seen in its training set. In addition to the three task-specific SFs, we propose a novel multi-task deep neural network (MT-Net) that is trained on data from the three tasks to simultaneously predict binding poses, affinities, and activity levels. We show that the performance of MT-Net is superior to conventional SFs and on a par with or better than models based on single-task neural networks.

BACKGROUND: Recently, significant efforts have been undertaken to find potential noninvasive biomarkers for predicting the response of locally advanced rectal cancer to neoadjuvant chemoradiotherapy. AIM: To assess the texture characteristics of locally advanced rectal cancer in primary T2-weighted imaging (T2-WI) as a potential predictor for the efficacy of standard neoadjuvant chemoradiotherapy and develop a prediction system for the efficacy of neoadjuvant chemoradiotherapy based on them. MATERIALS AND METHODS: The retrospective study enrolled 82 patients with locally advanced rectal cancer who received combination treatment with neoadjuvant chemoradiotherapy. Patient data were divided into the training (n=58) and control (n=24) sets. For texture analysis, primary high-resolution T2-WI at the level of the tumor center, oriented perpendicular to the intestinal wall, was used. The texture analysis was performed by second-order statistics based on the gray-level co-occurrence matrices using MAZDA ver. 4.6 featuring the calculation of 11 texture parameters. In the training set, based on the morphological assessment of surgical specimens, significantly different texture analysis parameters were found for two groups of patients: neoadjuvant chemoradiotherapy responders (good prognosis group) and nonresponders (poor prognosis group). Accordingly, a scoring system was created for assessing the efficacy of neoadjuvant chemoradiotherapy. The system was tested on the control set, and diagnostic efficacy parameters were determined. RESULTS: In the training set, the good and poor prognosis groups differed significantly in five texture parameters: AngScMom (p=0.021), SumofSqs (p=0.003), SumEntrp (p=0.003), Entropy (p=0.038), and SumVarnc (p=0.015), for which the cutoff points were found. These parameters were applied to create the scoring system (excluding the Entropy parameter, which had a strong direct correlation with SumEntrp and the lowest area under the curve, and SumofSqs, which had low reproducibility). The diagnostic efficiency of the scoring system for predicting the response had sensitivity, specificity, positive-predictive value, and negative- predictive value of 72%, 69%, 70%, and 71% for the training set and 80%, 64%, 62%, and 82% for the control set, respectively. The areas under the ROC curve were 0.77 and 0.72 for the training and control sets, respectively. CONCLUSIONS: Texture analysis of the primary T2-WI of tumors in patients with locally advanced rectal cancer allows for predicting the efficacy of neoadjuvant chemoradiotherapy with moderate diagnostic efficiency. The results suggest good prospects for further research in this area.

We present DrugScore2018, a new version of the knowledge-based scoring function DrugScore, which builds upon the same formalism used to derive DrugScore but exploits a training data set of nearly 40 000 X-ray complex structures, a highly diverse and the, by far, largest data set ever used for such an endeavor. About 2.5 times as many pair potentials than before now have a data basis required to yield smooth potentials, and pair potentials could now be derived for eight more atom types, including interactions involving halogen atoms and metal ions highly relevant for medicinal chemistry. Probing for dependence on training data set size and quality, we show that DrugScore2018 potentials are converged. We evaluated DrugScore2018 in comprehensive scoring, ranking, docking, and screening tests on the CASF-2013 data set, allowing for a comparison with >30 other scoring functions. There, DrugScore2018 showed similar or improved performance in all aspects when compared to either DrugScore, DrugScoreCSD, or DSX and was, overall, the scoring function showing the most consistently good performance in scoring, ranking, and docking tests. Applying DrugScore2018 as objective function in AutoDock3 in a large-scale docking trial, using 4056 protein-ligand complexes from PDBbind 2016, reproduced a docked pose to within 2 Å RMSD to the crystal structure in >75% of all dockings. These results are remarkable as the DrugScore2018 potentials were derived from crystallographic information only, without any further adaptation using binding affinity or docking decoy data. DrugScore2018 should thus be a competitive scoring and objective function for structure-based ligand design purposes.

(1) Background: In silico approaches to rational drug design are among the fastest evolving ones. Depending on the available structural information for the biomacromolecule and the small molecule, the in silico methods are classified as ligand- and structure-based. The latter predict ligand–receptor binding using 3D structures of both molecules, whose computational simulation is referred to as molecular docking. It aims at estimating the binding affinity (approximated by scoring function) and the ligand binding pose in the receptor’s active site, which postulates a key role of the scoring functions in molecular docking algorithms. This study focuses on the performance of different types of scoring functions implemented in molecular modelling software packages. (2) Methods: An InterCriteria analysis (ICrA) was applied to assess the performance of the scoring functions available in MOE, GOLD, SeeSAR, and AutoDock Vina software platforms. The InterCriteria analysis was developed to distinguish possible relations between pairs of criteria when multiple objects are considered. All 12 investigated scoring functions were tested by docking a set of protease inhibitors in the binding sites of two protein targets. The dataset consisted of 88 benzamidine-type compounds with experimentally measured inhibitory constants for thrombin and trypsin, which allows for the objective assessment of the scoring functions performance. The results generated by the molecular docking were subjected to ICrA in order to analyze both docking energies as approximations of the binding affinities and RMSDs (root-mean-square deviation) as measures of the experimental binding pose proximity between the compounds and the co-crystalized ligand, based on the atoms in the common scaffold. (3) Results: The results obtained for the best poses, the average of the best 5 or 30 poses retained after docking, were analyzed. A comparison with the experimentally observed inhibitory effects was also performed. The InterCriteria analysis application confirms that the performance of the scoring functions for the same dataset of ligands depends on the studied protein. The analysis reveals that none of the studied scoring functions is a good predictor of the compounds’ binding affinities for the considered protein targets. (4) Conclusion: In terms of this analysis, the investigated scoring functions do not produce equivalent results, which suggests the necessity for their combined use in consensus docking studies.

Angiographic no-reflow is associated with poor outcomes in patients with ST-segment elevation myocardial infarction (STEMI). We sought to develop and validate a score system to predict angiographic no-reflow in primary percutaneous coronary intervention (PCI). ST-segment elevation myocardial infarction patients undergoing primary PCI were consecutively enrolled and were randomly divided into the training and validation set. Angiographic no-reflow was defined as thrombolysis in myocardial infarction (TIMI) flow grade 0 to 2 after PCI. In the training set, independent predictors were identified by logistic regression analysis, and a score system (PredIction of Angiographic NO-reflow, the PIANO score) was constructed based on the β-coefficient of each variable. The established model was evaluated for discrimination and calibration. Angiographic no-reflow occurred in 362 (17.8%) of 2036 patients. Age ≥70years, absence of pre-infarction angina, total ischaemic time ≥4h, left anterior descending as culprit artery, pre-PCI TIMI flow grade ≤1 and pre-PCI TIMI thrombus score ≥4 were independent predictors of angiographic no-reflow. The PIANO score ranged from 0 to 14 points, yielding a concordance index of 0.857 (95% confidence interval: 0.833 to 0.880), with good calibration. In the high-risk (≥8 points) group, the probability of angiographic no-reflow phenomenon was 38.7%, while it was only 4.8% in the low-risk (<8 points) group. The score system performed well in the validation set. We establish and validate a score system based on six clinical variables to predict angiographic no-reflow in STEMI patients undergoing primary PCI, which may help choose the optimal individual treatment strategy.

Proposal and Validation of a Basic Progression Scoring System for Patients with Skull Base Chordoma

Structure-based drug design (SBDD) is an important in silico technique, used for the identification of enzyme inhibitors. Acetylcholinesterase (AChE), obtained from Electrophorus electricus (ee), is widely used for the screening of AChE inhibitors. It shares structural homology with the AChE of human and other organisms. Till date, the three-dimensional crystal structure of enzyme from ee is not available that makes it challenging to use the SBDD approach for the identification of inhibitors. A homology model was developed for eeAChE in the present study, followed by its structural refinement through energy minimisation. The docking protocol was developed using a grid dimension of 84 × 66 × 72 and grid point spacing of 0.375 Å for eeAChE. The protocol was validated by redocking a set of co-crystallised inhibitors obtained from mouse AChE, and their interaction profiles were compared. The results indicated a poor performance of the Autodock scoring function. Hence, a batch of machine learning-based scoring functions were developed. The validation results displayed an accuracy of 81.68 ± 1.73% and 82.92 ± 3.05% for binary and multiclass classification scoring function, respectively. The regression-based scoring function produced [Formula: see text] and [Formula: see text] values of 0.94, 0.635 and 0.634, respectively.

Background and AimPost‐endoscopic retrograde cholangiopancreatography (ERCP) pancreatitis (PEP) is the most serious adverse event of ERCP. Therefore, it is important to identify high‐risk patients who require prophylactic measures. The aim of this study was to develop a practical prediction model for PEP that identifies high‐risk patients.MethodsPatients who underwent ERCP at three tertiary hospitals between April 2010 and September 2012 were enrolled. The dataset was divided into a training set (two centers) and validation set (one center). Using the training set, univariable and multivariable analyses were performed to identify predictive factors for PEP. We constructed a scoring system and evaluated reproducibility using the validation set.ResultsA total of 2719 ERCPs were analyzed. PEP developed in 133 cases (4.9%). Risk factors (adjusted odds ratios [OR]) in the training set were a history of PEP (OR: 4.2), intact papilla (OR: 2.4), difficult cannulation (OR: 1.9), pancreatic guidewire‐assisted biliary cannulation (OR: 2.2), pancreatic injection (OR: 2.1), pancreatic intraductal ultrasonography (IDUS)/sampling from the pancreatic duct (OR: 2.2), and biliary IDUS/sampling from the biliary tract (OR: 2.8). A scoring system was constructed using these seven clinical variables. The areas under the receiver‐operating characteristic curve of this model were 0.799 in the training set and 0.791 in the validation set. In the high‐risk group at a score of 3 or higher, the incidence of PEP was 13.4%, and all severe/fatal cases were in the high‐risk group.ConclusionsThis scoring system helps to predict each patient's risk and select preventive measures.

Development and validation of a novel scoring system developed from a nomogram to identify malignant pleural effusion.

<h3>Background</h3> Endoscopy is increasingly important for the evaluation of patients with ulcerative colitis (UC). However, there were considerable differences in endoscopic assessment because of the endoscopists’ training experience, not only that, existing endoscopy scoring methods cannot reflect the inflammation’s details of the full-length endoscopic video; therefore, we sought to develop an automatic scoring system using deep learning technology for consistent and objective of endoscopic images and videos from patients with UC. <h3>Methods</h3> ResNet50 pre-trained on ImageNet was selected as our CNN main framework. We collected 5,875 endoscopic images and 16 full-length videos from patients with UC who underwent colonoscopy from January 2017 through March 2021as the dataset. The scoring system was trained by these images and could predict the Mayo score and UCEIS score by the feature it has learned on the training set. In addition, by using the predicted outcome, the CAD could show a ribbon in the shape of the colon which can reflect the conditions of the inflammation (location, severity). The scoring system is composed of clarity cleaning, similarity judgement and score prediction networks. This corresponding relation which combines the exiting endoscopy scores prediction, helps the system to output a new score with weighted calculation, as well as the lesions’ location and severity ratio. <h3>Results</h3> Two clinical reviewed our consistency statistics. The primary evaluation metric we used is quadratic weighted kappa (QWK). The trained CNN got high levels with 90.2% accuracy in the Mayo-scored task, while the QWK coefficient was 0.906 (95%CI, 0.880-0.933). The metrics of UCEIS-scored task were inspiring of which accuracy were 90.7%, 84.6%, 77.7%, QWK coefficient were 0.887 (95%CI, 0.860-0.915), 0.876 (95%CI, 0.844-0.907), 0.816 (95%CI, 0.739-0.892) for vascular pattern, erosions-and- ulcers and bleeding. Meanwhile, the CNN identified 16 full-length endoscopy videos and evaluated 1,360 local information from 100 intestine segments. The results of these localized intestinal tissue marked by the scored system were distributed in four ranges, and the quantities were 772(0.0-0.4), 124(0.72-2.35), 206(3.2-4.0), 258(5.2-8.4). It performed desirable compared with the clinical diagnosis. <h3>Conclusions</h3> The results of objective evaluations of UC based on this scoring system appeared acceptable. Compared with the previous scored methods (only refer to the most severe lesion and ignored other mild parts), this work is of clinical significance and will help clinicians by supplying a good capacity in identifying endoscopy data and stimulating the colon’s inflammatory environment.

In high-throughput proteomics the development of computational methods and novel experimental strategies often rely on each other. In certain areas, mass spectrometry methods for data acquisition are ahead of computational methods to interpret the resulting tandem mass spectra. Particularly, although there are numerous situations in which a mixture tandem mass spectrum can contain fragment ions from two or more peptides, nearly all database search tools still make the assumption that each tandem mass spectrum comes from one peptide. Common examples include mixture spectra from co-eluting peptides in complex samples, spectra generated from data-independent acquisition methods, and spectra from peptides with complex post-translational modifications. We propose a new database search tool (MixDB) that is able to identify mixture tandem mass spectra from more than one peptide. We show that peptides can be reliably identified with up to 95% accuracy from mixture spectra while considering only a 0.01% of all possible peptide pairs (four orders of magnitude speedup). Comparison with current database search methods indicates that our approach has better or comparable sensitivity and precision at identifying single-peptide spectra while simultaneously being able to identify 38% more peptides from mixture spectra at significantly higher precision.

Automated docking is one of the most important tools for structure-based drug design that allows prediction of ligand binding poses and also provides an estimate of how well small molecules fit in the binding site of a protein. A new scoring function based on AutoDock and AutoDock Vina has been introduced. The new hybrid scoring function is a linear combination of the two scoring function components derived from a multiple linear regression fitting procedure. The scoring function was built on a training set of 2412 protein-ligand complexes from pdbbind database (www.pdbbind.org.cn, version 2012). A test set of 313 complexes that appeared in the 2013 version was used for validation purposes. The new hybrid scoring function performed better than the original functions, both on training and test sets of protein-ligand complexes, as measured by the non-parametric Pearson correlation coefficient, R, mean absolute error (MAE), and root-mean-square error (RMSE) between the experimental binding affinities and the docking scores. The function also gave one of the best results among more than 20 scoring functions tested on the core set of the pdbbind database. The new AutoDock hybrid scoring function will be implemented in modified version of AutoDock.

BackgroundNasopharyngeal carcinoma (NPC), an endemic malignancy in Southeast Asia and southern China, necessitates precise delineation of the gross tumor volume (GTV) for radiotherapy to optimize patient prognosis. Positron emission tomography (PET) imaging offers valuable metabolic insights to guide radiotherapy planning but is hampered by manual GTV segmentation, which is labor-intensive and prone to inter- and intra-observer variability. Conventional segmentation methods are inherently limited, and comparative evaluations of deep learning (DL) frameworks for PET-based primary NPC lesion segmentation remain scarce. Thus, this study aimed to optimize the delineation of primary NPC radiotherapy target volumes by leveraging PET-image-based DL segmentation technology, with the goal of enhancing both the precision and efficiency of lesion delineation.MethodsResearchers retrospectively collected PET imaging data from 212 NPC patients at the Affiliated Hospital of Guangdong Medical University. The patients were randomly divided into training (170 patients) and testing (42 patients) datasets. A radiation oncologist and a nuclear medicine physician collaboratively delineated the cancer lesion boundaries through consensus. After data preprocessing, three DL models (Res-Unet, Nn-Unet, and Nn-Former) were used to automatically segment the lesions. Training was based on the training dataset. During evaluation, the models’ segmentation performance was comprehensively assessed using Dice similarity coefficient (DSC) and Hausdorff distance (HD). A visual analysis of the results was also conducted to intuitively understand the models’ segmentation capabilities.ResultsIn the training set, Nn-Unet achieved the highest DSC of 0.869, whereas in the testing set, its DSC was 0.833. The DSC for Nn-Former hovered around 0.8 in both the training and testing sets. In contrast, Res-Unet demonstrated the lowest DSC values among the three models, particularly in the testing set (DSC =0.794). Statistical analysis revealed that the DSC of Nn-Unet was significantly higher than that of both Res-Unet and Nn-Former in the training set (P<0.01 for both comparisons). Regarding the 95% HD and average surface distance (ASD) values, Nn-Unet outperformed the other two models in both the training and testing sets. However, Res-Unet exhibited a much higher HD value in the testing set compared to the training set. The Loss curves for all models gradually decreased as training progressed, indicating that the models were learning relevant features of NPC lesions. The final Loss value for Res-Unet was approximately 0.7, whereas Nn-Unet and Nn-Former had final Loss values below 0.8.ConclusionsThe PET-image-based automatic segmentation model for the primary tumor of NPC established in this study demonstrates the clinical potential of Nn-Unet in primary lesion of NPC segmentation tasks.