PNS15 PRAGMATIC ARTIFICIAL INTELLIGENCE-BASED REFERENCE SCREENING IN SYSTEMATIC REVEIWS. ARE TWO ROBOTS BETTER THAN ONE?

Semantic Role Labeling (SRL) is a crucial task in Natural Language Processing (NLP) that plays a vital role in extracting meaningful information from text. In the fields of economics and cybersecurity, accurately identifying and analyzing semantic roles within text is crucial due to the rapid increase in the amount and complexity of textual information. This abstract examines the significant role of SRL and its application in economic and cybersecurity contexts. It discusses the state-of-the-art NLP classifiers used for this purpose. By examining the relationship between language processing and these important areas, we aim to emphasize the importance of SRL in extracting useful information and improving decision-making in a constantly changing digital environment. The aim of the findings is to emphasize the significance of SRL in extracting valuable insights from text, as it serves as a fundamental technique in NLP. It is utilized in the economic context to analyze financial reports, news articles, and economic texts. It assists in decision-making and market analysis. It aids in identifying important participants, actions, and objects in economic discourse, leading to better decision-making and market analysis. In the field of cybersecurity, SRL assists parse and comprehending text data related to security, enabling faster responses to threats. NLP classifiers and machine learning models utilize SRL to automate the analysis of large amounts of text. These techniques are practically significant as they improve the ability to extract actionable insights, assess risks, and make informed decisions by organizing unstructured text data. The process of determining relevant information from a large corpus of data requires an optimal methodological basis. Relevant textual data is collected from sources such as financial reports, news articles, or cybersecurity incident reports. Textual data is cleaned, tokenized, and tagged with part-of-speech labels in preparation for NLP analysis. Human annotators label semantic roles in the text, identifying actors, actions, and objects. This creates a dataset that can be used to train classifiers. NLP classifiers, including machine learning models, are trained using annotated datasets to identify semantic roles. The accuracy and performance of the trained classifiers are evaluated using various metrics. NLP classifiers are used to automatically identify and label semantic roles in new, unseen textual data. The output helps extract insights, such as market trends or security threats, depending on the specific field. Researchers improve classifier models by iteratively training and applying them to increase accuracy.

BackgroundModern patient electronic health records form a core part of primary care; they contain both clinical codes and free text entered by the clinician. Natural language processing (NLP) could be...

2043 Background: Electronic health records (EHR) are used for retrospective cancer outcomes analysis. Sites and timing of recurrence are not captured in structured EHR data. Novel computerized methods are necessary to use unstructured longitudinal EHR data for large scale studies. Methods: We previously developed a neural network-based NLP algorithm to identify no recurrence vs. metastatic recurrence cases by analyzing physician notes, pathology and radiology reports in Stanford’s breast cancer database, Oncoshare (Cohort A). To validate this algorithm for local vs. distant recurrence, we identified a distinct Oncoshare cohort (Cohort B). Cases were manually curated for longitudinal development of local or distant recurrence and metastatic sites. A two-sided t-test was used to compare mean probabilities between local and distant recurrence cases. Next, we combined cases in Cohorts A and B to train and validate a novel NLP classifier that identifies metastatic site. The combined cohort was randomly divided into training and validation sets. Sensitivity and specificity were calculated for the NLP algorithm’s ability to detect metastatic sites compared to manual curation. Results: In Cohort B: 350 metastatic cases were identified. Mean probability for local and distant recurrence was 0.43 and 0.79, respectively and differed significantly for patients with local vs. distant recurrence (p<0.01). In Cohorts A and B: 632 metastatic cases were used for determination of sites. Sensitivity and specificity were highest for detection of peritoneal metastasis followed by liver, lung, skin, bone and central nervous system (table). Conclusions: This NLP algorithm is a scalable tool that uses unstructured EHR data to capture breast cancer recurrence, distinguishing local from distant recurrence and identifying metastatic site. This method may facilitate analysis of large datasets and correlation of outcomes with metastatic site. [Table: see text]

In clinical spine surgery research, manually reviewing surgical forms to categorize patients by their surgical characteristics is a crucial yet time-consuming task. Natural language processing (NLP) is a machine learning tool used to adaptively parse and categorize important features from text. These systems function by training on a large, labeled data set in which feature importance is learned prior to encountering a previously unseen data set. The authors aimed to design an NLP classifier for surgical information that can review consent forms and automatically classify patients by the surgical procedure performed. Thirteen thousand two hundred sixty-eight patients who underwent 15,227 surgeries from January 1, 2012, to December 31, 2022, at a single institution were initially considered for inclusion. From these surgeries, 12,239 consent forms were classified based on the Current Procedural Terminology (CPT) code, categorizing them into 7 of the most frequently performed spine surgeries at this institution. This labeled data set was split 80%/20% into train and test subsets, respectively. The NLP classifier was then trained and the results demonstrated its performance on the test data set using CPT codes to determine accuracy. This NLP surgical classifier had an overall weighted accuracy rate of 91% for sorting consents into correct surgical categories. Anterior cervical discectomy and fusion had the highest positive predictive value (PPV; 96.8%), whereas lumbar microdiscectomy had the lowest PPV in the testing data (85.0%). Sensitivity was highest for lumbar laminectomy and fusion (96.7%) and lowest for the least common operation, cervical posterior foraminotomy (58.3%). Negative predictive value and specificity were > 95% for all surgical categories. Utilizing NLP for text classification drastically improves the efficiency of classifying surgical procedures for research purposes. The ability to quickly classify surgical data can be significantly beneficial to institutions without a large database or substantial data review capabilities, as well as for trainees to track surgical experience, or practicing surgeons to evaluate and analyze their surgical volume. Additionally, the capability to quickly and accurately recognize the type of surgery will facilitate the extraction of new insights from the correlations between surgical interventions and patient outcomes. As the database of surgical information grows from this institution and others in spine surgery, the accuracy, usability, and applications of this model will continue to increase.

Abstract #3001 Purpose
 Pathology reports contain extensive research information that is inaccessible except through costly and time consuming chart reviews. This is due to the fact that pathology reports are recorded as semi-structured prose with critically important descriptive text intended for human interpretation. Key challenges for processing this data include interpreting multiple methods of describing the same finding, and subsequently aggregating the findings of multiple reports into episodes of care. Investigators tested NLP techniques in the processing of pathology reports into structured data and episodes of care, allowing for the rapid identification and epidemiologic modeling of high-risk breast lesions.
 Methods
 Using state-of-the-art NLP software (ClearForest, A Thomson Reuters Company, Waltham, MA), breast pathology reports stored as text files were processed into a structured electronic database using these steps: 1) identification of diagnosis of interest (i.e. high risk lesions, cancer), 2) use of NLP to identify all terms and phrases used to report each finding (e.g. atypical hyperplasia, hyperplasia with atypia), 3) grouping of relevant terms into categories, 4) identification of categories occurring in each patient report, and 5) grouping of patient reports into episodes of care (defined as all reports within 6 months of an initial diagnosis).
 Results
 Under IRB approval, 27,931 breast pathology reports from Massachusetts General Hospital in 16,208 patients seen between 1990-2007 were analyzed. The results were compared against manually reviewed pathology reports for quality control. For DCIS diagnoses, the initial error rate for both the NLP process and the manual process was 2%. The NLP process was then re-tuned using the identified discrepancies which reduced the error rate to zero. Using the refined model, we identified 1) patients with atypical lesions (atypical ductal hyperplasia (ADH), severe ADH, atypical lobular hyperplasia (ALH), and lobular carcinoma in situ (LCIS)) without prior or concurrent cancer, and 2) patients who developed cancer greater than 6 months post diagnosis.
 
 Conclusion
 This process successfully identified high-risk diagnoses that were otherwise relatively inaccessible, and appears to match the accuracy of a human research associate. The results of this first implementation are promising and will be further validated over time. In the future, this approach can be applied to other medical reports and diseases. NLP has significant potential to decrease the cost of research and for improving patient care. Citation Information: Cancer Res 2009;69(2 Suppl):Abstract nr 3001.

This paper examines the role of human factors in the design of automobiles. A prime objective of our human factors profession is to improve the design of machines, thereby benefiting users in terms of comfort, convenience, operating speeds, accuracy and safety. Although the purpose of an automotive human factors program may be to achieve all of these objectives by improving vehicle design, the mechanisms for doing so probably cannot be discovered by focusing research attention on the vehicle element of the driver/vehicle/road system. In fact, the nonvehicle parts of this system are probably by far the most productive topics for future human factors research. The abilities of drivers, their limitations, and the tasks imposed upon them by the traffic environment should indicate how vehicles can be designed to best serve the drivers' needs. After twenty years of automotive study, the human factors research community is surprisingly unprepared to participate in vehicle design projects. The vehicle has too often ended up the subject of human factors research and researchers have been faced with the job of finding ways to improve the vehicle or a vehicle component without knowing enough about the intended user or the job the user must perform. The research community has only rudimentary and often incomplete background information about drivers and their traffic environments. The meager data base which is available suggests that traditional empirical approaches for evaluating machine design may be too cumbersome and time consuming to keep pace with other aspects of automotive technological evolution. The tradition of developing alternative versions of hardware and subjecting the alternatives to human performance tests may not be a viable methodology in the future. A look at the total automotive system shows why. Drivers in the United States accumulate about 1.6 trillion miles of travel each year. During the year, a typical driver makes over 60,000 discrete control operations not counting steering wheel movements. The immensity of the automotive system means that very small driver error rates in control usage quickly accumulate into large numbers of error events nationwide. The best information available suggests that the D. S. driving public uses their turn signals 854 billion times a year. This amounts to a nationwide rate of 27,000 times per second. If the generic human error rate in using the turn signal can assumed to be one error per 1000 operations, then turn signal errors are being made at the rate of 27 per second nationwide. Human factors research has tended to avoid error rate as a principal measure of performance in research programs. The reason becomes apparent when the number of tests which must be conducted to detect changes in rare events such as turn signal errors is computed. If two turn signal designs are to be compared and the researcher wants to be able to detect with 95 percent certainty (at the 5% level of significance) that the error rate has been cut in half by one of the two designs, then a large experiment is required. At a base human error rate of 1 per thousand, about 130,000 observations must be conducted to reliably detect the desired reduction in errors. If the base human error rate is only 1 in 100, then only 13,000 observations will be needed. Unfortunately, information on the frequency of driver control usage is sketchy, and data on driver error rates when using controls under the natural loading of the driving task is all but nonexistent. Other measures of human performance, such as speed of operation and accident involvement rates, have limited application in automotive design for reasons that are discussed in this paper. Some of the data bases which have been accumulated for human factors evaluations by Ford Motor Company are described in this paper. It is concluded that, if the human factors profession is to keep pace with automotive technological evolution, more research effort is going to have to be devoted to the study of drivers and driving-environment factors. For the sake of research efficiency, human factors principles and systems models which can be reliably generalized across vehicle designs must be developed. Several systems models that are under development at Ford are briefly described.

IntroductionIn recent years, large language models (LLMs) have emerged as powerful tools in natural language processing, presenting diverse opportunities across various sectors. In this context, the Agency for Health Quality and Assessment of Catalonia (AQuAS) is actively investigating how LLMs can enhance the development of health technology assessment (HTA) reports.MethodsTo assess the potential of LLMs in the development of HTA reports, our initial step involved a comprehensive review of technical literature to understand the functionalities of existing LLM tools. This effort was followed by a systematic identification of specific HTA report development tasks that these models could potentially facilitate. We then rigorously evaluated the performance of these tools in executing the delineated tasks, with a focus on meticulously recording the applied prompts and areas of high and low effectiveness, alongside any encountered errors and identified risks.ResultsAmong a range of commercial tools, ChatGPT and Microsoft Copilot were selected for exploration due to their accessibility, ease of use, and versatility. Both tools demonstrated acceptable performance in certain tasks (e.g., data extraction), while their efficacy was notably lower in others (e.g., information search). Additionally, a range of risks requiring mitigation were identified (e.g., hallucination or authority bias). Consequently, we drafted a proposal for an internal guide with directives for the technical staff of the HTA unit on utilizing these two tools. Additionally, we constituted a working group.ConclusionsLLMs have emerged as promising tools in the field of HTA. Over the past months, AQuAS has been investigating the potential of these models to improve the HTA report development process, targeting enhanced efficiency and improved quality. This exploration has led to the identification of numerous opportunities and associated risks within this innovative application.

As part of an overall effort to upgrade and streamline methodologies for safety analyses of nonreactor nuclear facilities at the Savannah River Site (SRS), a human error data base has been developed and is presented in this report. The data base fulfills several needs of risk analysts supporting safety analysis report (SAR) development. First, it provides a single source for probabilities or rates for a wide variety of human errors associated with the SRS nonreactor nuclear facilities. Second, it provides a documented basis for human error probabilities or rates. And finally, it provides actual SRS-specific human error data to support many of the error probabilities or rates. Use of a single, documented reference source for human errors, supported by SRS-specific human error data, will improve the consistency and accuracy of human error modeling by SRS risk analysts. It is envisioned that SRS risk analysts will use this report as both a guide to identifying the types of human errors that may need to be included in risk models such as fault and event trees, and as a source for human error probabilities or rates. For each human error in this report, ffime different mean probabilities or rates are presented to cover a wide range of conditions and influencing factors. The ask analysts must decide which mean value is most appropriate for each particular application. If other types of human errors are needed for the risk models, the analyst must use other sources. Finally, if human enors are dominant in the quantified risk models (based on the values obtained fmm this report), then it may be appropriate to perform detailed human reliability analyses (HRAS) for the dominant events. This document does not provide guidance for such refined HRAS; in such cases experienced human reliability analysts should be involved.

Natural Language Processing and Machine Learning Classification Model for Injury Mechanism in Trauma.

The cognitive approach to psychotherapy aims to change patients’ maladaptive schemas, that is, overly negative views on themselves, the world, or the future. To obtain awareness of these views, they record their thought processes in situations that caused pathogenic emotional responses. The schemas underlying such thought records have, thus far, been largely manually identified. Using recent advances in natural language processing, we take this one step further by automatically extracting schemas from thought records. To this end, we asked 320 healthy participants on Amazon Mechanical Turk to each complete five thought records consisting of several utterances reflecting cognitive processes. Agreement between two raters on manually scoring the utterances with respect to how much they reflect each schema was substantial (Cohen’s κ = 0.79). Natural language processing software pretrained on all English Wikipedia articles from 2014 (GLoVE embeddings) was used to represent words and utterances, which were then mapped to schemas using k-nearest neighbors algorithms, support vector machines, and recurrent neural networks. For the more frequently occurring schemas, all algorithms were able to leverage linguistic patterns. For example, the scores assigned to the Competence schema by the algorithms correlated with the manually assigned scores with Spearman correlations ranging between 0.64 and 0.76. For six of the nine schemas, a set of recurrent neural networks trained separately for each of the schemas outperformed the other algorithms. We present our results here as a benchmark solution, since we conducted this research to explore the possibility of automatically processing qualitative mental health data and did not aim to achieve optimal performance with any of the explored models. The dataset of 1600 thought records comprising 5747 utterances is published together with this article for researchers and machine learning enthusiasts to improve upon our outcomes. Based on our promising results, we see further opportunities for using free-text input and subsequent natural language processing in other common therapeutic tools, such as ecological momentary assessments, automated case conceptualizations, and, more generally, as an alternative to mental health scales.

Support vector machines (SVMs) have been successfully applied to classification problems. The difficulty in selecting the most effective error penalty has been partly resolved with /spl nu/-SVM. However, the use of uneven training class sizes, which occurs frequently with target detection problems, results in machines with biases towards the class with the larger training set. We propose an extended /spl nu/-SVM to counter the effects of the unbalanced training class sizes. The resulting dual /spl nu/-SVM provides the facility to counter these effects, as well as to adjust the error penalties of each class separately. The parameter /spl nu/ of each class provides a lower bound to the fraction of support vector of that class, and the upper bound to the fraction of bounded support vector of that class. These bounds allow the control on the error rates allowed for each class, and enable the training of machines with specific error rate requirements.

<h3>Context:</h3> Electronic consultation, or eConsult, is a secure online application that allows primary care providers (PCP) such as family physicians and nurse practitioners to communicate electronically with specialists concerning a patient’s care. <h3>Objective:</h3> In this study, we apply Natural Language Processing (NLP) to correspondence between PCP and the medical specialists through an electronic consultation (eConsult) platform to assess its feasibility. NLP is the automated analysis of a natural language (i.e., the languages people use to communicate). NLP is also used to extract insightful knowledge from text. We want to answer two main questions:1) How can we identify PCPs communication types with specialists? 2) What are the types of concerns and questions they ask specific to the eConsult taxonomy or case-types about patients? <h3>Study Design:</h3> We will use NLP to text-mine eConsult communications between PCPs and specialists and train a classification model to identify the patients’ case types in eConsult transactions. Clinician classifications will serve as the gold standard to train our machine learning algorithms. It will be built into the eConsult system to notify PCPs when a patient’s case is predicted. <h3>Setting or Dataset:</h3> In this study, we selected a sample of eConsult cases submitted through the BASE™ service during 2018 that were annotated by human as to which of the four categories they belonged to: “Management”, “Other”, “Diagnosis” and “Drug Treatment” of about 1200 cases. <h3>Population Studied:</h3> PCPs can continue to communicate with the specialist until they deem the conversation complete, at which time they close the case. We take an unbiased random sample of the population. <h3>Outcome Measures:</h3> We used 10-fold cross-validation (CV) to measure the accuracies, precision, recall and F1 scores. <h3>Results:</h3> We have conducted experiments only using the 4 categories. So far, the Support Vector Machines (SVM) with TFIDF feature provided the top accuracy of 80% using 10% of the data set aside for testing or with accuracy of 75% (+/- 0.15) using 10-fold cross validation and is therefore considered as our champion model. <h3>Conclusion</h3> We have proven the concept, that is the feasibility of using NLP to categorize eConsult cases, and developed models with acceptable accuracy, to automatically categorize the types of questions asked or comments provided by the PCPs about patients.

BackgroundRapid triage reduces the patients' stay time at an emergency department (ED). The Korean Triage Acuity Scale (KTAS) is mandatorily applied at EDs in South Korea. For rapid triage, we studied machine learning-based triage systems composed of a speech recognition model and natural language processing-based classification.MethodsWe simulated 762 triage cases that consisted of 18 classes with six types of the main symptom (chest pain, dyspnea, fever, stroke, abdominal pain, and headache) and three levels of KTAS. In addition, we recorded conversations between emergency patients and clinicians during the simulation. We used speech recognition models to transcribe the conversation. Bidirectional Encoder Representation from Transformers (BERT), support vector machine (SVM), random forest (RF), and k-nearest neighbors (KNN) were used for KTAS and symptom classification. Additionally, we evaluated the Shapley Additive exPlanations (SHAP) values of features to interpret the classifiers.ResultsThe character error rate of the speech recognition model was reduced to 25.21% through transfer learning. With auto-transcribed scripts, support vector machine (area under the receiver operating characteristic curve [AUROC], 0.86; 95% confidence interval [CI], 0.81–0.9), KNN (AUROC, 0.89; 95% CI, 0.85–0.93), RF (AUROC, 0.86; 95% CI, 0.82–0.9) and BERT (AUROC, 0.82; 95% CI, 0.75–0.87) achieved excellent classification performance. Based on SHAP, we found “stress”, “pain score point”, “fever”, “breath”, “head” and “chest” were the important vocabularies for determining KTAS and symptoms.ConclusionWe demonstrated the potential of an automatic KTAS classification system using speech recognition models, machine learning and BERT-based classifiers.

The recognition of blood disorder through the visual observation is the most challenging job. In the current technological era computer become the most important part of medical science. The haematological disorders of white blood cells (WBC) are really frequent in medical practices. The objective of this research is to design and development of automated identification of Leukemia using microscopic blood smear image database. This proposed scheme uses the most significant steps of image processing like, pre-processing, image segmentation, extraction of features and classification. The Leukemia smear image database is segmented using Otsu image segmentation. The feature extraction extracts the area, perimeter, solidity, orientation, eccentricity, centroid, entropy and energy features. The classification method applied using neural network, Support vector machine and QDA approach. In the neural network the 60% dataset has been passed for the training, 35% towards the testing and remaining 05% is used for the validation. The SVM and QDA classify the dataset for the two groups such as normal and leukaemia. The classification is done on the extracted 08 features of each image. The performance of the neural network is achieving 98.97% with 1.0246 error rate. The support vector machine is shows the 99.35% accuracy with 0.6500 error rate. The QDA classification reported the 99.70% accuracy with 0.300% error rate. From the reported accuracy the QDA and support vector machine proved as dominant as the neural network.

기업의 부도를 예측하는 것은 회계나 재무 분야에서 중요한 연구주제이다. 지금까지 기업 부도예측을 위해 여러 가지 데이터마이닝 기법들이 적용되었으나 주로 단일 모형을 사용함으로서 복잡한 분류 문제에의 적용에 한계를 갖고 있었다. 본 논문에서는 최근에 각광받고 있는 SVM (support vector machine) 모형들을 결합한 앙상블 SVM 모형 (ensemble SVM model)을 부도예측에 사용하고자 한다. 제안된 앙상블 모형은 v-조각 교차 타당성 (v-fold cross-validation)에 의해 얻어진 여러 가지 모형 중에서 성능이 좋은 상위 k개의 단일 모형으로 구성하고 과반수 투표 방식 (majority voting)을 사용하여 미지의 클래스를 분류한다. 본 논문에서 제안된 앙상블 SVM 모형의 성능을 평가하기 위해 실제 기업의 재무비율 자료와 모의실험자료를 가지고 실험하였고, 실험결과 제안된 앙상블 모형이 여러 가지 평가척도 하에서 단일 SVM 모형들보다 좋은 성능을 보임을 알 수 있었다. Corporate bankruptcy prediction has been an important topic in the accounting and finance field for a long time. Several data mining techniques have been used for bankruptcy prediction. However, there are many limits for application to real classification problem with a single model. This study proposes ensemble SVM (support vector machine) model which assembles different SVM models with each different kernel functions. Our ensemble model is made and evaluated by v-fold cross-validation approach. The k top performing models are recruited into the ensemble. The classification is then carried out using the majority voting opinion of the ensemble. In this paper, we investigate the performance of ensemble SVM classifier in terms of accuracy, error rate, sensitivity, specificity, ROC curve, and AUC to compare with single SVM classifiers based on financial ratios dataset and simulation dataset. The results confirmed the advantages of our method: It is robust while providing good performance.