Using Brand Knowledge Bases and LLM Agents to Enhance E-commerce Retailers' Catalog Quality

Large Language Models (LLMs) are attracting an enormous amount of interest at the moment in many domains. Their general nature, and ability to "understand" natural language, has already stimulated multiple areas of research at our company. Here we successfully demonstrate a Natural Language querying system, which is able to search a large repository of unstructured exploration data. The system supports follow up querying on the returned results, plus automatic summarization of content. The system is integrated into our novel end-to-end data-mining platform, which continuously mines our unstructured exploration data for new changes and indexes the results. Important in our method are the enrichment processes that occur prior to use of the LLM. Our approach avoids usual "chunking" techniques, which in our experience results in inferior results, especially in the multiple domain areas of Exploration. By integrating our novel ontology-model AI in the enrichment of the initial Index, we drastically boost the performance of search resulting from the LLM steps. In order to perform the search, key parts of our unstructured data, plus the query itself, need to be transformed into a vector form. This is performed using the embedding feature of the LLM. For this work, we had around 500,000 embeddings to calculate. To improve performance these were indexed in a leading Analytics Engine as a vector object, allowing fast search via cosine or Euclidian similarity. A custom dashboard was made to allow fresh searches of the vector datastore to be returned for further analysis. Our current search time across 500,000 embeddings is under 20 milli-seconds. Our custom dashboard returns the top matches for further interrogation and analysis. This includes follow-up Natural Language question support on the returned matches for summarization tasks and other customised querying. Since our exploration-specific, ontology model is able to tag each piece of data with over 40 exploration-specific labels, we are able to cross-examine the LLM returned results with the tags. Agreement on a range of queries - ranging from targeted, highly specific questions to general, open-ended queries - was surprisingly good. Natural Language based querying of our unstructured data is opening a whole new approach to data discovery in our company. Tailoring it to the exploration domain has required specific domain expertise and a novel ontology-model be used to ensure relevant prompts and query results. Obtaining search results quickly has also required expertise and fine-tuning. Future directions include ingesting more data, scaling the support infrastructure and further capability enhancement.

Generative artificial intelligence is advancing at a blistering pace. Large Language Models, in particular, have sped up the development of machine learning applications. This work presents a large language model-based technique to query data collected during MD2 pineapple crop production. Retrieval Augmented Generation was used to feed structured and unstructured data to two large language models (GPT-4 and LLAMA2) to train and fine-tune the models. The performance of the models was then measured using actual and predicted question-answer pairs. Results showed that the models had a 78% - 87% correct answer rate for structured and 75% - 79% correct answer rate for unstructured data. However, results showed that the models had a 61%-68 % correct answer rate when an answer to a question needed to refer to structured and unstructured data. These results showed that large language models can be further investigated to give farmers useful insights when making crop management decisions.

BackgroundTreatment of sepsis requires prompt initiation of appropriate empiric antibiotics. Coverage for Gram-negative bacilli (GNB) is decided based on risk factors for resistant organisms, as documented in both structured and unstructured data in the electronic health record. We hypothesized that adding large language models (LLMs) that can decode clinical narratives would improve performance of deep learning models in predicting antimicrobial resistance in GNB sepsis.MethodsThis retrospective cohort analysis included all adult patients who met CDC sepsis criteria admitted to 10 BJC hospitals from 1/2018 to 12/2023. Sepsis episodes were classified as community-onset (< 48hrs from hospitalization) or hospital-onset. GNB were stratified as ceftriaxone-susceptible (SS), ceftriaxone-resistant but cefepime-susceptible (RS), and ceftriaxone- and cefepime-resistant (RR). Culture negative or non-GNB sepsis were labeled SS. Deep learning models (XGBoost, DeepAMR) were developed using all available structured data (e.g., demographics, vitals, labs). LLMs (BioGPT, Medical-LLaMa3-8B) were developed on full or partial history and physical exam notes. Models were assessed using area under the receiver operating characteristic curve (AUROC) and area under precision-recall curve (AUPRC).ResultsA total of 29,517 patients were included, for 23,949 community-onset and 25,101 hospital-onset sepsis episodes with RS+RR GNB in 9% and 17%, respectively (Table 1). Relevant features of structured data models included history of prior infections and resistant organisms. DeepAMR was the best performing deep learning model with AUROC 0.85/AUPRC 0.40 and AUROC 0.81/AUPRC 0.50 for community- and hospital-onset sepsis, respectively (Table 2). The best performing LLM was Medical-LLaMa3-8B on full notes (Figure 1), with AUROC 0.74/AUPRC 0.21 and AUROC 0.67/AUPRC 0.30 for community- and hospital-onset sepsis, respectively. Combining deep learning and LLMs did not significantly improve the prediction of resistant infections compared to deep learning alone.ConclusionStructured medical data provided the highest prediction of resistant GNB infections in sepsis. More analysis is needed to understand why the unstructured data did not improve model predictive performance.DisclosuresMaria Cristina Vazquez Guillamet, MD, Bionano: Stocks/Bonds (Public Company)|Charisma: Stocks/Bonds (Public Company)|Ocugen: Stocks/Bonds (Public Company)

ATLASky-AI: An Autonomous Framework for Physics-Based Trustworthy Verification of LLM-Generated Spatiotemporal Knowledge

Design is a highly creative and challenging task and research has already explored possible ways for using conversational agents (CAs) to support humans participating in co-design sessions. However, research reports that a) humans in these sessions expect more essential support from CAs, and b) it is important to develop CAs that continually learn from communication -like humans do- and not simply from labeled datasets. Addressing the above needs, this paper explores the specific question of how to extract useful knowledge from human dialogues observed during co-design sessions and make this knowledge available through a CA supporting humans in similar design activities. In our approach we explore the potential of the GPT-3 Large Language Model (LLM) to provide useful output extracted from unstructured data such as free dialogues. We provide evidence that by implementing an appropriate “extraction task” on the LLM it is possible to efficiently (and without human-in-the-loop) extract knowledge that can then be embedded in the cognitive base of a CA. We identify at least four major steps/assumptions in this process that need to be further researched, namely: A1) Knowledge modeling, A2) Extraction task, A3) LLM-based facilitation, and A4) Humans’ benefit. We provide demonstrations of the extraction and facilitation steps using the GPT-3 model and we also identify and comment on various worth exploring open research questions.

InsQABench: Benchmarking Chinese insurance domain question answering with large language models

The primary goal of this invention is to revolutionize production engineering by addressing the challenges of integrating unstructured intervention reports with structured production data. Existing systems are unable to provide a seamless integration that enables comprehensive analysis and decision-making. This solution leverages Generative AI and an end-to-end automated pipeline to transform data handling, optimize intervention planning, and provide real-time insights, ultimately enhancing the efficiency and intelligence of production engineers. The invention employs several interconnected modules. First, the **Data Ingestion Module** collects unstructured intervention reports and structured production data from various sources, including databases and cloud platforms. The **Unstructured Data Handling Module** uses domain-driven Generative AI to extract key performance indicators (KPIs) from intervention reports, converting them into structured data formats. Next, the **Integration Module** unifies structured and unstructured data, synchronizing it in real-time. The **Knowledge Enrichment Module** creates a high dimensional knowledge base using Retrieval Augmented Generation (RAG), allowing for interaction with thousands of reports. The **Visualization Module** provides interactive dashboards for users to explore results, while the **Intelligent Assistant Module** enables engineers to query data in real-time, enhancing decision-making capabilities. The integration of structured and unstructured data allows production engineers to assess intervention impacts comprehensively. By automating the extraction of insights from reports and merging them with production time-series data, the solution provides a holistic view of interventions. The ability to interact with large datasets in real-time through an intelligent assistant enables faster and more informed decisions. Case studies have demonstrated that this approach can identify trends in intervention outcomes, optimize future interventions, and reduce missed opportunities due to incomplete analysis. In one instance, it prevented premature shut-off of oil-bearing zones by analyzing water shut-off (WSO) interventions and recommending an alternative, more effective intervention. This work has been implemented on 1000-page scanned documents combined with daily production rate for more than 20 wells for a field. This solution represents a significant leap in production engineering data. analysis by combining the power of Generative AI with real-time data integration. Its ability to seamlessly process both unstructured and structured data is unmatched in the industry. Unlike existing solutions that rely on separate analysis or manual data handling, this system automates the entire process. The use of Retrieval-Augmented Generation (RAG) technology for enriched knowledge interaction and a real-time visualization platform provides a deeper and more scalable understanding of intervention impacts, offering engineers a powerful tool for optimizing production performance.

Logistics Performance, Ratings, and its impact on Customer Purchasing Behavior and Sales in E-commerce Platforms

Recommendations can improve customer experience, improve search results, and boost Web site KPIs. RichRelevance’s RichRecs stacks up very well against our evaluation criteria. If you are in retail ecommerce, RichRelevance’s RichRecs should be on your short list. Its automated optimization of its myriad recommendation strategies gets retailers closer to their specific goals, with the least possible manual effort. RichRelevance’s service delivery gets very high marks from us as well.

The success of applications that process data critically depends on the quality of the ingested data. Completeness of a data source is essential in many cases. Yet, most missing value imputation approaches suffer from severe limitations. They are almost exclusively restricted to numerical data, and they either offer only simple imputation methods or are difficult to scale and maintain in production. Here we present a robust and scalable approach to imputation that extends to tables with non-numerical values, including unstructured text data in diverse languages. Experiments on public data sets as well as data sets sampled from a large product catalog in different languages (English and Japanese) demonstrate that the proposed approach is both scalable and yields more accurate imputations than previous approaches. Training on data sets with several million rows is a matter of minutes on a single machine. With a median imputation F1 score of 0.93 across a broad selection of data sets our approach achieves on average a 23-fold improvement compared to mode imputation. While our system allows users to apply state-of-the-art deep learning models if needed, we find that often simple linear n-gram models perform on par with deep learning methods at a much lower operational cost. The proposed method learns all parameters of the entire imputation pipeline automatically in an end-to-end fashion, rendering it attractive as a generic plugin both for engineers in charge of data pipelines where data completeness is relevant, as well as for practitioners without expertise in machine learning who need to impute missing values in tables with non-numerical data.

PurposeThis study aimed to present the methodology of the text data analysis to establish marketing strategies for fintech companies in a practical way. Specifically, the methodology was presented to convert customers' review data, which consisted of the text data (unstructured data), to the numerical data (structured data) by using a text mining algorithm “Global Vectors for Word Representation,” abbreviated as “GloVe”; additionally, the authors presented the methodology to deploy the numerical data for marketing strategies with eliminate-reduce-raise-create (ERRC) value factor analytics.Design/methodology/approachFirst, the authors defined the background, features and contents of fintech services based on a review of related literature review. Additionally, they examined business strategies, the importance of social media for fintech services and fintech technology trends based on the literature review. Next, they analyzed the similarity between fintech-related keywords, which represent the trends in fintech services, and the text data related to fintech corporations and their services posted on Facebook and Twitter, which are two of the most popular social media globally, during the period 2017–2019. The similarity was then quantified and categorized in terms of the representative global fintech companies and the status of each fintech service sector. Furthermore, the similarity was visualized, and value elements were rebuilt using ERRC strategy analytics.FindingsThis study is meaningful in that it quantifies the degree of similarity between customers' responses, experiences and expectations regarding the rapidly growing global fintech firms' services and trends in fintech services.Originality/valueThis study suggests a practical way to apply in business by providing a method for transforming unstructured text data into structured numerical data it is measurable. It is expected that this study can be used as the basis for exploring sustainable development strategies for the fintech industry.

Hospital readmission prediction is a crucial area of research due to its impact on healthcare expenditure, patient care quality, and policy formulation. Accurate prediction of patient readmissions within 30 days post-discharge remains a considerable challenging, given the complexity of healthcare data, which includes both structured (e.g., demographic, clinical) and unstructured (e.g., clinical notes, medical images) data. Consequently, there is an increasing need for hybrid approaches that effectively integrate these two data types to enhance all-cause readmission prediction performance. Despite notable advancements in machine learning, existing predictive models often struggle to achieve both high precision and balanced predictions, mainly due to the variability in patients’ outcome and the complex factors influencing readmissions. This study seeks to address these challenges by developing a hybrid predictive model that combines structured data with unstructured text representations derived from ClinicalT5, a transformer-based large language model. The performance of these hybrid models is evaluated against text-only models, such as PubMedBERT, using multiple metrics including accuracy, precision, recall, and AUROC score. The results demonstrate that the hybrid models, which integrate both structured and unstructured data, outperform text-only models trained on the same dataset. Specifically, hybrid models achieve higher precision and balanced recall, reducing false positives and providing more reliable predictions. This research underscores the potential of hybrid data integration, using ClinicalT5, to improve hospital readmission prediction, thereby improving healthcare outcomes through more accurate predictions that can support better clinical decision making and reduce unnecessary readmissions.

We present an online multilingual system for event detection and comprehension from media feeds. The system retrieves information from news sites, aggregates them into events (event detection), and summarizes them by extracting semantic labels of its most relevant entities (event representation) in order to answer the journalism Ws: who, what, when and where. The generated events populate VLX-Stories -an event ontology- transforming unstructured text data to a structured knowledge base representation. Our system exploits an external entity Knowledge Graph (VKG) to help populate VLX-Stories. At the same time, this external knowledge graph can also be extended with a Dynamic Entity Linking (DEL) module, which detects emerging entities (EE) on unstructured data. The system is currently deployed in production and used by media producers in the editorial process, providing real-time access to breaking news. Each month, VLX-Stories detects over 9000 events from over 4000 news feeds from seven different countries and in three different languages. At the same time, it detects over 1300 EE per month, which populate VKG.

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

BACKGROUND Information overload in electronic health records requires effective solutions to alleviate clinicians’ administrative tasks. Automatically summarizing clinical text has gained significant attention with the rise of large language models. While individual studies show optimism, a structured overview of the research landscape is lacking. OBJECTIVE This study aims to present the current state of the art on clinical text summarization using large language models, evaluate the level of evidence in existing research and assess the applicability of performance findings in clinical settings. METHODS This scoping review complied with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Literature published between January 1, 2019, and June 18, 2024, was identified from 5 databases: PubMed, Embase, Web of Science, IEEE Xplore, and ACM Digital Library. Studies were excluded if they did not describe transformer-based models, did not focus on clinical text summarization, did not engage with free-text data, were not original research, were nonretrievable, were not peer-reviewed, or were not in English, French, Spanish, or German. Data related to study context and characteristics, scope of research, and evaluation methodologies were systematically collected and analyzed by 3 authors independently. RESULTS A total of 30 original studies were included in the analysis. All used observational retrospective designs, mainly using real patient data (n=28, 93%). The research landscape demonstrated a narrow research focus, often centered on summarizing radiology reports (n=17, 57%), primarily involving data from the intensive care unit (n=15, 50%) of US-based institutions (n=19, 73%), in English (n=26, 87%). This focus aligned with the frequent reliance on the open-source Medical Information Mart for Intensive Care dataset (n=15, 50%). Summarization methodologies predominantly involved abstractive approaches (n=17, 57%) on single-document inputs (n=4, 13%) with unstructured data (n=13, 43%), yet reporting on methodological details remained inconsistent across studies. Model selection involved both open-source models (n=26, 87%) and proprietary models (n=7, 23%). Evaluation frameworks were highly heterogeneous. All studies conducted internal validation, but external validation (n=2, 7%), failure analysis (n=6, 20%), and patient safety risks analysis (n=1, 3%) were infrequent, and none reported bias assessment. Most studies used both automated metrics and human evaluation (n=16, 53%), while 10 (33%) used only automated metrics, and 4 (13%) only human evaluation. CONCLUSIONS Key barriers hinder the translation of current research into trustworthy, clinically valid applications. Current research remains exploratory and limited in scope, with many applications yet to be explored. Performance assessments often lack reliability, and clinical impact evaluations are insufficient raising concerns about model utility, safety, fairness, and data privacy. Advancing the field requires more robust evaluation frameworks, a broader research scope, and a stronger focus on real-world applicability.