VIT-MEDBERT: A Multi-Modal Transformer-Based Framework for Oral Disease Classification

The image tagging task aims to assign relevant known tags to an image. It is an active research topic in computer vision and machine learning because of the diversity of its applications in semantic search and image retrieval. Earlier efforts on image tagging address this problem as a multi-level classification problem using visual features from images and semantic word vectors of tags. In most cases, a pre-trained language model like word2vec or Globe is used to obtain those word vectors. Because of using a pre-trained language model, an image tagging approach cannot scale itself to the context of the targeted application. This paper fine-tunes a language (BERT) model using text descriptions obtained from web (Wikipedia) scraping to learn a rich distributed representation of tags. Then, we employ word vectors of tags extracted from finetuned language (BERT) model to solve the image tagging task. Our method is more specialized to the particular application by incorporating context information between targeted tags and images. As a result, word vectors obtained from the fine-tuned model perform better than those from pre-trained language models. We evaluate our method on the widely used NUS-WIDE dataset and achieve competitive results compared with state-of-the-art methods.

Improving Breast Cancer Target Delineation: A Novel Computer Assisted Program of CT Overlays and Color Coded 3D Reconstructions

Pre-trained vision-language models (VLMs) and language models (LMs) have recently garnered significant attention due to their remarkable ability to represent textual concepts, opening up new avenues in vision tasks. In medical image segmentation, efforts are being made to integrate text and image data using VLMs and LMs. However, current text-enhanced approaches face several challenges. First, using separate pre-trained vision and text models to encode image and text data can result in semantic shifts. Second, while VLMs can establish the correspondence between visual and textual features when pre-trained on paired image-text data, this alignment often deteriorates during segmentation tasks due to misalignment between the text and vision components in ongoing learning. In this paper, we propose TeViA, a novel approach that seamlessly integrates with various vision and text models, irrespective of their pre-training relationships. This integration is achieved through a segmentation-specific text-to-vision alignment design, ensuring both information gain and semantic consistency. Specifically, for each training data, a foreground visual representation is extracted from the segmentation head and used to supervise projection layers, thereby adjusting the textual features to better contribute to the segmentation task. Additionally, a historic visual prototype is created by aggregating target semantics from all training data and is updated using a momentum-based manner. This prototype aims to enhance the visual representation of each data instance by establishing feature-level connections, which in turn refines the textual features. The superiority of TeViA is validated on five public datasets, exhibiting over 6% Dice improvements compared to vision-only methods. Code is available at: https://github.com/jgfiuuuu/TeViA.

A growing body of literature has explored the potential for arts-based methods to generate and disseminate research, particularly on sensitive or complex topics. This article presents DrawingOut, a metaphor-centred drawing workshop designed to collect visual and textual data about individuals’ experiences of sensitive or taboo health experiences. The visual data, consisting of the artwork produced by participants, and the textual data, all conversations and discussions occurring during the workshop, can also be used to create engaging outputs for dissemination. We piloted DrawingOut in a study of nine women with a minority ethnic or religious background in Cardiff, UK. The women were invited to participate in a series of structured drawing activities. The conversations occurring during the workshop were recorded and then subjected to thematic analysis. Results showed that women’s views and experiences could be grouped in eight major themes covering their wellbeing, relationship with others, and healthcare views and experiences. We produced an A5 16-page booklet that presents the pilot study results, illustrated with participants’ own drawings and quotes, which was disseminated among participants, their community and other relevant stakeholders. This pilot study supports the view that healthcare actors can use the DrawingOut method to engage people to talk about sensitive health topics, while simultaneously providing them with an enjoyable and empowering research experience. In our pilot study the DrawingOut was successful in eliciting rich visual and textual data that captures a diversity of views and experiences, with the added benefit that these can be used to produce engaging outputs for dissemination.

A Comparative Study of Outfit Recommendation Methods with a Focus on Attention-based Fusion

Convolution Neural Network plays an important role in the field of agriculture for the identification and classification of diseases in the different types of crops. Most of the families in India belong to the farming background. So, in farming the paddy crop plays the crucial role because most of the famers are interested in growing the paddy crop. There are many diseases which damages the paddy crop. and it effects the production of rice. Due to this, farmers can’t meet the required production of food. In order to reduce the damage from different types of diseases on paddy crop, the CNN model is chosen. There are other sources like machine learning technology for identification of diseases but in recent times due to increase in technology and accuracy metrics the deep learning is widely used. In paddy crop, there are more than 30 varieties of diseases which damage the crop. Among these, four types of diseases like Leaf Blast, Brown Spot, Hispa and Bacterial Leaf Blight are more commonly observed in the crop. And these diseases are similar to each other, so that they can’t even be identified by the naked eye. In order to reduce the damage from diseases they should be identified in early stage and the necessary precautions must be taken for the increase in the crop yield. The main objective of this project is to make a deep learning model for the identification and classification of paddy crop diseases through transfer learning and image processing and all the process is done in deep learning. This project is mainly focuses on four disease classes and the classification process, the accuracy metrics is analyzed. Among the different pre trained models such as InceptionV3, VGG 16, Resnet 50, the better accuracy is achieved from Inceptionv3 with an accuracy of 91.23%.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

Antimicrobial peptides (AMPs) are vital components of innate immunotherapy. Existing approaches mainly rely on either deep learning for the automatic extraction of sequence features or traditional manual amino acid features combined with machine learning. The peptide sequence contains symmetrical sequence motifs or repetitive amino acid patterns, which may be related to the function and structure of the peptide. Recently, the advent of large language models has significantly boosted the representational power of sequence pattern features. In light of this, we present a novel AMP predictor called UniproLcad, which integrates three prominent protein language models—ESM-2, ProtBert, and UniRep—to obtain a more comprehensive representation of protein features. UniproLcad utilizes deep learning networks, encompassing the bidirectional long and short memory network (Bi-LSTM) and one-dimensional convolutional neural networks (1D-CNN), while also integrating an attention mechanism to enhance its capabilities. These deep learning frameworks, coupled with pre-trained language models, efficiently extract multi-view features from antimicrobial peptide sequences and assign attention weights to them. Through ten-fold cross-validation and independent testing, UniproLcad demonstrates competitive performance in the field of antimicrobial peptide identification. This integration of diverse language models and deep learning architectures enhances the accuracy and reliability of predicting antimicrobial peptides, contributing to the advancement of computational methods in this field.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

Composed Image Retrieval (CIR) represents a novel retrieval paradigm that is capable of expressing users' intricate retrieval requirements flexibly. It enables the user to give a multimodal query, comprising a reference image and a modification text, and subsequently retrieve the target image. Notwithstanding the considerable advances made by prevailing methodologies, CIR remains in its nascent stages due to two limitations: 1) inhomogeneity between dominant and noisy portions in visual data is ignored, leading to query feature degradation, and 2) the priority of textual data in the image modification process is overlooked, which leads to a visual focus bias. To address these two limitations, this work presents a focus mapping-based feature extractor, which consists of two modules: dominant portion segmentation and dual focus mapping. It is designed to identify significant dominant portions in images and guide the extraction of visual and textual data features, thereby reducing the impact of noise interference. Subsequently, we propose a textually guided focus revision module, which can utilize the modification requirements implied in the text to perform adaptive focus revision on the reference image, thereby enhancing the perception of the modification focus on the composed features. The aforementioned modules collectively constitute the segmentatiOn-based Focus shiFt reviSion nETwork (OFFSET), and comprehensive experiments on four benchmark datasets substantiate the superiority of our proposed method. The codes and data are available on https://zivchen-ty.github.io/OFFSET.github.io/.

Texts, images and other information are posted everyday on the social network and provides a large amount of multimodal data. The aim of this work is to investigate if combining and integrating both visual and textual data permits to identify emotions elicited by an image. We focus on image emotion classification within eight emotion categories: amusement, awe, contentment, excitement, anger, disgust, fear and sadness. Within this classification task we here propose to adopt ensemble learning approaches based on the Bayesian model averaging method, that combine five state-of-the-art classifiers. The proposed ensemble approaches consider predictions given by several classification models, based on visual and textual data, through respectively a late and an early fusion schemes. Our investigations show that an ensemble method based on a late fusion of unimodal classifiers permits to achieve high classification performance within all of the eight emotion classes. The improvement is higher when deep image representations are adopted as visual features, compared with hand-crafted ones.

This paper describes the models designated for the MEDIQA 2019 shared tasks by the team PANLP. We take advantages of the recent advances in pre-trained bidirectional transformer language models such as BERT (Devlin et al., 2018) and MT-DNN (Liu et al., 2019b). We find that pre-trained language models can significantly outperform traditional deep learning models. Transfer learning from the NLI task to the RQE task is also experimented, which proves to be useful in improving the results of fine-tuning MT-DNN large. A knowledge distillation process is implemented, to distill the knowledge contained in a set of models and transfer it into an single model, whose performance turns out to be comparable with that obtained by the ensemble of that set of models. Finally, for test submissions, model ensemble and a re-ranking process are implemented to boost the performances. Our models participated in all three tasks and ranked the 1st place for the RQE task, and the 2nd place for the NLI task, and also the 2nd place for the QA task.

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration

meaning representation (AMR) highlights the core semantic information of text in a graph structure. Recently, pre-trained language models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, respectively. However, PLMs are typically pre-trained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs.

Focus on software