Multi-Modal Mining in Web image retrieval

Vision-language constraint graph representation learning for unsupervised vehicle re-identification

Multimodal image retrieval, which involves retrieving images using various modalities such as text, audio, or other images, has significant research importance due to its wide-ranging applications and potential to enhance user experiences across multiple domains. Traditional image retrieval systems, content-based image retrieval (CBIR) systems rely solely on visual features, which can be limiting. By integrating multiple modalities, multimodal retrieval systems can offer more robust and accurate results. A research problem is considered multimodal when it integrates information from more than one type of data source. In multimodal image retrieval (MMIR) systems, one form of data is used to search for outcomes in the same or different modalities. In contrast, cross-modal systems strictly retrieve information from a different modality. A significant challenge in these systems is the effective comparison of input-output queries from different data types, due to their basic forms and the subjective nature of content similarity. Researchers have proposed various techniques to address this challenge and to bridge the semantic gap in information retrieval across different modalities. This article presents comprehensive analysis of various research works pertained in this multimodal image retrieval. The results and comparative analysis of current research works on benchmark datasets have also been discussed. At the end of the paper, some of open issues are presented for future research directions.

To bridge the semantic gap between low-level visual features and high-level semantic concepts, this paper puts forward a novel semi-supervised learning framework of combining visual and keyword features. We assume all of images in the database have been annotated. In this framework, the visual space graph (VSG) and keyword space graph (KSG) are first constructed by means of manifold-based algorithm to explore the submanifold of Visual space and keyword space. Then, semantic space graph (SSG) is constructed by using neighbor information of KSG and VSF. Afterwards, the VSG and KSG are modified and unified by propagating semantic matrix. Finally, we propose a flexible ranking formula and introduce different relevance feedback methods of different query styles. Experimental results on COREL 5000 images show our method improves image retrieval performance from all aspects.

Bag of Features (BoF) has been successfully exploited in content-based image retrieval for several years. Due to its performance and popularity, several extensions have been proposed that involve feature description, dictionary building, feature encoding and post-query process, etc. This paper proposes a dynamically weighting scheme for BoF-based image retrieval based on feedback. It involves two contributions: (i) analyzing the statistical distribution characteristic of similar BoF representations and (ii) computing weights dynamically based on the feedback obtained from different initial query results. We quantitatively evaluate the proposed method on two different databases. Experiments confirm that the proposed weighting scheme has better performance than the baseline of BoF-based image retrieval systems. Meanwhile, the results demonstrate the effectiveness of the weighting scheme in terms of the precision of top-N returned images.

Ontologies represent domain concepts and relations in a form of semantic network. Many research works use ontologies in the information matchmaking and retrieval. This trend is further accelerated by the convergence of various information sources supported by ontologies. In this paper, we propose a novel multi-modality ontology model that integrates both the low-level image features and the high-level text information to represent image contents for image retrieval. By embedding this ontology into an image retrieval system, we are able to realize intelligent image retrieval with high precision. Moreover, benefiting from the soft-coded ontology model, this system has good flexibility and can be easily extended to the larger domains. Currently, our experiment is conducted on the animal domain canine. An ontology has been built based on the low-level features and the domain knowledge of canine. A prototype retrieval system is set up to assess the performance. We compare our experiment results with traditional text-based image search engine and prove the advantages of our approach.

Although relevance feedback (RF) has been extensively studied in the content-based image retrieval community, no commercial Web image search engines support RF because of scalability, efficiency, and effectiveness issues. In this paper, we propose a unified relevance feedback framework for Web image retrieval. Our framework shows advantage over traditional RF mechanisms in the following three aspects. First, during the RF process, both textual feature and visual feature are used in a sequential way. To seamlessly combine textual feature-based RF and visual feature-based RF, a query concept-dependent fusion strategy is automatically learned. Second, the textual feature-based RF mechanism employs an effective search result clustering (SRC) algorithm to obtain salient phrases, based on which we could construct an accurate and low-dimensional textual space for the resulting Web images. Thus, we could integrate RF into Web image retrieval in a practical way. Last, a new user interface (UI) is proposed to support implicit RF. On the one hand, unlike traditional RF UI which enforces users to make explicit judgment on the results, the new UI regards the users' click-through data as implicit relevance feedback in order to release burden from the users. On the other hand, unlike traditional RF UI which hardily substitutes subsequent results for previous ones, a recommendation scheme is used to help the users better understand the feedback process and to mitigate the possible waiting caused by RF. Experimental results on a database consisting of nearly three million Web images show that the proposed framework is wieldy, scalable, and effective.

Iterative brain tumor retrieval for MR images based on user’s intention model

BackgroundEveryday modifiable lifestyle factors play a crucial role in the development and maintenance of successful cognitive and neurodegenerative trajectories in later life. Although research has focused on independent associations of physical exercise (PE), diet, body mass index (BMI), and sleep, how all four factors simultaneously work in a larger multimodal association has not been studied. We examine the synergistic associations of all four risk factors to predict two markers of future white matter injury (cerebral free water [FW] content, a sensitive and early biomarker of white matter injury and white matter hyperintensities [WMH]) in Hispanic/Latino‐population.MethodWe used cognitively normal older adults (n = 1386; mean age = 50.96(10.14) years; 66.2% female) with cross‐sectional data from the Study of Latinos‐Investigation of Neurocognitive Aging‐Magnetic Resonance Imaging (SOL‐INCA‐MRI). First, we applied path analyses to test independent associations of diet(Healthy‐Eating‐Index‐2010), PE(total activity), BMI, and sleep‐duration (measured 2008‐2015) on FW and WMH (measured 2018‐2022). Higher PE score represents lower exercise. Second, we tested mediations using significant independent predictors of FW and WMH. Third, we explored the best possible models that incorporated all six factors. We accounted for age, sex, and education in the final model.ResultFirst, lower PE predicted greater WMH (β = 0.171,SE = 0.064,p = 0.008) and FW (β = 0.004,SE = 0.002, p = 0.013). Higher BMI (β = 0.018,SE = 0.007, p = 0.017), and higher FW (β = 22.230,SE = 0.939, p<0.001) was associated with greater WMH. Second, we did not find that any of the three modifiable factors mediated the relationship between PE and FW. However, the association between PE and WMH was mediated by BMI (indirect model[borderline]:β = 0.012,SE = 0.006, p = 0.050) and by FW (indirect model:β = 0.066,SE = 0.025, p = 0.007). Third, the addition of diet and sleep in this double mediation model strengthened both mediations (PE→BMI→WMH:β = 0.012,SE = 0.006, p = 0.039; PE→FW→WMH:β = 0.065,SE = 0.025, p = 0.008) resulting in an overall indirect model (PE→BMI→WMH+ PE→FW→WMH:β = 0.077,SE = 0.025,p = 0.002) (Figure 1). Better diet was associated with lower BMI (β = ‐0.026,SE = 0.012,p = 0.029) and lower PE with more sleep (β = 0.190,SE = 0.061,p = 0.002).ConclusionWe observed that the association between PE and WMH is mediated by both BMI and FW, and the overall double mediation is clearer when diet and sleep are accounted for in the model. Our complex multimodal model provides a potential framework on how everyday modifiable risk factors synergistically predict WMH years later.

The significance of this research lies in the application of data mining techniques to analyze badminton competitions, allowing for the identification of players’ tactics, strategies, and weaknesses. By uncovering these insights, targeted training and improvement plans can be developed, enhancing player performance and optimizing competitive outcomes. Association rules in data mining techniques are used, and the Apriori algorithm in association rules is analyzed and improved. The ACARMI algorithm (an improved algorithm of constrained association rule mining based on item) is then applied. Then, data from six top badminton competitions is collected and converted into Boolean data suitable for this algorithm. A data mining module is designed to mine the tactical association rules of all players, and a specific analysis is conducted on one of the players, successfully analyzing their tactical strategies. Finally, to verify the effectiveness of the algorithm proposed in this article, FP-Growth algorithm and DHP algorithm are applied to comparative experiments. The experiment shows that compared with the FP-Growth algorithm, the initial and final running times of the algorithm in this article with minimum support and different constraint numbers are around “355 ms and 75 ms,” “150 ms and 45 ms,” respectively; similarly, compared with the DHP algorithm, they are around “165 ms and 35 ms,” “175 ms and 55 ms,” respectively. Under two sets of experiments, the algorithm proposed in this article presents better operational efficiency. This article demonstrates that the use of data mining techniques can provide effective tactical optimization, opponent research, real-time decision support, and long-term development planning assistance for athletes and coaches.

In this paper, we design and construct a multi-label image annotation and retrieval system. Various MPEG-7 low level visual features are employed for representing images. For image annotation, a bi-coded genetic algorithm is employed to select optimal feature subsets and corresponding optimal weights for every one vs. one SVM classifiers. After an unlabelled image is segmented into several regions, pre-trained SVMs are used to annotate each region, final label is obtained by merging all the region labels. Based on multi-label of image, image retrieval system provides keyword-based image retrieval service. Multi-labels can provide abundant descriptions for image content in semantic level, high precision annotation algorithm further improve annotation performance.

The rapid popularization of digital cameras and mobile phone cameras has lead to an explosive growth of consumer photo collections. In this paper, we present a (quasi) real-time textual query based personal photo retrieval system by leveraging millions of web images and their associated rich textual descriptions (captions, categories, etc.). After a user provides a textual query (e.g., pool), our system exploits the inverted file method to automatically find the positive web images that are related to the textual query pool as well as the negative web images which are irrelevant to the textual query. Based on these automatically retrieved relevant and irrelevant web images, we employ two simple but effective classification methods, k Nearest Neighbor (kNN) and decision stumps, to rank personal consumer photos. To further improve the photo retrieval performance, we propose three new relevance feedback methods via cross-domain learning. These methods effectively utilize both the web images and the consumer images. In particular, our proposed cross-domain learning methods can learn robust classifiers with only a very limited amount of labeled consumer photos from the user by leveraging the pre-learned decision stumps at interactive response time. Extensive experiments on both consumer and professional stock photo datasets demonstrated the effectiveness and efficiency of our system, which is also inherently not limited by any predefined lexicon.

Series feature aggregation for content-based image retrieval

최근에 많이 활용되고 있는 데이터 분석을 위한 연관성 규칙 마이닝은 대용량 데이터베이스에 많이 활용되고 있는 서 두 항목간의 관계를 측도화 함으로써 두 개 이상의 항목간의 관련성을 표시하여 주는 기법이다. 연관성 규칙의 여부를 판단하기 위한 연관성 평가 기준에는 지지도, 신뢰도, 그리고 향상도 등이 있으며, 이들 세 가지 기준을 이용하여 연관성 규칙 생성 여부를 판단하게 된다. 이에 대한 기존의 연구 결과는 결정함수를 이용하는 방법과 회귀모형을 이용하는 방법으로 분류할 수 있다. 회귀모형을 이용하여 수행한 연구에는 지지도와 신뢰도에 의한 모형, 세 가지 평가 기준의 쌍에 의한 모형, 표준화 향상도를 포함한 세 가지 평가 기준의 쌍에 의한 모형, 그리고 세 가지 평가 기준 전부를 고려한 모형 등이 있다. 본 논문에서는 기존의 연구를 확장하는 의미에서 표준화 향상도를 포함한 세가지 평가 기준 전부를 고려한 비선형 회귀모형을 이용하여 연관성 규칙의 수를 추정하는 방안에 대해 강구하고자 한다. 또한 분산분석에서의 F 통계량과 수정 결정계수를 이용하여 각 모형의 유의한 정도를 비교하는 동시에 분산팽창계수에 의한 공선성 문제를 진단함으로써 가장 유용한 회귀 모형을 탐색하고자 한다. Among data mining techniques, the association rule is one of the most used in the real fields because it clearly displays the relationship between two or more items in large databases by quantifying the relationship between the items. There are three primary quality measures for association rule; support, confidence, and lift. We evaluate association rules using these measures. The approach taken in the previous literatures as to estimation of association rule number has been one of a determination function method or a regression modeling approach. In this paper, we proposed a few of non-linear regression equations useful in estimating the number of rules and also evaluated the estimated association rules using the quality measures. Furthermore we assessed their usefulness as compared to conventional regression models using the values of regression coefficients, F statistics, adjusted coefficients of determination and variation inflation factor.

Global gene expression profiling, both at the transcript level and at the protein level, can be a valuable tool in the understanding of genes, biological networks, and cellular states. As larger and larger gene expression data sets become available, data mining techniques can be applied to identify patterns of interest in the data. Association rules, used widely in the area of market basket analysis, can be applied to the analysis of expression data as well. Association rules can reveal biologically relevant associations between different genes or between environmental effects and gene expression. An association rule has the form LHS --> RHS, where LHS and RHS are disjoint sets of items, the RHS set being likely to occur whenever the LHS set occurs. Items in gene expression data can include genes that are highly expressed or repressed, as well as relevant facts describing the cellular environment of the genes (e.g. the diagnosis of a tumor sample from which a profile was obtained). We demonstrate an algorithm for efficiently mining association rules from gene expression data, using the data set from Hughes et al. (2000, Cell, 102, 109-126) of 300 expression profiles for yeast. Using the algorithm, we find numerous rules in the data. A cursory analysis of some of these rules reveals numerous associations between certain genes, many of which make sense biologically, others suggesting new hypotheses that may warrant further investigation. In a data set derived from the yeast data set, but with the expression values for each transcript randomly shifted with respect to the experiments, no rules were found, indicating that most all of the rules mined from the actual data set are not likely to have occurred by chance. An implementation of the algorithm using Microsoft SQL Server with Access 2000 is available at http://dot.ped.med.umich.edu:2000/pub/assoc_rules/assoc_rules.zip. Our results from mining the yeast data set are available at http://dot.ped.med.umich.edu:2000/pub/assoc_rules/yeast_results.zip.

Association rules provide an important clue on how hidden knowledge and patterns can be extracted and the attributes extracted are associated when big data is mined by means of a particular data mining algorithm. The context of HEIs, where big data mining is uncommon, there is a need to know how association rules could be utilized to discover hidden knowledge and forecast student performance based on time-to-degree and cumulative grade point average (CGPA) and also make decisions. This research has demonstrated this by taking the example of the dataset of 337 graduated students of a university in Bahrain belonging to the Bachelor's degree in Accounting and Finance. Two types of data mining concepts were applied, regression and a priori algorithm. Regression results showed that there is a statistically substantial link between time-to-degree and course-difficulty and not CGPA. Similarly, association rules showed the link between time-to-degree, course-taking pattern, no. of courses and course-difficulty pattern but not CGPA. The association rules generated provided the basis to predict optimum time-to-degree by controlling the no. of courses, course-taking pattern, and course-taking difficulty. The findings of this research provide an important window of opportunity for HEIs to enhance student performance using data mining techniques. This investigation has contributed theoretically showing the application of data mining theory using apriori algorithm and regression, for HEI big data mining where student factors and organizational factors are said to be fairly different from that of the HEIs where research on big data is seen to be a well-known activity.