Naïve Random Neighbor Selection for memory based Collaborative Filtering

Evaluation Of Recommendation System On A Movie Dataset

With the rapid development of e-commerce, collaborative filtering recommendation system has been widely used in various network platforms. Using recommendation system to accurately predict customers’ preferences for goods can solve the problem of information overload faced by users and improve users’ dependence on the network platform. Because the recommendation system based on collaborative filtering technology has the ability to recommend more abstract or difficult to describe goods in words, the research related to collaborative filtering technology has attracted more and more attention. According to the past research, in collaborative filtering algorithm, if Pearson correlation coefficient is used, errors will occur under special circumstances. In this study, the normal recovery similarity measure is used to modify the similarity value to correct the error value of a collaborative filtering recommendation algorithm. Based on this, a big data analysis method based on a modified collaborative filtering recommendation algorithm is proposed. This research implemented it in the cloud Hadoop environment, and measure the execution time with 2, 5 and 8 nodes. Then the research compared it with the execution time of a single machine, and analyze its speedup ratio and efficiency. The experimental results show that the execution time increases with the number of neighbors. When the number of nodes is 5 and 8, the execution time is greatly improved, which improves the efficiency of collaborative filtering algorithm and can cope with massive data in the future.

With rapid increase in volume of text documents over Internet, the collections of document in digitilized form piling up every day. There is a need of effective and efficient measurement techniques in order to deal with such an enormous amount of data. Text similarity measurement is one of the most important metrics for proper understanding of documents in information retrieval and text mining problem. The conventional similarity measurement and cosine similarity measurement are based on Euclidean distance (L2 norm) that give better results only for lower dimensions of data. Hellinger distance-based similarity measurement (L1 norm) having constant value of metric works well only for some data mining applications. In this paper, we proposed a new similarity measurement based on Lk metric. It incorporated with existing L1 metric-based cosine similarity measurement with decreasing values of k as (0.49, 0.48, and 0.47) that depicts relative contrast of distances to query point. Performance evaluation shows that the proposed method is indeed effective as compared to existing one and also suitable for the query-based search for assigning a rank to the documents with respect to the query document.

A new similarity measure for collaborative filtering based recommender systems

Traditionally, many information retrieval models assume that terms occur in documents independently. Although these models have already shown good performance, the word independency assumption seems to be unrealistic from a natural language point of view, which considers that terms are related to each other. Therefore, such an assumption leads to two well‐known problems in information retrieval (IR), namely, polysemy, or term mismatch, and synonymy. In language models, these issues have been addressed by considering dependencies such as bigrams, phrasal‐concepts, or word relationships, but such models are estimated using simple n‐grams or concept counting. In this paper, we address polysemy and synonymy mismatch with a concept‐based language modeling approach that combines ontological concepts from external resources with frequently found collocations from the document collection. In addition, the concept‐based model is enriched with subconcepts and semantic relationships through a semantic smoothing technique so as to perform semantic matching. Experiments carried out on TREC collections show that our model achieves significant improvements over a single word‐based model and the Markov Random Field model (using a Markov classifier).

The automatic detection of shared content in written documents –which includes text reuse and its unacknowledged commitment, plagiarism– has become an important problem in Information Retrieval. This task requires exhaustive comparison of texts in order to determine how similar they are. However, such comparison is impossible in those cases where the amount of documents is too high. Therefore, we have designed a model for the proper pre-selection of closely related documents in order to perform the exhaustive comparison afterwards. We use a similarity measure based on word-level n-grams, which proved to be quite effective in many applications As this approach becomes normally impracticable for real-world large datasets, we propose a method based on a preliminary word-length encoding of texts, substituting a word by its length, providing three important advantages: (i) being the alphabet of the documents reduced to nine symbols, the space needed to store n-gram lists is reduced; (ii) computation times are decreased; and (iii) length n-grams can be represented in a trie, allowing a more flexible and fast comparison. We experimentally show, on the basis of the perplexity measure, that the noise introduced by the length encoding does not decrease importantly the expressiveness of the text. The method is then tested on two large datasets of co-derivatives and simulated plagiarism.

Internet sources contain a vast amount of information about items that people desire to purchase. It is impossible to evaluate these resources and come to an informed decision. People need automated systems that evaluate previous information and propose item alternatives. Recommending items using a smart system, which is based on the previous user preferences, has growing importance since the available product data is exponentially growing. Additionally, it is difficult to find new and correct things that a user would like among this massive amount of data. To make accurate recommendations with a smart system, researchers and practitioners use collaborative filtering methods with similarity calculation based on user preferences. The crucial point in collaborative filtering is to find a valuable measure that resembles correct similarity between users. The current similarity metrics in the literature have some disadvantages in conducting accurate recommendations. To improve the recommendation performance, this study proposes a novel similarity measure that assesses the distance between the user’s ratings and the median score. Considering distance from the median score is essential since some users may prefer to rate close to the median rather than the extremes. Experiments were conducted with a famous collaborative filtering dataset. Results showed that proposed similarity measure demonstrated superior performance regarding the recommendation accuracy. Implications of our results for XYZ are discussed.

High Rating Recent Preferences Based Recommendation System

Memory-based collaborative filtering (CF) aims at predicting the rating of a certain item for a particular user based on the previous ratings from similar users and/or similar items. Previous studies in finding similar users and items have several drawbacks. First, they are based on user-defined similarity measurements, such as Pearson Correlation Coefficient (PCC) or Vector Space Similarity (VSS), which are, for the most part, not adaptive and optimized for specific applications and data. Second, these similarity measures are restricted to symmetric ones such that the similarity between A and B is the same as that for B and A, although symmetry may not always hold in many real world applications. Third, they typically treat the similarity functions between users and functions between items separately. However, in reality, the similarities between users and between items are inter-related. In this paper, we propose a novel unified model for users and items, known as Similarity Learning based Collaborative Filtering (SLCF) , based on a novel adaptive bidirectional asymmetric similarity measurement. Our proposed model automatically learns asymmetric similarities between users and items at the same time through matrix factorization. Theoretical analysis shows that our model is a novel generalization of singular value decomposition (SVD). We show that, once the similarity relation is learned, it can be used flexibly in many ways for rating prediction. To take full advantage of the model, we propose several strategies to make the best use of the proposed similarity function for rating prediction. The similarity can be used either to improve the memory-based approaches or directly in a model based CF approaches. In addition, we also propose an online version of the rating prediction method to incorporate new users and new items. We evaluate SLCF using three benchmark datasets, including MovieLens, EachMovie and Netflix, through which we show that our methods can outperform many state-of-the-art baselines.

Providing high quality recommendations is significant for e-commerce systems to assist users in making effective selection decisions from a plethora of choices. Collaborative filtering (CF) is one of the most well-known and successful techniques to generate recommendations. However, CF suffers from an inherent issue that does not think over the auxiliary information such as item content information. This paper proposes an approach to combine the similarity of auxiliary information with the similarity of items based on user-item ratings. Specifically, due to the high dimensions and linear correlation of the auxiliary information, this paper uses Principal Component Analysis (PCA) to reduce the dimension to improve the predictive accuracy. In addition, Pearson correlation is better than the Euclidean distance as the similarity measurement to predict accuracy of preference value through the analysis of the different experimental results. Experimental results on real-world data sets demonstrate that the effectiveness of our approach.

The fact that a lexeme can appear in various forms causes problems in information retrieval. As a solution to this problem, we have developed methods for automatic root lemmatization, automatic truncation and automatic splitting of compound words. All the methods have as their basis a set of rules which contain information regarding inflected and derived forms of words – and not a dictionary. The methods have been tested on several collections of texts, and have produced very good results. By controlled experiments in text retrieval, we have studied the effects on search results. These results show that both the method of automatic root lemmatization and the method of automatic truncation make a considerable improvement on search quality. The experiments with splitting of compound words did not give quite the same improvement, however, but all the same this experiment showed that such a method could contribute to a richer and more complete search request.

With the explosive growth of digital resources in libraries, the limitations of traditional recommendation systems have become increasingly prominent. The collaborative filtering algorithm is affected by the sparsity of the user-item matrix and is difficult to capture long-tail requirements. The low efficiency of association rule mining leads to serious homogenization of recommendation results. How to break through technical bottlenecks and achieve precise and diverse resource recommendations has become the key to enhancing user experience and promoting the construction of smart libraries. Therefore, a library resource recommendation model that integrates collaborative filtering and association rules is developed to optimize the accuracy and personalization of library resource recommendations. First, the research introduces user attribute features and clustering algorithms to improve collaborative filtering algorithms. Then, a library resource recommendation model is constructed by integrating association rules. Finally, the model is tested. The test results showed that in terms of recommendation diversity, when the number of neighbors was 50, the research model generated 560 unique recommendation items, which was significantly better than traditional content-based recommendation models. In terms of coverage indicators, the research model stabilized at 42%, which was 17% to 30% higher than comparison methods. In real library scenario testing, the research model could identify and recommend six randomly selected user accounts. This research provides an effective technical solution for improving the personalized service level of libraries, which has important practical value for smart libraries.

Collaborative Filtering (CF) is a successful technology that has been implemented in E-commerce recommender systems. However, the risks of shilling attacks have already aroused increasing concerns of the society. Current solutions mainly focus on attack detection methods and robust CF algorithms that have flaws of unassured prediction accuracy. Furthermore, attack detection methods require a threshold to distinguish normal users from fake users and suffer from the problems of false positive if the threshold is too high and false negative if too low. This paper proposes a soft-decision method, Neighbor Selection with Variable-Length Partitions (VLPNS), to reduce false positive rate through marking suspicious fakers instead of deleting them directly such that misclassified normal users can still contribute to the similarity calculation. The method works as follows: First, it gets user's suspicion probability by applying SVM. It then generates partitions of variable sizes from which different numbers of neighbors can be selected by using the bisecting c-means clustering algorithm. Finally, it chooses neighbors considering the user's suspicion degree and similarity with target user at the same time. Theoretical and experimental analysis show that our approach ensures an excellent prediction accuracy against shilling attacks.

Learning expressive representations for high-dimensional yet sparse features has been a longstanding problem in information retrieval. Though recent deep learning methods can partially solve the problem, they often fail to handle the numerous sparse features, particularly those tail feature values with infrequent occurrences in the training data. Worse still, existing methods cannot explicitly leverage the correlations among different instances to help further improve the representation learning on sparse features since such relational prior knowledge is not provided. To address these challenges, in this paper, we tackle the problem of representation learning on feature-sparse data from a graph learning perspective. Specifically, we propose to model the sparse features of different instances using hypergraphs where each node represents a data instance and each hyperedge denotes a distinct feature value. By passing messages on the constructed hypergraphs based on our Hypergraph Transformer (HyperFormer), the learned feature representations capture not only the correlations among different instances but also the correlations among features. Our experiments demonstrate that the proposed approach can effectively improve feature representation learning on sparse features.

Learning to diversify web search results with a Document Repulsion Model