Personalized Recommendation System on Hadoop and HBase

Nowadays, large volumes of valuable data of different degrees of veracity can be easily generated or collected at a high velocity from a wide variety of data sources. Embedded in these big data are implicit, previously unknown and potentially useful information and knowledge. Hence, big data analytics are in demand as they assist users to analyze the big data and discover new knowledge. This in turn helps users gain insight into big data, obtain useful information, and make personalized recommendation. In this paper, we focus on big data analytics on music-i.e., music analysis-for personalized music recommendation systems. In particular, we present a system that exploits the frequent tags attached to the tracks in an user's listening history and a hidden Markov model (HMM) for generating predictions and thus making personalized recommendation. Knowledge learned from this personalized music recommendation can be transferred to building other personalized recommendation systems.

In the big data environment, there are many new demands with data storage, the traditional data storage mode which based on relational database cannot meet these needs, many application systems tend to use NoSQL to solve the problem of big data storage. However, NoSQL gave up on the relationship between operation support, so that part of existing application system is difficult to use the simple way of transplantation. This thesis with reference to some typical existing big data storage scheme, we presents a big data compatible with relational storage model storage scheme, this scheme can not only meet the big data storage requirements, but also can support the most relational operation, so that the original system based on relational database can easily be ported to new storage schemes in.

There is a growing opportunity for self-improvement within the realm of higher education, represented by the combination of big data technologies and higher education. The Education Big Data Processing Framework (EBDPF) is the name of the platform that is the subject of this article, which focuses on the development of an advanced big data processing platform. In order to process huge datasets in an effective manner, the platform makes use of Hadoop's big data storage architecture, in addition to Hive, Flume, and Sqoop for data collection and synchronization, respectively. In this study, the implementation of standard data mining techniques is investigated through the use of MapReduce programming. More specifically, the study investigates the execution efficiency and scalability of these algorithms within the Hadoop platform. The data clustering task in data mining is chosen as a representative problem in this study so that the effectiveness of the EBDPF may be evaluated accordingly. An implementation of the clustering job that is based on MapReduce is created and tested on the Hadoop platform. Extensive experiments are conducted with a range of cluster sizes and data sizes. The findings provide more evidence that Hadoop distributed systems are more efficient and successful than traditional methods when it comes to managing data mining applications. The extended performance analysis of computing power not only provides evidence that the platform is currently effective, but it also provides a signal that it has a significant amount of potential that has not yet been utilized. At the convergence of big data technology and higher education, the EBDPF emerges as a strong and promising framework, demonstrating considerable steps in accelerating data processing within educational contexts. In conclusion, the EBDPF is a framework that demonstrates significant progress.

Among the most revolutionary technologies are big data analytics, artificial intelligence (AI) and robotics, machine learning (ML), cybersecurity, blockchain technology, and cloud computing. The project was focused on how to create a Big Data Analytics Architecture Framework to increase the production capability and global trade for Sub-Saharan Africa’s oilseeds and textile industries (SSA). The infrastructure, e-commerce, and disruptive technologies in the oilseeds and textile industries, as well as global e-commerce, all demand large investments. The pragmatic paradigm served as the foundation for the research approach. This study employed a review of the literature, document analysis, and focus groups. For the oilseeds and textile sectors in SSA, a Big Data analytics architectural framework was created. The Hadoop platform was created as a framework for big data analytics. The open-source Hadoop platform offers the analytical tools and computing capacity needed to handle such massive data volumes. It supports E-commerce and is based on the Hadoop platform, which offers the analytical tools and computing power needed to handle such massive data volumes. The low rate of return on investments made in breeding, seed production, processing, and marketing limits the competitiveness of the oil crop or legume seed markets.

Purpose – The purpose of this paper is not only to design a more effective recommendation system for libraries, but also to eliminate many of the weaknesses found in the existing library recommender systems.Design/methodology/approach – A novel library recommender system was designed for English collections by integrating personal ontology model and collaborative filtering model with domain specification.Findings – The trend of the traditional library is evolving toward that of digital library. The personal ontology recommender (PORE) system offers a friendly user interface and provides several personalized services.Research limitations/implications – This system is only implemented and tested in the Library of National Chung Hsing University in Taiwan.Originality/value – The paper demonstrates a good methodology to offer an active, effective, and personalized recommendation system for library patrons.

More and more tourists are sharing their travel feelings and posting their real experiences on the Internet, generating tourism big data. Online travel reviews can fully reflect tourists’ emotions, and mining and analyzing them can provide insight into the value of them. In order to analyze the potential value of online travel reviews by using big data technology and machine learning technology, this paper proposes an improved support vector machine (SVM) algorithm based on travel consumer sentiment analysis and builds an Hadoop Distributed File System (HDFS) system based on Map-Reduce model. Firstly, Internet travel reviews are pre-processed for sentiment analysis of the review text. Secondly, an improved SVM algorithm is proposed based on the main features of linear classification and kernel functions, so as to improve the accuracy of sentiment word classification. Then, HDFS data nodes are deployed on the basis of Hadoop platform with the actual tourism application context. And based on the Map-Reduce programming model, the map function and reduce function are designed and implemented, which greatly improves the possibility of parallel processing and reduces the time consumption at the same time. Finally, an improved SVM algorithm is implemented under the built Hadoop platform. The test results show that online travel reviews can be an important data source for travel big data recommendation, and the proposed method can quickly and accurately achieve travel sentiment classification.

Big data is the biggest challenges as we need huge processing power system and good algorithms to make an decision. We need Hadoop environment with pig hive, machine learning and hadoopecosystem components. The data comes from industries. Many devices around us and sensor, and from social media sites. According to McKinsey There will be a shortage of 15000000 big data professionals by the end of 2020. There are lots of technologies to solve the problem of big data Storage and processing. Such technologies are Apache Hadoop, Apache Spark, Apache Kafka, and many more. Here we analyse the processing speed for the 4GB data on cloudx lab with Hadoop mapreduce with varing mappers and reducers and with pig script and Hive querries and spark environment along with machine learning technology and from the results we can say that machine learning with Hadoop will enhance the processing performance along with with spark, and also we can say that spark is better than Hadoop mapreduce pig and hive, spark with hive and machine learning will be the best performance enhanced compared with pig and hive, Hadoop mapreduce jar.

With the constant development of socialist modernization construction in our country, our country's computer information technology has made effective progress, and makes the world into an information age. People's production and life style experience a series of change. However, with the emergence of information diversity and multi-user mode, traditional computer information technology already can't satisfy people's needs, and the development and application of cloud computing technology come to the stage. This article focuses on analyzing the big data user behavior engine design under cloud computing technology, covering big data flow management, multi-user system design, and other aspects to explore the effect of the system test. In recent years, China's mobile Internet technology has made full development, which makes the Internet operators in our country face a new development opportunity and began from traffic management to flow management. The law of user behavior is analyzed to explore the real demand of market and the users. In order to fully meet the changing needs of users, operators must constantly develop and launch new products and strengthen the function of computer technology. Cloud computing technology is such kind of engine system which can satisfy the analysis and processing of huge amounts of data. RESEARCH ON CLOUD COMPUTING SYSTEM OVERALL DESIGN The overall architecture of cloud computing system This study mainly uses cloud computing technology’s huge amounts of data computing to set up complete mobile Internet data mining analysis system. Realize the analysis for Internet user behavior engine, and according to the user's preferences online habits and behavior, provide users with targeted personalized service and form a unified organic whole of data collection, analysis and service type and marketing strategy to improve enterprise's marketing efficiency. In addition, cloud computing system mainly realizes data collection with the help of FTP server, and makes distributed computing and data batch processing at the system interface. Large data shall be deposited in Hbase database. The system can not only realize mass data storage, but those unstructured data storage. Then through Hive integration layer and summary EIL processing, use MapReduce data analysis model to transmit the processing result into the database. The system’s overall structure is shown in Figure 1: 3rd International Conference on Materials Engineering, Manufacturing Technology and Control (ICMEMTC 2016) © 2016. The authors Published by Atlantis Press 1351 Figure 1 The overall architecture of cloud computing system System topology and function distribution System topology mainly refers that a server is taken as Hapdoop platform’s master node server, and the others are Hapdoop platform’s node servers. In general, node server can be dynamically extended according to the actual need, and the master node server not only assigns tasks and flow to node server, but also monitors the work execution of the node server. More than one node server participates to complete the task, and it can improve data processing efficiency. The master node server’s software running status is shown in Figure 2: Figure 2 Master node server function structure

A Personalized Recommender System from Probabilistic Relational Model and Users’ Preferences

Recommender System was widely used in commercial websites for the past few years. These systems track past activities of customers and recommend them the relevant items. The emergence of E-learning activities over a few decades develops a variety of E-learning content available for virtual learning environments (VLE). A large amount of learning objects is present in E-learning repositories. Dealing with problems of the diversity of data, Educational Recommender System (ERS) plays a vital role in the educational sector. Educational recommender systems track the learners' past activities, know the users' preferences, assist the educators and learners, provide relevant content to learners, and enhance their learning outcomes. A personalized recommender system will intensify the learners' interest in particular content and reduce the course dropout rate. The recommender system makes the decision-making process for choosing the appropriate content easy for learners. ERS uses various approaches and technologies for assisting the learners' and helps them to run their learning process smoothly. Collaborative filtering, Content-based, and knowledge-based are the basic techniques of recommender systems. The research shows that a combination of these approaches will give more effective and efficient results. This mixing of approaches refers to the hybridization techniques. Traditional approaches with deep learning networks will improve the recommendations and provide results with higher accuracy. This paper provides how E-learning support recommender systems produce recommendations using different techniques. The technical results of recommender techniques help to find the best approach for making a recommender system in the future.

A distributed remote sensing data storage system based on Hadoop platform is designed to solve the problem of low storage efficiency and low security of mass remote sensing data. Uesd virtualization technology to PC as hardware based virtual computing node the Hadoop Cluster required, without increasing the dedicated server clusters and does not change the original PC use conditions, built the Hadoop platform by the virtual machine cluster. Using LandSat-8 satellite image data as the experimental object, through the design of file tree structure, the remote sensing image data according to the band and split into several blocks of image coordinates, combined with the Hadoop distributed file system (HDFS) characteristics of block storage data block, the remote sensing data stored in the distribution of the whole virtual computer cluster in. Build Hadoop platform remote sensing data distributed storage system with low cost.

There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.

Very Large Online Platforms (VLOPs) use personalized recommender systems to optimize their main performance metric: attention-based user engagement. In doing so, these systems might however amplify systemic risks by promoting controversial or polarizing content, thereby exacerbating issues such as misinformation, societal polarization, and the manipulation of civic discourse. To mitigate these risks, regulations such as the European Union’s Digital Services Act (DSA) mandate increased data access and transparency, including for the auditing of personalized recommender systems. However, the data access provided by VLOPs remains limited—often restricted to specific user demographics, aggregate statistics, or curated datasets—hindering meaningful oversight. Consequently, new methods are needed to audit recommender systems effectively at the user level. In this paper, based on an analysis of the legal context and technical alternatives for data access, we present SOAP, the System for Observing and Analyzing Posts. SOAP is an open-source framework for auditing recommender systems using sock-puppet accounts. It enables fine-grained user-level analysis beyond the constrained data access typically provided by platforms. We detail SOAP’s technical implementation and evaluate its ability to scrutinize systemic risks. Additionally, we tested SOAP in a workshop with over 100 participants and observed a measurable increase in participants’ algorithmic literacy. This demonstrates SOAP’s potential not only for research and regulatory auditing, but also as an educational framework to foster public awareness of algorithmic influence.

The rapid development of Internet and mobile communication, smart phone, tablet computer and other mobile devices become widespread, they have become an indispensable part of life, changing the human lifestyle, way of working and learning style, it brings great convenience and fun to us. This paper mainly discusses the status of personalized recommendation system, mobile application and analyse the main problems of recommendation system. This research proposes a rising mobile application domain, which uses web crawling technique to get explicit feedback from a real-world mobile application market. The explicit feedback is the users’ ratings data on Apps and the corresponding category information. To establish a mobile application recommendation model based on user interest and categorization information and propose a novel collective matrix factorization algorithm. Through the crawled data, the improved algorithm, and the traditional collaborative. The traditional mobile application recommendation systems are mostly based on keyword search, popularity, download usage and categories, thus they do not make personalized recommendation to users, and invalid to find applications that users are interested in. Through the user’s historical behaviour, a personalized recommendation system can be established to recommend the applications that fit user’s interests effectively. This paper is based on the collective matrix factorization using user interest such as user’s ratings and category information, designs a personalized recommendation system for mobile applications by using the relationship between them.

In the traditional e-commerce system, the keyword matching algorithm is used to implement the function of commodity search, and only the goods containing the customer input keyword can be obtained. In view of this limitation, a personalized recommendation system for e-commerce based on big data analysis is studied. The use of improving the search algorithm for goods, personalization system rules of e-commerce and text matching algorithm are introduced, so in the search of goods, it not only displays the exact matching of goods, but provides the goods similar to their requirements, for reference. The research shows that the personalized recommendation system of e-commerce based on big data analysis can not only increase the transaction opportunities, explore potential customers, but also improve the level of personalized service. This is very important for the enterprise economy and the personalized development of e-commerce.