Data preprocessing algorithm for Web Structure Mining

The world wide web is expanding, everyday huge amount of data is added to the web. Finding relevant information in the web is becoming a difficult task. Web Mining is the process of analysing and mining the web to find useful information. By web mining we extract information that are implicitly present in the web. Web Mining is classified into Web Content Mining (WCM), Web Structure Mining (WSM), Web Usage Mining (WUM) based on the type of data mined. Web Structure Mining analyses the structure of the web considering it as a graph. WSM can be used to rank pages present in the web, to improve the efficiency of search engines. This paper discusses about Web Mining, its types, and various ranking algorithms used in Web structure Mining.

The purpose of this paper is to provide a more current evaluation and update of web mining research and techniques available. Current advances in each of the three different types of web mining are reviewed in the categories of web content mining, web usage mining, and web structure mining. For each tabulated research work, we examine such key issues as web mining process, methods/techniques, applications, data sources, and software used. Unlike previous investigators, we divide web mining processes into the following five subtasks: (1) resource finding and retrieving, (2) information selection and preprocessing, (3) patterns analysis and recognition, (4) validation and interpretation, and (5) visualization. This paper also reports the comparisons and summaries of selected software for web mining. The web mining software selected for discussion and comparison in this paper are SPSS Clementine, Megaputer PolyAnalyst, ClickTracks by web analytics, and QL2 by QL2 Software Inc. Applications of these selected web mining software to available data sets are discussed together with abundant presentations of screen shots, as well as conclusions and future directions of the research.

In the present day scenario the World Wide Web (WWW) is an important and popular information search tool. It provides convenient access to almost all kinds of information – from education to entertainment. The main objective of the chapter is to retrieve information from websites and then use the information for website quality analysis. In this chapter information of the website is retrieved through web mining process. Web mining is the process is the integration of three knowledge domains: Web Content Mining, Web Structure Mining and Web Usage Mining. Web content mining is the process of extracting knowledge from the content of web documents. Web structure mining is the process of inferring knowledge from the World Wide Web organization and links between references and referents in the Web. The web content elements are used to derive functionality and usability of the website. The Web Component elements are used to find the performance of the website. The website structural elements are used to find the complexity and usability of the website. The quality assurance techniques for web applications generally focus on the prevention of web failure or the reduction of chances for such failures. The web failures are defined as the inability to obtain or deliver information such as documents or computational results requested by web users. A high quality website is one that provides relevant, useful content and a good user experience. Thus in this chapter, all areas of website are thoroughly studied for analysing the quality of website design.

Web mining is an emerging research area due to the rapid growth of websites. Web mining is classified into Web Content Mining (WCM), Web Usage Mining and Web Structure Mining. Extraction of required information from web page content available on World Wide Web (WWW) is WCM. The WCM is further classified into two categories first category is to directly mine the content on documents and second category is to mine the content using search engine. The mining method focuses on the information extraction and integration. The content of Web may be text, image, audio, video. Web pages typically contain a large amount of information that is not part of the main contents of the pages, like banner advertisements, navigation bars, copyright notices, etc. Such noises on Web pages usually lead to poor results in Web mining. This paper focuses on the problem of Noise free Information retrieval on web pages, which means the pre-processing of Web pages automatically to detect and eliminate noises. This paper proposes an approach for eliminating noises from web pages for the purpose of improving the accuracy and efficiency of web content mining. The main objective of removing noise from a Web Page is to improve the performance of the search. It is very essential to differentiate important information from noisy content that may misguide users' interest. This approach mainly concentrates on removing the following noises in stages: (1) Primary noises-Navigation bars, Panels and Frames, Page Headers and Footers, Copyright and Privacy Notices, Advertisements and other Uninteresting Data such as audio, video, multiple links. (2) Duplicate Contents and (3) Noise Contents according to block importance. The removal of these noises is done by performing three operations. Firstly, using the Block Splitting operation, primary noises are removed and only the useful text contents are partitioned into blocks. Secondly, using simhash algorithm, the duplicate blocks are removed to obtain the distinct blocks. For each block, three parameters namely Keyword Redundancy (KR), Linkword Percentage (LP) and Titleword Relevancy (TR) calculated. Using these three parameters block importance value (BI) is calculated, which is called Simhash algorithm. The importance of the block is then calculated using simhash algorithm. Based on a threshold value the important blocks are selected using sketching algorithm and the keywords are extracted from those important blocks.

Data and web mining involves the use of data mining techniques to extract information from the web in order to gain insights into online behavior, customer preferences, and trends. The process requires data preprocessing to clean and prepare the data for analysis, and pattern recognition to identify relationships and patterns within the data. The benefits of data and web mining include improved decision-making and business growth, while challenges include the large volume of data and ensuring data quality. Web mining can be divided into three main categories: web content mining, web structure mining, and web usage mining. Web content mining involves extracting information from the content of web pages, such as text, images, and videos. Web structure mining involves analyzing the links between web pages to identify patterns and relationships. Web usage mining involves analyzing user behavior on the web, such as clickstream data and session logs, to extract useful information. KEYWORDS: data mining, web mining, information extraction, data preprocessing, pattern recognition, online behavior, customer preferences,

Web usage mining is the branch of web mining that deals with mining of data over the web. Web mining can be categorized as web content mining, web structure mining, web usage mining. In this paper, we have summarized the web usage mining results executed over the user tool WMOT (web mining optimized tool) based on the WEKA tool that has been used to apply various classification algorithms such as Naïve Bayes, KNN, SVM and tree based algorithms. Authors summarized the results of classification algorithms on WMOT tool and compared the results on the basis of classified instances and identify the algorithms that gives better instances accuracy.

The raw data obtained from the logs may be noisy, incomplete, and inconsistent that’s why data pre-processing is an essential step in data mining. The quality of data plays a vital role during the evaluation process. The results of the evaluation process primarily depend upon the quality of the data input. So, data pre-processing is the primary and most crucial step before knowledge discovery. This paper is based on two main steps- data pre-processing techniques and results after applied data pre-processing on mutual funds’ data. Data pre-processing transforms the raw data into a structured, understandable format. Moreover, data pre-processing performs not only the transformation of data but also makes it understandable according to need. It is mainly divided into four steps, i.e., data integration, data cleaning, data transformation, and data reduction. This paper takes the fifteen-year NAV data of twenty mutual funds for analysis propose. This paper explains several techniques of data pre-processing to transform the raw data into an understandable format.

The rising popularity of electronic commerce makes data mining an indispensable technology for business competitiveness. The World Wide Web provides abundant raw data in the form of Web access logs, Web transaction logs and Web user profiles. Without data mining tools, it is impossible to make any sense of such massive data. We focus on Web usage mining because it deals most appropriately with understanding user behavioral patterns which is the key to successful customer relationship management. Previous work dealt separately with specific issues of Web usage mining and made assumptions without taking a holistic view and thus, had limited practical applicability. We formulate a novel and more holistic version of Web usage mining termed transactionized logfile mining (TRALOM) to effectively and correctly identify transactions as well as to mine useful knowledge from Web access logs. We also introduce a new data structure, called the WebTrie, to efficiently hold useful preprocessed data so that TRALOM can be done in an online and incremental fashion. Experiments conducted on real Web server logs verify the usefulness and practicality of our proposed techniques.

The prolific growth of web-based applications and the enormous amount of data involved therein led to the development of techniques for identifying patterns in the web data. Web mining refers to the application of data mining techniques to the World Wide Web. Web usage mining is the process of extracting useful information from web server logs based on the browsing and access patterns of the users. The information is especially valuable for business sites in order to achieve improved customer satisfaction. Based on the user’s needs, Web Usage Mining discovers interesting usage patterns from web data in order to understand and better serve the needs of the web based application. Web Usage Mining is used to discover hidden patterns from weblogs. It consists of three phases like Preprocessing, pattern discovery and Pattern analysis. In this paper, we present each phase in detail, the process of extracting useful information from server log files and some of application areas of Web Usage Mining such as Education, Health, Human-computer interaction, and Social media.

Internet is the biggest source of information which is easily accessible but unstructured. Web mining - is the application of data mining techniques to discover patterns from the Web. According to analysis targets, web mining can be divided into three different types, which are Web usage mining, Web content mining and Web structure mining. [3] Web usage mining reveals about user's navigation pattern at any website through analyzing web logs. In this study we have tried to analyze the users online job seeking tendency pattern through two available web-based tools viz. Google Trends and Google Zeitgeist.

this paper provides an in-depth survey of challenges in the design of intelligent information retrieval systems, pointing out some similarities and differences in the core data mining and web based search operations. The procedures for evaluation of search engine performance and implicit feedback of the user with respect to a search result are studied with references to the different algorithms. We have proposed a novel neural satisfaction based feedback vector in contrast to the existing activity pattern based feedback as a future research direction in Intelligent IR. We addressed the select research work in the area of soft information retrieval using fuzzy sets, artificial neural networks, genetic algorithms and probabilistic information retrieval. As an instance of information retrieval, web mining is a set of operations to retrieve relevant documents from preprocessed, crawled and indexed web, and it can be categorized into more specialized tasks of web content mining, web structure mining and web usage mining. We have given a survey of the important reviews on topic of web mining and its associated tasks. On the basis of Identified challenges in information retrieval in general, and web mining in particular, we have concentrated on applicability of soft computing techniques and their hybrids in web mining, the performance related issues of select solutions, future of web mining and the next generation user's expectations from a search engine.

Over the last decade, we have witnessed an explosive growth in the information available on the Web. Today, Web browsers provide easy access to myriad sources of text and multimedia data. Search engines index more than a billion pages and finding the desired information is not an easy task. This profusion of resources has prompted the need for developing automatic mining techniques on Web, thereby giving rise to the term “Web mining” (Pal, Talwar, & Mitra, 2002). Web mining is the application of data mining techniques on the Web for discovering useful patterns and can be divided into three basic categories: Web content mining, Web structure mining, and Web usage mining. Web content mining includes techniques for assisting users in locating Web documents (i.e., pages) that meet certain criteria, while Web structure mining relates to discovering information based on the Web site structure data (the data depicting the Web site map). Web usage mining focuses on analyzing Web access logs and other sources of information regarding user interactions within the Web site in order to capture, understand and model their behavioral patterns and profiles and thereby improve their experience with the Web site. As citizens requirements and needs change continuously, traditional information searching, and fulfillment of various tasks result to the loss of valuable time spent in identifying the responsible actor (public authority) and waiting in queues. At the same time, the percentage of users who acquaint with the Internet has been remarkably increased (Internet World Stats, 2005). These two facts motivate many governmental organizations to proceed with the provision of e-services via their Web sites. The ease and speed with which business transactions can be carried out over the Web has been a key driving force in the rapid growth and popularity of e-government, e-commerce, and e-business applications. In this framework, the Web is emerging as the appropriate environment for business transactions and user-organization interactions. However, since it is a large collection of semi-structured and structured information sources, Web users often suffer from information overload. Personalization is considered as a popular solution in order to alleviate this problem and to customize the Web environment to users (Eirinaki & Vazirgiannis, 2003). Web personalization can be described, as any action that makes the Web experience of a user personalized to his or her needs and wishes. Principal elements of Web personalization include modeling of Web objects (pages) and subjects (users), categorization of objects and subjects, matching between and across objects and/or subjects, and determination of the set of actions to be recommended for personalization. In the remainder of this article, we present the way an e-government application can deploy Web mining techniques in order to support intelligent and personalized interactions with citizens. Specifically, we describe the tasks that typically comprise this process, illustrate the future trends, and discuss the open issues in the field.

Over the last decade, we have witnessed an explosive growth in the information available on the Web. Today, Web browsers provide easy access to myriad sources of text and multimedia data. Search engines index more than a billion pages and finding the desired information is not an easy task. This profusion of resources has prompted the need for developing automatic mining techniques on Web, thereby giving rise to the term “Web mining” (Pal, Talwar, & Mitra, 2002). Web mining is the application of data mining techniques on the Web for discovering useful patterns and can be divided into three basic categories: Web content mining, Web structure mining, and Web usage mining. Web content mining includes techniques for assisting users in locating Web documents (i.e., pages) that meet certain criteria, while Web structure mining relates to discovering information based on the Web site structure data (the data depicting the Web site map). Web usage mining focuses on analyzing Web access logs and other sources of information regarding user interactions within the Web site in order to capture, understand and model their behavioral patterns and profiles and thereby improve their experience with the Web site. As citizens requirements and needs change continuously, traditional information searching, and fulfillment of various tasks result to the loss of valuable time spent in identifying the responsible actor (public authority) and waiting in queues. At the same time, the percentage of users who acquaint with the Internet has been remarkably increased (Internet World Stats, 2005). These two facts motivate many governmental organizations to proceed with the provision of e-services via their Web sites. The ease and speed with which business transactions can be carried out over the Web has been a key driving force in the rapid growth and popularity of e-government, e-commerce, and e-business applications. In this framework, the Web is emerging as the appropriate environment for business transactions and user-organization interactions. However, since it is a large collection of semi-structured and structured information sources, Web users often suffer from information overload. Personalization is considered as a popular solution in order to alleviate this problem and to customize the Web environment to users (Eirinaki & Vazirgiannis, 2003). Web personalization can be described, as any action that makes the Web experience of a user personalized to his or her needs and wishes. Principal elements of Web personalization include modeling of Web objects (pages) and subjects (users), categorization of objects and subjects, matching between and across objects and/or subjects, and determination of the set of actions to be recommended for personalization. In the remainder of this article, we present the way an e-government application can deploy Web mining techniques in order to support intelligent and personalized interactions with citizens. Specifically, we describe the tasks that typically comprise this process, illustrate the future trends, and discuss the open issues in the field.

The World Wide Web (WWW) has grown quickly in the past two decades from a small research community to the biggest and most popular infrastructure for communication, information dissemination, search, social interaction and commerce. The continuous growth in size and use of the WWW creates a need for methods to process these wicked volumes of data. Web Intelligence (WI), as a research direction, has a broad agenda to deal with the issues that arise around the WWW phenomenon. WI corresponds to research and development to explore the fundamental roles, artificial intelligence and advanced information technology on the web-empowered systems, services, and activities. In this context, WI is concerning of mining in web data and user behaviors. This creates a demand for using mining technologies to search large volume of data for gaining hidden knowledge. This hidden knowledge helps in gaining competitive advantages, better customers’ relationships, and even fraud detection. Achieving the intelligence to the web enhances Business Intelligence (BI) for the enterprises. Web Usage Mining (WUM) and Web Opinion Mining (WOM) are considered a leading mining approaches in mining the user behavior and reviews. Most of previous studies depended in building intelligent web on mining and analyzing user’s profiles or opinions separately. This becomes not fair enough and cause of limitations. This limitations could be narrowed if both of user preferences and opinions are considered in building recommendations. The paper proposes a framework for achieving the intelligence via using WUM and WOM. The proposed framework would contribute in solving the problem. The paper also surveys the background of using the intelligent web as an approach to enhance the Business Intelligence (BI) applications

With the proliferation of very large data repositories hidden behind web interfaces, e.g., keyword search, form-like search and hierarchical/graph-based browsing interfaces for Amazon.com, eBay.com, etc., efficient ways of searching, exploring and/or mining such web data are of increasing importance.