Distributed fuzzy clustering algorithm for mixed-mode data in Apache SPARK

Partition-and-merge based fuzzy genetic clustering algorithm for categorical data

Image segmentation plays an important role in machine vision, image recognition, and imaging applications. Based on the fuzzy c-means clustering algorithm, a kernel-distance-based intuitionistic fuzzy c-means clustering (KIFCM) algorithm is proposed. First, a fuzzy complement operator is used to generate the membership degree whereby the hesitation degree of intuitionistic fuzzy set is generated; second, a kernel-induced function is used to calculate the distance from each point to the cluster center instead of the Euclidean distance; third, a new objective function that includes the hesitation degree is established, and the optimization of the objective function results in new iterative expressions for the membership degree and the cluster center. The proposed KIFCM algorithm is compared with the fuzzy c-means clustering (FCM) algorithm, the kernel fuzzy c-means clustering (KFCM) algorithm, and the intuitionistic fuzzy c-means clustering (IFCM) algorithm in segmenting five images. The experimental results verify the effectiveness and superiority of our proposed KIFCM algorithm.

Clustering method, which groups thousands of genes by their similarities of expression levels, has been used for identifying unknown functions of genes. Fuzzy clustering method that is one category of clustering assigns one sample to multiple groups according to their membership degrees. It is more appropriate than hard clustering algorithms for analyzing gene expression profiles since single gene might involve multiple genetic functions. However, general clustering methods have problems that they are sensitive to initialization and can be trapped into local optima. To solve the problems, we propose an evolutionary fuzzy clustering algorithm with knowledge-based evaluation. It uses a genetic algorithm for clustering and prior knowledge of data for evaluation. Yeast cell-cycle dataset has been used for experiments to show the usefulness of the proposed method.KeywordsFuzzy ClusterMembership DegreeCluster PartitionAnalyze Gene Expression ProfileFuzzy Cluster MethodThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The sheer boom in data creation from multiple sources beginning from IoT sensors to social media has changed the technology landscape, rendering predictive analytics in big data environments more important than ever before. This work suggests a hybrid machine learning model for improving predictive accuracy, computational performance, and scalability in big data environments. The Model employs Random Forest (RF) for strong feature selection and Gradient Boosted Decision Trees (GBDT) for enhanced classification accuracy, both optimised using Apache Spark's MLlib in a distributed setting. The Model takes advantage of hybridisation to beat the single-algorithm model drawback in accommodating high-dimensional, heterogeneous data. The architecture is designed to handle data in real time and includes a dynamic pre-processing layer, parallel training pipeline, and continuous evaluation modules. Performance metrics like accuracy, precision, recall, F1 score, and AUC are used to validate performance against benchmark data sets. System-level metrics like latency, Throughput, and scalability are also monitored to validate usability for real-world deployments. Experiments using healthcare and e-commerce industry datasets yield better prediction power and operational effectiveness than single ML models. Visualisation using scatter plots and 3D graphs shows sharp jumps in model accuracy and processing time for different volumes of data. It is a scalable, adaptable, and precise predictive analytics tool for the big data era. Ease of use of the solution with Apache Hadoop and Spark deployment is a willingness to be utilised at the enterprise level.

Forecasting activities play an important role in our daily life. In recent years, fuzzy time series (FTS) methods were developed to deal with forecasting problems. FTS attracted researchers because of its ability to predict the future values in some critical situations where most standard forecasting models are doubtfully applicable or produce bad fittings. However, some critical issues in FTS are still open; these issues are often subjective and affect the accuracy of forecasting. In this paper, we focus on improving the accuracy of FTS forecasting methods. The new method integrates the fuzzy clustering and genetic algorithm with FTS to reduce subjectivity and improve its accuracy. In the new method, the genetic algorithm is responsible for selecting the proper model. Also, the fuzzy clustering algorithm is responsible for fuzzifying the historical data, based on its membership degrees to each cluster, and using these memberships to defuzzify the results. This method provides better forecasting accuracy when compared with other extant researches.

A fuzzy minimax clustering model and its applications

Hyperspectral image (HSI) clustering has been widely used in the field of remote sensing. However, most traditional clustering algorithms are not suitable for dealing with large-scale HSI due to their low clustering performance and high computational complexity. In this letter, we propose a novel fuzzy clustering algorithm, called fuzzy embedded clustering based on the bipartite graph (FECBG), to efficiently deal with large-scale HSI clustering problems. First, we propose the FECBG method that incorporates fuzzy clustering with the nonnegative regularization term based on the bipartite graph into a unified model, which has good clustering performance and reduces the sensitivity of fuzzy clustering to the initial cluster centers. Second, we adopt the fast spectral embedded method to obtain the low-dimensional representation of HSI data, to reduce the computational complexity. At last, we add the nonnegative regularization term based on the bipartite graph to fuzzy clustering, to constrain the solution space of the fuzzy membership matrix. Experimental results on several HSI data sets have demonstrated the efficiency and effectiveness of the proposed FECBG algorithm.

This paper introduces a prototype web search results clustering engine that enhances search results by performing fuzzy clustering on web documents returned by conventional search engines, as well as ranking the results and labeling the resulting clusters. This is done using a fuzzy transduction-based clustering algorithm (FTCA), which employs a transduction-based relevance model (TRM) to generate document relevance values. These relevance values are used to cluster similar documents, rank them, and facilitate a term frequency based label generator. The membership degrees of documents to fuzzy clusters also facilitates effective detection and removal of overly similar clusters. FTCA is compared against two other established web document clustering algorithms: Suffix Tree Clustering (STC) and Lingo, which are provided by the free open source Carrot <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Document Clustering Workbench. To measure cluster quality, an extended version of the classic precision measurement is used to take into account relevance and fuzzy clustering, along with recall and F1 score. Results from testing on five different datasets show a considerable clustering quality and performance advantage over STC and Lingo in most cases.

Fuzzy clustering: More than just fuzzification

Hard and fuzzy clustering algorithms are part of the partition-based clustering family. They are widely used in real-world applications to cluster numerical and categorical data. While in hard clustering an object is assigned to a cluster with certainty, in fuzzy clustering an object can be assigned to different clusters given a membership degree. For both types of method an entropy can be incorporated into the objective function, mostly to avoid solutions raising too much uncertainties. In this paper, we present an extension of a fuzzy clustering method for categorical data using fuzzy centroids. The new algorithm, referred to as Categorical Fuzzy Entropy (CFE), integrates an entropy term in the objective function. This allows a better fuzzification of the cluster prototypes. Experiments on ten real-world data sets and statistical comparisons show that the new method can efficiently handle categorical data.

This research examines what impact the Big Data Processing Framework (BDPF) has on Artificial Intelligence (AI) applications within Corporate Marketing Communication (CMC), and thereby the research question stated is: What is the potential impact of the BDPF on AI applications within the CMC tactical and managerial functions? To fulfill the purpose of this research, a qualitative research strategy was applied, including semi-structured interviews with experts within the different fields of examination: management, AI technology and CMC. The findings were analyzed through performing a thematic analysis, where coding was conducted in two steps. AI has many useful applications within CMC, which currently mainly are of the basic form of AI, so-called rule-based systems. However, the more complicated communication systems are used in some areas. Based on these findings, the impact of the BDPF on AI applications is assessed by examining different characteristics of the processing frameworks. The BDPF initially imposes both an administrative and compliance burden on organizations within this industry, and is particularly severe when machine learning is used. These burdens foremost stem from the general restriction of processing personal data and the data erasure requirement. However, in the long term, these burdens instead contribute to a positive impact on machine learning. The timeframe until enforcement contributes to a somewhat negative impact in the short term, which is also true for the uncertainty around interpretations of the BDPF requirements. Yet, the BDPF provides flexibility in how to become compliant, which is favorable for AI applications. Finally, BDPF compliance can increase company value, and thereby incentivize investments into AI models of higher transparency. The impact of the BDPF is quite insignificant for the basic forms of AI applications, which are currently most common within CMC. However, for the more complicated applications that are used, the BDPF is found to have a more severe negative impact in the short term, while it instead has a positive impact in the long term.

Data analytic applications built upon big data processing frameworks such as Apache Spark are an important class of applications. Many of these applications are not latency-sensitive and thus can run as batch jobs in data centers. By running multiple applications on a computing host, task co-location can significantly improve the server utilization and system throughput. However, effective task co-location is a non-trivial task, as it requires an understanding of the computing resource requirement of the co-running applications, in order to determine what tasks, and how many of them, can be co-located. State-of-the-art co-location schemes either require the user to supply the resource demands which are often far beyond what is needed; or use a one-size-fits-all function to estimate the requirement, which, unfortunately, is unlikely to capture the diverse behaviors of applications.

Apache Spark is a leading open-source data processing engine used for batch processing, machine learning, stream processing, and large-scale SQL (structured query language).It has been designed to make big data processing quicker and easier.Since its inception, Spark has gained huge popularity as a big data processing framework and is extensively used by different industries and businesses that are dealing with large volumes of data.This paper will exhibit actionable solutions to maximize our chances of reducing computation time by optimizing Spark jobs.The strategy lays out different run stages, wherein each run stage builds upon the previous and improves the computation time by making new enhancements and recommendations.

Advertisements(Ads) are the main revenue earner for Television (TV) broadcasters. As TV reaches a large audience, it acts as the best media for advertisements of products and services. With the emergence of digital TV, it is important for the broadcasters to provide an intelligent service according to the various dimensions like program features, ad features, viewers' interest and sponsors' preference. We present an automatic ad recommendation algorithm that selects a set of ads by considering these dimensions and semantically match them with programs. Features of the ad video are captured interms of annotations and they are grouped into number of predefined semantic categories by using a categorization technique. Fuzzy categorical data clustering technique is applied on categorized data for selecting better suited ads for a particular program. Since the same ad can be recommended for more than one program depending upon multiple parameters, fuzzy clustering acts as the best suited method for ad recommendation. The relative fuzzy score called "degree of membership" calculated for each ad indicates the membership of a particular ad to different program clusters. Subjective evaluation of the algorithm is done by 10 different people and rated with a high success score.

As generalization ability of neural network was restricted by overfitting problem in the network's training. Early stopping algorithm based on fuzzy clustering was put forward to solve this problem in this paper. Subtractive clustering and fuzzy c-means clustering (FCM) were combined to realize optimal division of training set, validation set and test set. How to realize this algorithm in backpropagation (BP) network by utilizing neural network toolbox and fuzzy logic toolbox in MATLAB was dwelled on. Early stopping algorithm based on fuzzy clustering and other early stopping algorithms were applied in function approximation and pattern recognition problems in validation experiments. Experiments results indicate that early stopping algorithm based on fuzzy clustering has higher precision in comparison to other early stopping algorithms. Outputs of training set, validation set and test set are more accordant.