Submodular Maximization with Nearly Optimal Approximation, Adaptivity and Query Complexity

Submodular function optimization is a fundamental tool in modeling complex interactions in machine learning and graph mining problems. We propose to study constrained submodular optimization to improve the current state of the art. Our goals are to design evolutionary algorithms with stronger approximation guarantees, the study of submodular maximization under submodular constraints, fairness in submodular optimization, and k-submodular optimization. We begin by exploring a basic submodular maximization problem in the context of viral marketing/information diffusion in social networks. From there, we broaden our scope to include a broader range of scenarios and formulate them into more optimization problems. Looking ahead, we plan to tackle scalable submodular optimization problems in fairness, dynamic constraints, dynamic streams, and distributed fashion. Lastly, we discuss a series of real-world applications that can be formulated as submodular optimization problems. In the future, we aim to apply algorithmic ideas to solve more real-world problems.

Submodularity is one of the most well-studied properties of problem classes in combinatorial optimization and many applications of machine learning and data mining, with strong implications for guaranteed optimization. In this thesis, we investigate the role of submodularity in provable non-convex optimization and validation of algorithms. A profound understanding which classes of functions can be tractably optimized remains a central challenge for non-convex optimization. By advancing the notion of submodularity to domains (termed continuous submodularity), we characterize a class of generally non-convex and non-concave functions -- submodular functions, and derive algorithms for approximately maximizing them with strong approximation guarantees. Meanwhile, submodularity captures a wide spectrum of applications, ranging from revenue maximization with general marketing strategies, MAP inference for DPPs to mean field inference for probabilistic log-submodular models, which renders it as a valuable domain knowledge in optimizing this class of objectives. Validation of algorithms is an information-theoretic framework to investigate the robustness of algorithms to fluctuations in the input/observations and their generalization ability. We investigate various algorithms for one of the paradigmatic unconstrained submodular maximization problem: MaxCut. Due to submodularity of the MaxCut objective, we are able to present efficient approaches to calculate the algorithmic information content of MaxCut algorithms. The results provide insights into the robustness of different algorithmic techniques for MaxCut.

Optimization of complex mechanics simulations with object-oriented software designs

Synthetic scene generation models depend on the repeated evaluation or iteration of often computationally expensive functions to create 'texture' or structure in a modeled natural scene. The computational burden of 'texture' function evaluation is dependent upon the spatial frequency response characteristics of the optical system through which the modeled scene is to be 'propagated.' All real 'cameras' utilize detector elements of finite size to transduce the resulting MXN pixels of the image. Two general situations arise in modeling the effects of cameras (sensors) on the imaged scene. The first is the 'oversampled' case where the Nyquist frequency of the detector spacing exceeds the cutoff of the optics transfer (aperture) function. The 'undersampled' case is when such Nyquist frequency is below the aperture cutoff spatial frequency, which results in aliasing. The application of brute force in the oversampled case results in MXN scene 'texture' function evaluations, while for the undersampled case, (mM)X(nN) (m, n > 1) points are required. Several computationally efficient methods significantly reduce the stated brute force computational burden for these two cases. This paper discusses three such methods. For the oversampled case, a correct and efficient method is 2D sample 'interpolation,' in the multirate digital signal processing sense. This method expressly avoids signal aliasing caused by simpler but inappropriate bilinear interpolation of the sparser set of (mM)+(nN) (m, n fractions < 1) scene samples onto the MXN imaged scene. The second and third techniques discussed are applicable to the undersampled case. Each relies upon MXN scene 'texture' function evaluations. The second technique extrapolates the frequency spectrum of the MXN grid with a synthetic spectrum beyond Nyquist which follows the 1/f(beta ) decrease in power typical of natural (fractal) textures. The third technique, 'fractal interpolation,' operates in the spatial domain where spatial detail, generated from the computed ('texture' function) MXN grid onto a larger (nM)X(nN) grid, is synthesized at the same local fractal dimension as that of the MXN 'undersampled' data. In both cases, the synthesized frequencies above the camera Nyquist are 'folded back' in the spectral domain to approximate the aliasing of spatial frequencies into the transduced image.

In recent years, the issue of maximizing submodular functions has attracted much interest from research communities. However, most submodular functions are specified in a set function. Meanwhile, recent advancements have been studied for maximizing a diminishing return submodular (DR-submodular) function on the integer lattice. Because plenty of publications show that the DR-submodular function has wide applications in optimization problems such as sensor placement impose problems, optimal budget allocation, social network, and especially machine learning. In this research, we propose two main streaming algorithms for the problem of maximizing a monotone DR-submodular function under cardinality constraints. Our two algorithms, which are called StrDRS1 and StrDRS2, have (1/2−ϵ), (1−1/e−ϵ) of approximation ratios and O(nϵlog(logBϵ)logk), O(nϵlogB), respectively. We conducted several experiments to investigate the performance of our algorithms based on the budget allocation problem over the bipartite influence model, an instance of the monotone submodular function maximization problem over the integer lattice. The experimental results indicate that our proposed algorithms not only provide solutions with a high value of the objective function, but also outperform the state-of-the-art algorithms in terms of both the number of queries and the running time.

Today, because of the exponential growth of the volume of data in big data applications, large-scale optimization problems (i.e., optimization problems with a large number of decision variables) have become ubiquitous in the real world [219, 220]. In this chapter, building on the algorithms and observations presented for solving both continuous and combinatorial optimization problems, we further present a novel multi-space evolutionary search framework based on EMT for solving large-scale single-objective optimization problem.

Submodular maximization has wide applications in machine learning and data mining, where massive datasets have brought the great need for designing efficient and parallelizable algorithms. One measure of the parallelizability of a submodular maximization algorithm is its adaptivity complexity, which indicates the number of sequential rounds where a polynomial number of queries to the objective function can be executed in parallel. In this paper, we study the problem of non-monotone submodular maximization subject to a knapsack constraint, and propose the first combinatorial algorithm achieving an (8+epsilon)-approximation under O(log n) adaptive complexity, which is optimal up to a factor of O(loglog n). Moreover, under slightly larger adaptivity, we also propose approximation algorithms with nearly optimal query complexity of O(n), while achieving better approximation ratios. We show that our algorithms can also be applied to the special case of submodular maximization subject to a cardinality constraint, and achieve performance bounds comparable with those of state-of-the-art algorithms. Finally, the effectiveness of our approach is demonstrated by extensive experiments on real-world applications.

Preface. 1 Introduction. 1.1 Two examples of combinatorial optimization. 1.2 Why study combinatorial optimization using statistical physics? 1.3 Textbooks. Bibliography. 2 Algorithms. 2.1 Pidgin Algol. 2.2 Iteration and recursion. 2.3 Divide-and-conquer. 2.4 Dynamic programming. 2.5 Backtracking. Bibliography. 3 Introduction to graphs. 3.1 Basic concepts and graph problems. 3.2 Basic graph algorithms. 3.3 Random graphs. Bibliography. 4 Introduction to complexity theory. 4.1 Turing machines. 4.2 Church's thesis. 4.3 Languages. 4.4 The halting problem. 4.5 Class P. 4.6 Class NP. 4.7 Definition of NP-completeness. 4.8 NP-complete problems. 4.9 Worst-case vs. typical-case complexity. Bibliography. 5 Statistical mechanics of the Ising model. 5.1 Phase transitions. 5.2 Some general notes on statistical mechanics. 5.3 The Curie-Weiss model of a ferromagnet. 5.4 The Ising model on a random graph. Bibliography. 6 Algorithms and numerical results for vertex covers. 6.1 Definitions. 6.2 Heuristic algorithms. 6.3 Branch-and-bound algorithm. 6.4 Results: Covering random graphs. 6.5 The leaf-removal algorithm. 6.6 Monte Carlo simulations. 6.7 Backbone. 6.8 Clustering of minimum vertex covers. Bibliography. 7 Statistical mechanics of vertex covers on a random graph. 7.1 Introduction. 7.2 The first-moment bound. 7.3 The hard-core lattice gas. 7.4 Replica approach. Bibliography. 8 The dynamics of vertex-cover algorithms. 8.1 The typical-case solution time of a complete algorithm. 8.2 The dynamics of generalized leaf-removal algorithms. 8.3 Random restart algorithms. Bibliography. 9 Towards new, statistical-mechanics motivated algorithms. 9.1 The cavity graph. 9.2 Warning propagation. 9.3 Belief propagation. 9.4 Survey propagation. 9.5 Numerical experiments on random graphs. Bibliography. 10 The satisfiability problem. 10.1 SAT algorithms. 10.2 Phase transitions in random K-SAT. 10.3 Typical-case dynamics of RandomWalkSAT. 10.4 Message-passing algorithms for SAT. Bibliography. 11 Optimization problems in physics. 11.1 Monte Carlo optimization. 11.2 Hysteric optimization. 11.3 Genetic algorithms. 11.4 Shortest paths and polymers in random media. 11.5 Maximum flows and random-field systems. 11.6 Submodular functions and free energy of Potts model. 11.7 Matchings and spin glasses. Bibliography. Index.

Attribute Reduction problem is an important content in Rough Set theory, which has been proven to be NP-hard. It tends to find a subset of attributes that preserve the same discriminatory power as the entire attributes. Heuristics methods are used to find the minimum attribute subset. In this paper, we use three typical heuristic methods to do Attribute Reduction. They are positive-region-based, discernibility-matrix-based and mutual-information-based Attribute Reduction algorithm. Experimental results validate that all the three algorithms have good performance on classification for symbolic data. However, the computational complexity of the three Attribute Reduction algorithms are all O(n^2), so it s hard to apply them to massive data. In order to improve the efficiency of the three Attribute Reduction algorithms, we remodel the objective of them from the perspective of Submodular Function Optimization with submodular optimization strategies. We first optimize the attribute selection process with Lazy-Greedy strategy and then extend the serial algorithm to a parallel version according to the parallel optimization strategy of submodular functions. Experimental results validate that the efficiency of the proposed Attribute Reduction algorithms.

Neighborhood Covering Reduction (NCR) methods learn rules for classification through formulating the covering of data space with neighborhoods. NCR method transforms original data into neighborhood systems and facilitates the data generalization. However, the computational complexity of extant NCR methods is \(O(n^2)\) which impedes the application of NCR on massive data and the error bound analysis is insufficient. In this paper, we remodel the objective of NCR from the view of Submodular Function Optimization and thereby improve the efficiency of NCR based on submodular optimization strategies. We first optimize the reduction process of neighborhoods with Lazy-Greedy strategy and further extend the serial algorithm to a parallel version according to the parallel optimization strategy of submodular functions. The error bounds of the proposed NCR algorithms are also analyzed. Experimental results validate the efficiency of the proposed NCR algorithms.

Algorithms for some combinatorial optimization problems : balanced flows, submodular flows, and scheduling on parallel facilities with changeover costs

We develop a framework for the distributed minimization of submodular functions. Submodular functions are a discrete analog of convex functions and are extensively used in large-scale combinatorial optimization problems. While there has been a significant interest in the distributed formulations of convex optimization problems, distributed minimization of submodular functions has received relatively little research attention. Our framework relies on an equivalent convex reformulation of a submodular minimization problem, which is efficiently computable. We then use this relaxation to exploit methods for the distributed optimization of convex functions. The proposed framework is applicable to submodular set functions as well as to a wider class of submodular functions defined over certain lattices. We also propose an approach for solving distributed motion-planning problems in discrete state space based on submodular function minimization. We establish through a challenging setup of the capture the flag game that submodular functions over lattices can be used to design artificial potential fields for multiagent systems with discrete inputs. These potential fields are designed such that their minima correspond to desired behaviors, that is, agents are attracted toward their goals and are repulsed from obstacles and from each other for collision avoidance. Finally, we demonstrate that the proposed distributed framework can be employed effectively for generating feasible trajectories in such motion coordination problems.

We study dynamic algorithms for the problem of maximizing a monotone submodular function over a stream of n insertions and deletions. We show that any algorithm that maintains a (0.5+є)-approximate solution under a cardinality constraint, for any constant є>0, must have an amortized query complexity that is polynomial in n. Moreover, a linear amortized query complexity is needed in order to maintain a 0.584-approximate solution. This is in sharp contrast with recent dynamic algorithms of [LMN+20, Mon20] that achieve (0.5−є)-approximation with a polylog(n) amortized query complexity.

While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up. As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis. In this study, we develop an algorithm to automatically detect free text radiology reports that have a follow-up recommendation using natural language processing (NLP) techniques and machine learning models. The data set used in this study consists of 6000 free text reports from the author's institution. NLP techniques are used to engineer 1500 features, which include the most informative unigrams, bigrams, and trigrams in the training corpus after performing tokenization and Porter stemming. On this data set, we train naive Bayes, decision tree, and maximum entropy models. The decision tree model, with an F1 score of 0.458 and accuracy of 0.862, outperforms both the naive Bayes (F1 score of 0.381) and maximum entropy (F1 score of 0.387) models. The models were analyzed to determine predictive features, with term frequency of n-grams such as "renal neoplasm" and "evalu with enhanc" being most predictive of a follow-up recommendation. Key to maximizing performance was feature engineering that extracts predictive information and appropriate selection of machine learning algorithms based on the feature set.

The number of social individuals who interact with their friends through social networks is increasing, leading to an undeniable fact that word-of-mouth marketing has become one of the useful ways to promote sale of products. The Constrained Profit Maximization in Attribute network (CPMA) problem, as an extension of the classical influence maximization problem, is the main focus of this paper. We propose the profit maximization in attribute network problem under a cardinality constraint which is closer to the actual situation. The profit spread metric of CPMA calculates the total benefit and cost generated by all the active nodes. Different from the classical Influence Maximization problem, the influence strength should be recalculated according to the emotional tendency and classification label of nodes in attribute networks. The profit spread metric is no longer monotone and submodular in general. Given that the profit spread metric can be expressed as the difference between two submodular functions and admits a DS decomposition, a three-phase algorithm named as Marginal increment and Community-based Prune and Search(MCPS) Algorithm frame is proposed which is based on Louvain algorithm and logistic function. Due to the method of marginal increment, MPCS algorithm can compute profit spread more directly and accurately. Experiments demonstrate the effectiveness of MCPS algorithm.