Graph-Based Data Engineering: Implementing High-Performance Query Optimization in Large-Scale Networks

Multiobjective shortest path problem (MSPP) is one of the most critical issues in network optimization, aimed at identifying all efficient paths across conflicting objectives. Nowadays, existing methods face substantial bottlenecks in addressing the diverse preferences of decision makers and high spatiotemporal overhead caused by the calculation process, particularly in cases with large-scale networks. To overcome these obstacles, a generalized MSPP in large-scale networks is investigated with the aim of solving it with diverse preferences of decision makers satisfied and low spatiotemporal overhead. Toward this end, with a novel concept, the generalized dominance relation is introduced, and the generalized multiobjective shortest path algorithm via the generalized dynamic programming approach is developed. Moreover, the H-reducible technique is further employed to accelerate the convergence of the proposed algorithm. Additionally, several rigorous proofs are provided for the conclusions that all efficient paths could be found within a tolerable time by the developed algorithm and the algorithm could be implemented in a distributed manner under mild assumptions. Finally, numerous routing experiments are conducted on large-scale communication networks for demonstrating the effectiveness and competitiveness of our approach.

Consideration was given to the design of the topological structure of the doubly connected large-scale networks by the cost criterion and constraints on the diameter. An exact combinatorial algorithm to solve the problem and a parallel algorithm to carry out calculations on the multiprocessor data-flow computer system were proposed. Parallelization was shown to provide a substantial gain in the time of computations in the optimal topological structure of the large-scale networks.

This paper presents a mathematical stochastic methodology for transit route network optimization. The goal is to provide an effective computational tool for the optimization of a large-scale transit route network to minimize transfers with reasonable route directness while maximizing service coverage. The methodology includes representation of transit route network solution search spaces, representation of transit route and network constraints, and a stochastic search scheme based on an integrated simulated annealing and genetic algorithm solution search method. The methodology has been implemented as a computer program, tested using previously published results, and applied to a large-scale realistic network optimization problem.

Accurate traffic classification plays a pivotal role in the management and optimization of modern computer networks. Traditional machine learning methods have shown promise, but they often fail to capture the complex and dynamic nature of network traffic. This paper proposes the use of Graph Neural Networks (GNNs) for traffic classification in large-scale networks, leveraging the relational information among flows and nodes. Our method models traffic data as a graph, with flows and their metadata represented as nodes and edges. The GNN learns from these interactions to classify traffic into predefined categories. Experimental results demonstrate that our GNN-based approach significantly outperforms baseline models in terms of accuracy and generalization across varied datasets.

Latency-optimized high performance Data Vortex optical switching network

With the tremendous increased scale of networks, data transferring plays a very important role in high performance computing. Traditionally, when a task is completed, the data associated with the task will be released to free the space for the following tasks. However, the following task may require the same data, which will be fetched back and the efficiency will be affected too. In this paper, we implement a data caching method so as to improve the performance of the system. When a task is completed, the data will be stored for a longer time. If the previous used data is requested before releasing, there will be no transferring cost. With the help of data caching, the performance of the network in handling real-time tasks is significantly improved while the traffic jamming is also decreased. We extend the previous single data model to multiple type of data among servers. Tasks arrive in an online fashion, and when handling a task on some network server, we have no information about the future tasks. Assuming the storage cost is submodular, which is well adopted in reality, we propose an online algorithm to minimize the total transfer and caching costs of the system. Via theoretical analysis, the competitive ratio of the algorithm is proved to be a constant.

We present a detailed description of the diagrammatic technique, recently devised in [V. V. Kuzmin et.al., npj Quantum Information 5, 115 (2019)], for semi-analytical description of large-scale quantum-repeater networks. The technique takes into account all essential experimental imperfections, including dissipative Liouville dynamics of the network quantum memories and the classical communication delays. The results obtained with the semi-analytic method match the exact Monte Carlo simulations while the required computational resources scale only linearly with the network size. The presented approach opens new possibilities for the development and efficient optimization of future quantum networks.

It was recently observed from empirical traffic data that by aggregating the highly scattered plots of flow versus density from individual loop detectors for city regions with homogeneous spatial distribution of congestion, the scatter significantly decreases and a well-defined Macroscopic Fundamental Diagram (MFD) exists between space-mean flow and density. This result can be of great importance to unveil simple perimeter control policies in such a way that maximizes the network outflow (trip endings). Single-region perimeter control might be sub-optimal if there is a significant number of destinations outside the region of analysis or if the network is heterogeneously loaded. This paper integrates an MFD modeling to perimeter and boundary control optimization for large-scale networks with multiple centers of congestion, if these networks can be partitioned into a small number of homogeneous regions. Perimeter control actions may be computed in real-time through a linear multivariable feedback regulator. The impact of the perimeter control actions to a three-region real urban network is demonstrated via micro-simulation. A key advantage of the proposed approach is that it does not require high computational effort and future demand data if the current state of each region can be observed.

This paper describes various practical aspects, in some detail, of inmplementiing an algorithm for the design of large-scale radial distribution networks. This algorithm is one which has been published previously, so this paper should be considered as the second part of that work. This algorithm simultaneously optimizes the line layout and type of conductor used. A branch-exchange algorithm for layout optimization, a dynamic programming algorithm for conductor optimization, and a decomposition algorithm enabling large-scale problem solving are all combined in this algorithm. To demonstrate the benefits of the proposed algorithm, numerical results have been provided from a reference network built for a low-voltage rural zone comprised of 1088 loads and 20 distribution substations. It can be verified that using the decomposition technique enables large-scale problem solving and produces results that are better than those techniques which come from a mere partition of the original problem into smaller parts.

Reinforcement learning is a classic tool to solve network control and policy optimization problems in unknown environments. The original Q-learning algorithm suffers from performance and complexity challenges when applied to large networks. For networks which admit Markov decision process (MDP) models, a novel ensemble reinforcement learning method which adapts the original Q-learning is proposed for policy optimization in unknown environments. The learning algorithm is driven through the optimization of the Jensen-Shannon diver-gence (JSD). Multiple Q-learning algorithms are run on distinct, synthetically created, but structurally related Markovian systems in parallel, and their outputs are fused based on an adaptive weighting mechanism. The JSD is analyzed, and the convergence of key statistics and Q-functions are provided. Numerical results for several network classes show that the proposed algorithm can achieve up to 50 % less policy error with up to 40 % less runtime complexity versus other state-of-the-art Q-learning algorithms.

The IEEE 802.15.4 Time-Slotted Channel Hopping (TSCH) protocol has received considerable attention in many industrial applications. However, analytical models for fast performance estimation of TSCH-based networks by considering the interaction between Medium Access Control (MAC) and Physical (PHY) layers is an open problem. In this paper, we propose a stochastic model for performance analysis of TSCH-based networks including dedicated and shared links with non-ideal wireless link properties. The proposed model is scalable and is able to evaluate the MAC performance of a large-scale network quickly. The developed model is verified by simulations and real-world experiments. The results confirm the accuracy of the proposed model for large-scale networks with orders of magnitude faster execution compared to the existing model in the literature. This confirms the speed and scalability of the model, which makes it a perfect tool for network design and optimization.

Day-to-day dynamic traffic assignment with imperfect information, bounded rationality and information sharing

Food safety management is an important issue in human health and social stability. It is prohibitively expensive to monitor all network nodes because of the large scale of food supply networks and the limited monitoring resources. We propose a directed graph-based social network model for design optimization of hazard monitoring network. Following the principles of nodes in social networks, the concepts and calculating methods of k-step observability/controllability and accessible node sets are proposed to measure node accessibility. Based on the above concepts, a 0–1 linear programming model is developed for the selection of the optimal monitoring points. The selected combination of monitoring points can access the whole food supply network with constrained monitoring resources. The model can be solved by using a common commercial software package. We apply the approach to a real local supply network of infant foods to test the validity of the model. A reasonable and satisfactory result is achieved. The numerical analysis of several randomly generated examples with different parameters shows the validity and effectiveness of the recommended approach.

Enterprise cloud architectures face unprecedented challenges in supporting global-scale operations while maintaining operational excellence, security, reliability, performance efficiency, and cost optimization. Contemporary cloud platforms must address complex requirements across diverse industry verticals, from microsecond-level latency demands in financial services to carrier-grade availability exceeding five nines of uptime for telecommunications providers. Modern cloud scaling encompasses multiple dimensions, including computational scalability, large-scale data management, network optimization, and geographic distribution strategies. Stateless design principles form the foundation of horizontally scalable systems, enabling seamless load distribution across compute resources without session affinity constraints. Advanced architectural patterns, including serverless APIs, microservices decomposition, and service mesh topologies, provide sophisticated traffic management capabilities while maintaining service isolation and independent scaling characteristics. Large data volume optimization strategies demonstrate significant performance improvements through columnar storage format, achieving substantial compression ratios and reduced query execution times. Geographic failover architectures ensure business continuity through strategic resource distribution, implementing active-active and active-passive configurations with defined recovery objectives. Platform resilience incorporates multi-layer fault tolerance mechanisms, including redundancy, circuit breakers, and automated recovery systems to maintain service availability during component failures.

The purpose of this paper is to establish algorithms in the design of rural low-voltage (LV) distribution networks. Planning of rural LV distribution networks involves the optimal design of radial configuration and location of MV/LV substations considering cost functions. In this work both goals are considered by taking into account different conductors, voltage drop constraints, power losses in lines and deterministic loads. The algorithms developed in this paper are based on dynamic programming; and were implemented on large-scale rural LV distribution networks, but they could also be used for other network optimization.