A Novel Node Management in Hadoop Cluster by Using DNA

ZigBee network is a distributed system, consisting of large numbers of nodes. For network analysis, design and maintenance, it is quite necessary to obtain network structure and to visually display active nodes and the effect of failure nodes in a network. In this paper, a method for structure analysis of ZigBee network based on node address is proposed. Using this method, it is easy to get a ZigBee network's structure information using node addresses and the network's global parameters without additional communication burden on the ZigBee network. Finally, a Qt software module to analyze and visually display ZigBee network is introduced.

A Call Detail Record (CDR) is a data record produced by telecommunication equipment consisting of detail of call transaction logs. It contains valuable information for many purposes of several domains such as billing, fraud detection and analytical purposes. However, in the real world, these needs face a big data challenge. Billions CDRs are generated every day and the processing systems are expected to deliver results in a timely manner. In our case, the system also has constraint that is running in limited computation resources. We found that our current production system was not enough to meet these needs. We had successfully analyzed the current system bottleneck and found the root cause. Based on this analysis, we designed and implemented a better performance system which is based on MapReduce and running on Hadoop cluster. This paper presents the analysis of previous system and the design and implementation of new system, called MS2. In this paper, we also provide empirical evidence demonstrating the efficiency and linearity of MS2. In a test case of telecommunication mediation system, our test has shown that MS2 reduces overhead by 44% and speedup performance by nearly twice compared to previous system. From benchmarking with several related technologies in large scale data processing, MS2 is also shown to perform better in case of CDR batch processing. Running on a cluster consists of eight core CPU and two conventional disks, MS2 is able to process 67,000 CDRs/second.

In this manuscript, an energy management system (EMS) is proposed to the distribution system (DS) using Internet of Things (IoT) framework with a hybrid system. The proposed hybrid method is the combination of the Seagull Optimization Algorithm (SOA) and Owl Search Algorithm (OSA), hence it is called SO2SA technique. The principle objective of the SO2SA technique is to optimize managing distribution system power and resources through continuous monitoring of the data from a communication framework based on IoT. In SO2SA technique, every home device is connected to the module of data acquisition, which indicates an IoT object along with a unique IP address as a result of huge mesh wireless network devices. The sending data are processed through SO2SA technique. Similarly, the IoT architecture of the distribution system enhances the flexibility of these networks and gives optimal utilization of obtainable resources. In addition, the SO2SA technique is responsible for meeting the overall power and supply requirements. The proposed method is implemented in MATLAB/Simulink site and the efficiency is likened to the other different methods. In 50 trail numbers, the RMSE, MAPE, and MBE range of SO2SA technique represents 5.63, 0.90, and 1.035. Thus, the proposed technique is highly competent over all the existing approaches.

Improving OLAP query performance in a distributed system such as Hadoop and Spark is a challenging task. An OLAP query is composed of several operations, such as projection, filtering, join, and grouping. The star join operation is the most expensive one and usually involve considerable communication cost. The common method used to decrease the network traffic for the star join operation is to co-partition some tables of a data warehouse on their join key. However, this operation still requires many MapReduce cycles in existing data warehouses partitioning schemes. In this paper, we propose a new physical design of distributed big data warehouses over Hadoop cluster. We propose two methods called “FKey” and “NewKey” based on a data mining technique to guide our physical design. Our partitioning and distribution scheme helps the query’s optimizer to make a good query processing plan, such it can performing the star join operation in only one Spark stage without the shuffle phase. To evaluate our approach we have done some experiments on a cluster of data nodes using the TPC-DS benchmark. The results show that our proposal outperforms the existing approaches in terms of query runtime.

Apache Hadoop is a widely used distributed system in large-scale production environment. With the increasing size of data volume and cluster scale, its performance is limited by inappropriate resources utilization. This paper introduces a resources utilization predictor (RUPredHadoop) to predict utilization of cpu, memory, read/write rate of disk and network, especially for large-scale Hadoop clusters. In terms of the similarity of data and workflow in Hadoop, the pattern of resource utilization for a single task is proposed, and then formulized by a single task model. Besides that, the distribution of fine-grained runtime is studied, so that a parallel-batch-tasks-based model could regenerate the whole Mapreduce job by migrating the single task model from the minimum cluster to a large-scale production cluster. With RUPredHadoop, we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data. The performance of RUPredHadoop is validated by a test cluster with 35 nodes and a production cluster with 80 nodes. Results show that the normalization error is below 10% for benchmark applications with maximum 100 TB data.

Hadoop and Spark are well-known big data processing platforms. The main technologies of Hadoop are Hadoop Distributed File System and MapReduce processing. Hadoop stores intermediary data on Hadoop Distributed File System, which is a disk-based distributed file system, while Spark stores intermediary data in the memories of distributed computing nodes as Resilient Distributed Dataset. In this paper, we show how memory size affects distributed processing of large volume of data, by comparing the running time of K-means algorithm of HiBench benchmark on Hadoop and Spark clusters, with different size of memories allocated to data nodes. Our results show that Spark cluster is faster than Hadoop cluster as long as the memory size is big enough for the data size. But, with the increase of the data size, Hadoop cluster outperforms Spark cluster. When data size is bigger than memory cache, Spark has to replace disk data with memory cached data, and this situation causes performance degradation.

The key elements of logistics management of the distribution system of goods, in particular international trade networks, are studied. The parameters of evaluating the efficiency of individual distribution channels of goods in the distribution system are considered. The concept of "bottleneck" in the distribution network in general and individual distribution channels in particular has been clarified. The causes of bottlenecks in the distribution system channels, the points of localization of possible current restrictions - bottlenecks in the distribution system of goods are analyzed and the ways of their elimination are investigated. The results of research, methods and recommendations can be used to form the optimal distribution strategy of modern companies in conditions of unstable demand and under the influence of other negative factors. In view of the above, the task is to study in more detail the key elements of logistics procurement management for the defense sector in space and time and to develop practical recommendations for logistics management of the organization of defense procurement. Therefore, the aim of the study is to analyze the structure of the distribution network in conditions of unstable demand and under the influence of other negative factors in order to identify bottlenecks in the distribution system and find ways to eliminate them optimally.To achieve this goal, the following were investigated: priorities and guidelines for logistics management of the distribution system of goods from the manufacturer to the network of end users; structure and configuration of distribution channels in the distribution system as a subject of optimization; causes of bottlenecks in the distribution system channels and ways to eliminate them. Thus, logistics solutions localized in the field of distribution are a "trade-off" relationship with the sphere of production. That is, the field of distribution focuses on two important issues: first, the rationale and formation of distribution channels; secondly, the design and implementation of physical distribution of goods. In addition, the logistics management of the distribution system should be understood as the management of the process of moving goods from producer to end consumer, or their network, integrated with relevant financial, information and service flows to ensure a mix of logistics tasks.

In-memory key-value stores (KVS) or caches form the backbone of many commercial and HPC applications. The basic operation of KVS revolves around storing or updating the mapping from keys to their corresponding values and looking up that mapping when requested by a client. We observe that the memory management unit (MMU) in modern processors does something similar – it looks up the mapping between virtual addresses and physical addresses stored in the per-process page table. We leverage the MMU to gain hardware acceleration for key-value lookup for free in a new key-value store design called memwalkd. We hash keys to unique virtual addresses. These addresses map to the physical addresses that hold the corresponding values. Thus, GET/SETs are performed by simply issuing loads/stores to the hash of a key. Across a wide range of workloads, memwalkd achieves 1.8× better throughput over a highly-optimized implementation of memcached called MICA [1].

JiroT Technology is an implementation of the Federated Management Architecture developed by Sun Microsystems Incorporated (SMI). It defines middleware to support the development of distributed system management applications. Since it has not yet gained the widespread popularity of other Java technologies there has been little published discussion of how architects and designers can take advantage of the specialized features it offers. In this paper, some of those key features are examined with respect to using the Unified Modeling Language (UML) during the development process.

Local hospitals must deal with large numbers of patients during mass casualty incidents, and the wireless sensor networks (WSNs) can help in these situations by monitoring vital signs. Conventional ZigBee nodes can obtain the ID of a device by assigning a unique 16-bit short address or by burning firmware into an IC. These methods tend to complicate node management and lack portability. The study developed a node management mechanism to deal with a large number of patients in real-time, through the wireless monitoring of physiological signals. The mechanism proposed for the ZigBee WSN is based on a three-layer (Coordinator, Control Router, and End Device) tree topology. The proposed system includes a node deployment process to formulate a ZigBee WSN as a tree topology, an algorithm to automatically number ZigBee nodes for monitoring and control system (MCS), and an algorithm to automatically obtain the short addresses of nodes for data collection. Specifically, an algorithm automatically collects data from ZigBee nodes for display on a computer graphical user interface (GUI). We also developed a reliable data transmission method capable of resolving the problem of packet loss. The proposed method has been applied in a local hospital. Our research findings provide a valuable reference for the development of ZigBee-based MCS. The proposed node management mechanism is faster, more reliable, and more intuitive to use, than traditional methods.

This article introduces a comprehensive framework for building trustworthy and interpretable Large Language Model (LLM)-driven control planes for autonomous distributed systems. As LLMs are increasingly integrated into critical infrastructure management, from database administration to resource allocation, they introduce unique challenges related to trustworthiness, interpretability, and robustness. To identify two fundamental issues: the Recursive AI Problem, where AI systems manage other AI components, creating complex feedback loops, and the Semantic Gap Issue, where LLMs operate at high levels of abstraction while distributed systems require precise operational semantics. Our proposed framework addresses these challenges through a multi-layered approach combining formal verification techniques, explainability mechanisms, and human-in-the-loop integration. We develop novel methods for system-specific LLM fine-tuning, symbolic-neural integration, and semantic alignment verification. Experimental evaluation across database management, Kubernetes resource allocation, and data pipeline orchestration scenarios demonstrates significant improvements in trustworthiness and interpretability while maintaining performance benefits. This work bridges the critical gap between advanced LLM capabilities and the requirements for safe, reliable system management in mission-critical environments.

Smart home based on the Internet of things is gradually changing our daily life, while it faces such serious problems as secure measuring and controlling to appliances and protection of end user’s privacy. To guarantee the safety and reliability of smart home system, a secure measuring and controlling method embedded the domestic SM4 cryptographic algorithm for smart home is proposed in this paper, which integrates such secure mechanisms as access control based on the physical addresses of smartphone terminal or sensor nodes used in measuring and controlling operations, authentication based on the keys used in the encrypted transmission of instructions of measuring, and controlling and abnormal detection based on analysis of data. On such basis, the comparison in performances is carried out by secure measuring and controlling methods embedded domestic SM4 algorithm or typical AES one. The results of test suggest that the proposed secure measuring and controlling methods for smart home is feasible and effective, and the delay time consumed in information processing of the proposed method is only 2.5 % more than the currently pervading ones without secure mechanism. The rate of delay based on the SM4 algorithm is about 4 % less on average than the original AES one embedded in nodes, and SM4 algorithm owns independent intellectual property right with more flexible in realization of system.KeywordsInternet of thingsSmart homeSafetySM4 algorithmMeasuring and controlling

The AGGREGATE project team successfully developed and validated various modules for outage management. Brief summaries of each module are provided to showcase their strength for outage management and restoration for a distribution system with a high penetration of connected distribution energy resources (DERs).In recent years, inverter-based DERs have been widely deployed in distribution system. A most of behind-the-meter (BTM) solar power generation is not visible to the utility. The data-driven DER and load estimation modules are using machine learning (ML) and artificial intelligence (AI) to manage this issue, which provides an opportunity for distribution system operators (DSOs) to operate systems and make decisions in real-time for a distribution system with a high penetration of DERs deployed. Also, the estimated DER and true load can be further leveraged in network aggregation and cold-load pick up estimation for reducing the computing complexity and providing for fast restoration. After load demand and DER power generations have been estimated, the information will support topology and state estimation(SE). The topology estimation module demonstrated the viability of mixed integer linear programming (MILP) formulation to estimate the most likely operational radial topology and outage sections using power flow measurements, historical/estimated load and DERs data and smart meter ping measurements. Formulation includes continuous (power flow, load and DERs data) and binary measurements (smart meter ping measurements) in a single formulation. Errors in continuous data and binary data are modeled as normal distribution and Bernoulli distribution, respectively. In the future distribution grid, the power injection from controllable DERs will be essential for efficient and resilient grid operation. However, determining the optimal DER injections and restoration actions is dependent on knowledge of the system states. State estimation (SE), already the cornerstone of transmission energy management systems, will become commonplace in distribution management systems as more measurements become available from deployment of automated metering infrastructure (AMI). Observability analysis is the first step in SE, as it determines the sufficiency of the available measurements for accurately estimating the current system states. A new type of pseudo-measurement called a Correlational Measurement (CM) is introduced in this module, to enhance the observability of the system to enable more accurate SE. CMs encapsulate knowledge of correlation between demand patterns for similar classes of loads as well as injection patterns for same-technology renewable DERs. During grid contingency scenarios, DERs have been traditionally disconnected, without any fault ride-through capabilities. However, with new regulations and better technology, it is feasible for these resources to contribute to the grid’s restoration after an adverse event and hence enhance resilience. The controllability module proposes a two-step restoration scheme for the power system restoration process by leveraging additional degrees of freedom in power electronics interfaced DERs for mitigating voltage problems. In a resilience mode without the utility system, the distribution grid relies on DERs to serve critical load. In such a severe event with multiple faults on the distribution feeders, actuation of various protective devices (PDs) divides the distribution system into electrical islands. The undetected actuated PDs due to fault current contributions from DERs can delay the restoration process, thereby reducing the system resilience. The Advanced Outage Management (AOM) and the Advanced Feeder Restoration (AFR) modules developed in this project provide improved system resilience with multiple DERs. AOM identifies the faulted sections and actuated PDs in a distribution system with DERs by incorporating smart meter data. The most credible outage scenario including fault locations, PD actuations, and fault indicator (FI) failures is identified by a set of binary integer linear programming incorporating hypotheses. The AFR module serves to restore a distribution system with available energy resources taking into consideration the availability of utility sources and DERs. By partitioning the system into islands, critical load will be served with the available generation resources within islands based on the solution of a MILP. When the utility systems become available, the optimal path will be determined by a spanning tree search algorithm that reconnects these islands back to substations and restores the remaining load. The transmission and distribution (T&D) co-simulation module was used to validate the effect of a control action performed on the distribution side assets as it propagates to the transmission side. This ensures that the control action performed results in a feasible operating point on both the transmission and the distribution system. In addition to validation, the team used the T&D co-simulation module to demonstrate how distribution system assets can be used to mitigate issues on the transmission system. Specifically, the team demonstrated that appropriate switching operations on the distribution side can alleviate the line overload condition on the transmission side without causing new operational constraint violations.

In all fields of human society, occasional emergencies are almost inevitable. Once an emergency occurs, rapid and proper decision making is required. The purpose of this paper is to explore the design and development of computerized support systems for emergency decision making (EDM). First the characteristics of EDM problems are examined. Then, in view of limited human computer rationality, requirements for a computerized support system for EDM are determined. A conceptual structure for knowledge-based distributed emergency decision support systems is proposed. Finally, a prototype system for safety protection and disaster response in coal mines, developed using the proposed structure, is briefly described.

With projected lifespans of many decades, infrastructure initiatives such as Europe’s Distributed System of Scientific Collections (DiSSCo), USA’s Integrated Digitized Biocollections (iDigBio), National Specimen Information Infrastructure (NSII) of China and Australia’s digitisation of national research collections (NRCA Digital) aim at transforming today’s slow, inefficient and limited practices of working with natural science collections. The need to borrow specimens (plants, animals, fossils or rocks) or physically visit collections, and absence of linkages to other relevant information represent significant impediments to answering today’s scientific and societal questions. A logical extension of the Internet, Digital Object Architecture (Kahn and Wilensky 2006) offers a way of grouping, managing and processing fragments of information relating to a natural science specimen. A ‘digital specimen’ acts as a surrogate in cyberspace for a specific physical specimen, identifying its actual location and authoritatively saying something about its collection event (who, when, where) and taxonomy, as well as providing links to high-resolution images. A digital specimen exposes supplementary information about related literature, traits, tissue samples and DNA sequences, chemical analyses, environmental information, etc. stored elsewhere than in the natural science collection itself. By presenting digital specimens as a new layer between data infrastructure of natural science collections and user applications for processing and interacting with information about specimens and collections, it’s possible to seamlessly organise global access spanning multiple collection-holding institutions and sources. Virtual collections of digital specimens with unique identifiers offer possibilities for wider, more flexible, and ‘FAIR’ (Findable, Accessible, Interoperable, Reusable) access for varied research and policy uses: recognising curatorial work, annotating with latest taxonomic treatments, understanding variations, working with DNA sequences or chemical analyses, supporting regulatory processes for health, food, security, sustainability and environmental change, inventions/products critical to the bio-economy, and educational uses. Adopting a digital specimen approach is expected to lead to faster insights for lower cost on many fronts. We propose that realising this vision requires a new TDWG standard. OpenDS is a specification of digital specimen and other object types essential to mass digitisation of natural science collections and their digital use. For five principal digital object types corresponding to major categories of collections and specimens’ information, OpenDS defines structure and content, and behaviours that can act upon them: Digital specimen: Representing a digitised physical specimen, contains information about a single specimen with links to related supplementary information; Storage container: Representing groups of specimens stored within a single container, such as insect tray, drawer or sample jar; Collection: Information about characteristics of a collection; Organisation: Information about the legal-entity owning the specimen and collection to which it belongs; and, Interpretation: Assertion(s) made on or about the specimen such as determination of species and comments. Digital specimen: Representing a digitised physical specimen, contains information about a single specimen with links to related supplementary information; Storage container: Representing groups of specimens stored within a single container, such as insect tray, drawer or sample jar; Collection: Information about characteristics of a collection; Organisation: Information about the legal-entity owning the specimen and collection to which it belongs; and, Interpretation: Assertion(s) made on or about the specimen such as determination of species and comments. Secondary classes gather presentation/preservation characteristics (e.g., herbarium sheets, pinned insects, specimens in glass jars, etc.), the general classification of a specimen (i.e., plant, animal, fossil, rock, etc.) and history of actions on the object (provenance). Equivalencing concepts in ABCD 3.0 and EFG extension for geo-sciences, OpenDS is also an ontology extending OBO Foundry’s Biological Collection Ontology (BCO) (Walls et al. 2014) from bco:MaterialSample, which has preferred label dwc:specimen from Darwin Core, thus linking it also with that standard. OpenDS object content can be serialized to specific formats/representations (e.g. JSON) for different exchange and processing purposes.