Incremental Data Migration for Multi-Database Systems Based on MySQL with SPIDER Storage Engines

Data migration schemes are critical to balance the load in storage clusters for performance improvement. However, as NAND flash based SSDs are widely deployed in storage systems, extending the lifespan of SSD storage clusters becomes a new challenge for data migration. Prior approaches designed for HDD storage clusters, however, are inefficient due to excessive write amplification during data migration, which significantly decrease the lifespan of SSD storage clusters. To overcome this problem, we propose EDM, an endurance aware data migration scheme with careful data placement and movement to minimize the data migrated, so as to limit the worn-out of SSDs while improving the performance. Based on the observation that performance degradation is dominated by the wear speed of an SSD, which is affected by both the storage utilization and the write intensity, two complementary data migration policies are designed to explore the trade-offs among throughput, response time during migration, and lifetime of SSD storage clusters. Moreover, we design an SSD wear model and quantitatively calculate the amount of data migrated as well as the sources and destinations of the migration, so as to reduce the write amplification caused by migration. Results on a real storage cluster using real-world traces show that EDM performs favorably versus existing HDD based migration techniques, reducing cluster-wide aggregate erase count by up to 40%. In the meantime, it improves the performance by 25% on average compared to the baseline system which achieves almost the same effectiveness of performance improvement as previous migration techniques.

In large scale data storage systems, RAID-6 has received more attention due to its capability to tolerate concurrent failures of any two disks, providing a higher level of reliability. However, a challenging issue is its scalability, or how to efficiently expand the disks. The main reason causing this problem is the typical fault tolerant scheme of most RAID-6 systems known as Maximum Distance Separable (MDS) codes, which offer data protection against disk failures with optimal storage efficiency but they are difficult to scale. To address this issue, we propose a novel Stripe-based Data Migration (SDM) scheme for large scale storage systems based on RAID-6 to achieve higher scalability. SDM is a stripe-level scheme, and the basic idea of SDM is optimizing data movements according to the future parity layout, which minimizes the overhead of data migration and parity modification. SDM scheme also provides uniform data distribution, fast data addressing and migration. We have conducted extensive mathematical analysis of applying SDM to various popular RAID-6 coding methods such as RDP, P-Code, H-Code, HDP, X-Code, and EVENODD. The results show that, compared to existing scaling approaches, SDM decreases more than 72.7% migration I/O operations and saves the migration time by up to 96.9%, which speeds up the scaling process by a factor of up to 32.

Big Data Analytics is a big word now days. As per demanding and more scalable process data generation capabilities, data acquisition and storage become a crucial issue. Cloud storage is a majorly usable platform; the technology will become crucial to executives handling data powered by analytics. Now a day’s trend towards “big data-as-a-service” is talked everywhere. On one hand, cloud-based big data analytics exactly tackle in progress issues of scale, speed, and cost. But researchers working to solve security and other real-time problem of big data migration on cloud based platform. This article specially focused on finding possible ways to migrate big data to cloud. Technology which support coherent data migration and possibility of doing big data analytics on cloud platform is demanding in natute for new era of growth. This article also gives information about available technology and techniques for migration of big data in cloud.

This book explores the implications of non-volatile memory (NVM) for database management systems (DBMSs). The advent of NVM will fundamentally change the dichotomy between volatile memory and durable storage in DBMSs. These new NVM devices are almost as fast as volatile memory, but all writes to them are persistent even after power loss. Existing DBMSs are unable to take full advantage of this technology because their internal architectures are predicated on the assumption that memory is volatile. With NVM, many of the components of legacy DBMSs are unnecessary and will degrade the performance of data-intensive applications. We present the design and implementation of DBMS architectures that are explicitly tailored for NVM. The book focuses on three aspects of a DBMS: (1) logging and recovery, (2) storage and buffer management, and (3) indexing. First, we present a logging and recovery protocol that enables the DBMS to support near-instantaneous recovery. Second, we propose a storage engine architecture and buffer management policy that leverages the durability and byte-addressability properties of NVM to reduce data duplication and data migration. Third, the book presents the design of a range index tailored for NVM that is latch-free yet simple to implement. All together, the work described in this book illustrates that rethinking the fundamental algorithms and data structures employed in a DBMS for NVM improves performance and availability, reduces operational cost, and simplifies software development.

Considerable effort has been devoted to the migration of multicomponent data in elastic media with wavefield separation techniques being the most successful. Most of this work has been carried out in the time domain. In this paper, we formulate a multicomponent migration technique in the frequency domain. Reverse-time migration can be viewed as the zero-lag cross-correlation between virtual source and back-propagated wavefields. Cross-correlating the Helmholtz decomposed wavefields rather than directly correlating the vector displacement fields results in sharper, more interpretable images, contaminated by fewer crosstalk artefacts. The end products are separate P and S wave (and if desired, PS and SP) migration images. We test our migration algorithm on synthetic seismic data generated using the SEG/EAGE salt-dome, Overthrust and Marmousi-2 models. We correctly image the location and shape of the target zone for oil exploration using these data sets. Furthermore, we demonstrate that our new migration technique provides good images even when the initial velocity model is only approximate.

Ground-penetrating radar (GPR) profiling is the primary tool to provide detail information of the internal structure and characteristics inside a sand dune in the desert area. However, with the severe elevation change in a short horizontal distance and on the rugged and complex surface topography of the sand dunes, getting clear imaging of the internal structure of sand dunes from the GPR profiling is still a big challenge. The classic imaging technique such as the Kirchhoff migration has been applied to process GPR data on sand dunes, with limited success. The reverse-time migration (RTM) technique is the most advanced imaging technique that can handles GPR data acquired on rugged surface with severe topographic relief to generate subsurface structural images with high fidelity and has been tried to process GPR data on sand dunes. The results are encouraging, and the imaging quality is significantly improved. The finite-difference time-domain (FDTD) method is the major numerical tool for forward and backward continuations of the wave field during the RTM process. There are two aspects for using FDTD in RTM of GPR data on sand dunes still need improvement: the numerical scattering caused by the staircase approximation of the ground surface by using gridding in the Cartesian coordinate, and the negligence of the possible anisotropy of the electromagnetic material properties due to the calcareous cementation bedding inside a sand dune. In this paper, we develop the RTM algorithm based on the staggered grid FDTD that handles the rugged topographic surface by using the curvilinear coordinate, and the possible anisotropic radar wave velocity of the sand dune media. We first demonstrate the equivalency of the nonuniform, isotropic medium and the uniform, anisotropic medium for justifying using the uniform, anisotropic velocity in the RTM by synthetic modeling. Next, we validate our approach of using the synthetic data with the comparison of using the Cartesian coordinate and the curvilinear coordinate in an isotropic medium. The results indicate that the RTM algorithm using the curvilinear coordinate can efficiently eliminate the adverse effect of the staircase approximated boundary of the topography surface. Finally, we processed the real GPR data collected on a sand dune in the Badain Jaran desert by using the curvilinear coordinate and the uniform, anisotropic velocity in FDTD forward and backward wave field continuation. Comparison of the results indicates that the RTM imaging using the boundary-conforming curvilinear coordinate and anisotropic velocity gains more coherent and higher resolution images for the calcareous cementation layers and the water table.

The fractured granite basement is the primary oil and gas reservoir in the Cuu Long Basin, Vietnam. Due to the complexity of this nonlayered unconventional target, combined with complicated fault and fracture systems, the seismic data quality near and within the basement section is very low. For this reason, it is important to apply improved seismic data processing workflows, filtering and migration techniques, as wells as attribute processing methods to enhance the imaging quality. Our studies show that applying different types of filters, including the f-k, Radon transform and Tau-P, improves signal to noise ratio, removing multiples, revealing basement’s top and its related fractured and fault zones. In addition, the application of multi-arrival-solution migration algorithms, such as Kirchhoff Migration and Controlled Beam Migration, provides improved imaging for identifying basement top and faults and fractures within the basement. Furthermore, the application of seismic attributes such as curvature, apparent dip, or energy gradient, is important in locating faults and fractures, whereas mapping of intensity and orientation of such structures assists the delineation of “sweet spots” and assists the planning of exploration.

In this paper we demonstrate the potential of a new migration technique for easy detection of subtle reflector discontinuities, such as due to faults with minor throw or reflector pinch-outs in wedge-shaped geological units. Such discontinuities cannot easily be identified on images produced with conventional migration methods. The novelty of our method concerns the computation of edge-diffraction-specific weightings for migration operator amplitudes. These weighting factors can be easily obtained from the operators already computed for a conventional true-amplitude Kirchhoff migration of reflection data. An advantage of combining a conventional true-amplitude Kirchhoff migration and our edge-diffraction migration method, providing a second migration image containing only the diffracting edges, are the diagnostic power for velocity model refinement and seismic interpretation of complex structures. The simplicity of use and relatively small computing effort are important practical benefits. The additional cost of computing the edge-diffraction migration weightings in addition to the conventional true-amplitude Kirchhoff operator attributes is negligible and therefore the extra cost for computing the additional edge-diffraction migration image is approximately equal to the cost of applying one additional migration with just new amplitude weightings applied to already available migration operators.

New migration technique of massive 3D VSP data is presented. The method is based on energy hashing algorithm. This technique is designed to enhance the quality of seismic images of complex geological structure derived from limited a priori information. The method effectiveness is demonstrated on experimental data processing result.

In purpose of data searching acceleration, the fastest data response is the major concern for latest cloud environment. Regarding this, the intellectual decision is to enrich the SaaS level applications. Amongst the SaaS based applications, service level database integration is the recent trend to provide the integrated view of the heterogeneous cloud databases through shared services using DBaaS. But the generic limitations interacted during the database integration are dynamic adaptability of multiple databases structure, dynamic data location identification in the concern databases, data response using the data commonality. Data migration technique and single query approach are the two individual solutions for the proposed limitations. But the side effects during data migration technique are extra space utilisation and excess time consumption. Again, the single query approach suffers from worst case time complexity for data connectivity, data aggregation and query evaluation. So, to find a suitable data response solution by eliminating these combined major issues, a graph based Middleware Database Integrator Platform or MDIP model has been proposed. This integrator platform is actually the flexible metadata representation technique for the concerned heterogeneous cloud databases. The associativity and commonality among components of multiple databases would be further helpful for efficient data searching in an integrated way. For the incorporation within the service level but not in the services, MDIP is considered as the different platform. It is applicable over any service based database integration in purpose of data response efficiency. Finally, the quality assessment using evaluated query time compared with already proposed SLDI shows better data access quality. Thus, its expertise dedication in data response can overcome summarised challenges like data adaptation flexibility, dynamic identification of data location, wastage of data storage, data accessing within minimal time span and optimised cost in presence of data consistency, data partitioning and user side scalability.

A case study is presented on increasing operational efficiencies using fiber optic-equipped coiled tubing to perform distributed temperature sensing (DTS) and distributed acoustic sensing (DAS) in conjunction with millout operations. In unconventionals with plug-and-perf or sliding-sleeve completions, hydraulic workover or coiled tubing is deployed to millout and clean the wellbore to total depth (TD) before the well can be put on production. Traditionally, the millout and cleanout process is a standalone operation. In the event a diagnostic service is performed on the well, a workover rig, coiled tubing unit, or wireline would be deployed in a second intervention after the initial millout is performed. This method presents a challenge to operators who want to bring wells online in a timely and economical manner. By combining the millout and diagnostic cababilities into a single operation, operational efficiencies were significantly increased for the operator. Operators were able to minimize exposure to in-well operations and reduce the number of interventions necessary to bring a single well online. This resulted in a reduction of well shut-in time and faster data turnaround time on diagnostic evaluation. Simultaneous milling and diagnostic services were completed on 31 wells throughout 2018. Plug/Sleeve/Sand millouts were performed to well TD with a fiber-optic capable coil-tubing unit. Then the same work string was used to gather diagnostic data on the entire completion interval during the same run. These diagnostic data were collected and provided to the operator in an effort to better understand the completion design and hydraulic fracture placement. The operational efficiency gained by deploying diagnostic services in conjunction with the millout services lowered the operator's operation costs by USD 100,000 and operations time by 36 hr on average.

To be considered fast, algorithms for operations on large data structures should operate in polylog time, i.e., with the number of steps bounded by a polynomial in log(N) where N is the size of the data structure. Example: an ordered list of reasonably short strings can be searched in log2 (N) time via binary search. To measure the time and space complexity of such operations, the usual Turing machine with its serial-access input tape is replaced by a random access model. To compare such problems and define completeness, the appropriate relation is loglog reducibility: the relation generated by random-access transducers whose work tapes have length at most log(log(N)). The surprise is that instead of being a refinement of the standard log space, polynomial time, polynomial space, ... hierarchy, the complexity classes for these random-access Turing machines form a distinct parallel hierarchy, namely, polylog time, polylog space, exppolylog time, ... . Propositional truth evaluation, context-free language recognition and searching a linked list are complete for polylog space. Searching ordered lists and searching unordered lists are complete for polylog time and nondeterministic polylog time respectively. In the serial-access hierarchy, log-space reducibility is not fine enough to classify polylog-time problems and there can be no complete problems for polylog space even with polynomial-time Turing reducibility

Cloud storage services allow files to be synchronized among multiple users or devices easily. To minimize the amount of network traffic, these services utilize incremental data synchronization techniques. However, little is known about their particular mechanisms and corresponding efficiency. In this paper, we focus on Dropbox, the most popular cloud storage service, as a case study. We examine the bandwidth consumption on the Dropbox client under typical synchronization scenarios, and find that Dropbox wastes a lot of traffic due to matching unrelated chunks to compute incremental changes. More seriously, when file conflicts among clients, Dropbox directly downloads the whole file instead of the incremental changes. To solve these problems, we design and implement an efficient incremental synchronization system named Minbox. MinBox employs an efficient locality-sensitive hash for better similar chunk matching. Moreover, Minbox could forward the incremental changes during confliction by maintaining the relation of conflicting files. In comparison with Dropbox, Minbox significantly reduces network traffic and resolves file conflict with little overhead.

The ever changing network traffic reveals new attack types, which represent a security threat that poses a serious risk for enterprise resources.Therefore, the security administrators are in a real need to employ efficient Intrusion Detection and Prevention Systems, IDPS.Such systems might be capable to learn from the network behavior.In this paper, we present an incremental Learnable Model for Anomaly Detection and Prevention of Zero-day attacks, LMAD/PZ.To facilitate the ability of learning from observations that can provide a reliable model for automatic prevention, a comparison has been carried out between supervised and unsupervised learning techniques.Thus, in LMAD/PZ, the intrusion detection step is integrated with an intrusion prevention plan.To ensure that the prevention plan is dependable and automatic, it must be backed and sustained with robust and accurate detection process.Therefore, two incremental data mining techniques are deeply investigated and implemented on NSL-KDD'99 intrusion dataset.The first technique is the Algorithm Quasi-optimal (AQ), which is a supervised Attributional Rules Learner, ARL, while the second is the Cobweb; an unsupervised hierarchical conceptual clustering algorithm.These algorithms categorize the network connections as either normal or anomalous.The performance of AQ is compared to Cobweb, and the best performance result is integrated with the prevention plan, to afford a fully automated system.The experimental results showed that, the model automatically adapts its knowledge base from continuous network streams, in addition to offering the advantage of detecting novel and zero day attacks.Many experiments have verified that AQ performance outperforms the Cobweb clustering, in terms of accuracy, detection rate and false alarm rate.

This paper talks about the reduction of in plant operations time and reduction of turn-around time of the trucks in any FMCG Company. The plant has almost 300 vehicles for loading and unloading. The current systems and process for receiving and dispatching of goods is poised with several challenges. This process with the inspection of material at the entry of a truck to the billing procedures in an effective manner is a challenge due to various constraints. The load, which needs to be dispatched in a day, requires a very less turnaround time of trucks but the actual time, when analysed, was significantly higher than what was required. Higher turnaround time was causing ineffective operations in terms of placing the vehicles in the plant by the transporters. Subsequently, there was a considerable deviation between the planned orders to be dispatched and actual dispatch of goods. The complete process was studied for identifying the loopholes and time motion study was conducted to analyse the current issues, which was leading to more VIVO time of vehicles. The whole process was analysed and thus major factors, which were causing the delay of the vehicles were identified. A survey was conducted to pen down the issues causing the problems in managing the in-plant logistics of the company. Results were analysed and was found that deployment of more manpower and optimizing the entire process would help only marginally towards achieving the goal. Hence, automating the redundant and time-consuming activities would help in achieving a considerable amount of reduction in the turn-around time of trucks. This will indirectly increase the dispatch quantities of the company. In addition, there is a requirement to redefine work, which would primarily mean altering the original process. Thus, various issues in the existing processes were identified and reduction in turnaround time was observed with the implementation of the new process and tools used to reduce the redundant processes