“Atrain Distributed System” (ADS): An Infinitely Scalable Architecture for Processing Big Data of Any 4Vs
The present day world dealing with big data (expanding very fast in 4Vs: Volume, Varity, Velocity and Veracity) needs New advanced kind of logical and physical storage structures, New advanced kind of heterogeneous data structures, New mathematical theories and New models: all these four together we call by 4Ns. As on today, the 4N-set lagging behind in the race with 4V-set. If 4V-set continues its dominance over the 4N-set with respect to time, then it will be difficult to the world to think of “BIG DATA: A Revolution That Will Transform How We Live, Work, and Think”. The main objective of this chapter is to report the latest development of an easy and efficient method for processing big data with 4Ns. For processing of giant big data in 4Vs, neither any existing data structure alone nor any existing type of distributed system alone is sufficient. Even the existing network topologies like tree topology or bus/ring/star/mesh/hybrid topologies seem to be weak topologies for big data processing. For a success, there is no other way but to develop ‘new data structures’ for big data, ‘new type of distributed systems’ for big data having a very fast and tremendous extent of mutual compatibility and mutual understanding with the new data structures, ‘new type of network topologies’ for big data to support the new distributed system, and of course ‘new mathematical/logical theory’ models for big data science. The next important issue is how to integrate all these ‘new’ to make ultimately a single and simple scalable system to the laymen users with a new simple ’big data language’. With these views in mind, a special type of distributed system called by ’Atrain Distributed System’ (ADS) is designed which is very suitable for processing big data using the heterogeneous data structure r-atrain for big data (and/or the homogeneous data structure r-train for big data). Consequently, (horizontally) and depth (vertically) can be achieved in ADS by a simple process and this is the unique and extremely rich merit of ADS to challenge the 4Vs of big data. Where r-atrain and r-train are the fundamental data structures for processing big data, the data structures ‘heterogeneous data structure MA’ and ‘homogeneous data structureMT’ are higher order data structures for the processing of big data including temporal big data. Both MA andMT can be well implemented in multi-tier ADS. In fact ‘Data Structures for Big Data’ is to be regarded as a new subject, not just a new topic in big data science. The classical Matrix Theory has been studied by the world with unlimited volume of applications in all branches of Science, Engineering, Statistics, Optimization, Numerical Analysis, Computer Science, Medical Science, Economics, etc. The classical matrices are mainly two dimensional having the elements from the set of real numbers in most of the cases. An infinitely scalable notion of ‘solid matrix’ (SM) i.e. n-dimensional hyper-matrix, is introduced as a generalization of the classical matrix, and also introduced the notion of ‘solid latrix’ (SL). In the ‘Theory of Solid Matrices’, the corresponding matrix-algebra is developed explaining various operations, properties and propositions on them for the case while the elements are objects of the real region RR. As a generalization of the n-SM, another new notion called by ‘solid hematrix’ (SH) (solid heterogeneous matrix i.e. n-dimensional hyper-matrix with heterogeneous data) is introduced.A method is proposed on howto implement Solid Matrices, n-dimensional arrays, n-dimensional larrays etc. in a computer memory using the data structures MT and MA for big data. The combination of r-atrain (r-train) with ADS and the combination of MA (MT) with multitier ADS can play a major role in a new direction to the present data-dependent giant galaxies of organizations, institutions and individuals to process any big data irrespective of the influence of 4Vs.
Read more