Optimizing cloud infrastructure efficiency through advanced multimedia data deduplication techniques

Public auditing and data deduplication are integral considerations in providing efficient and secure cloud storage services. Nevertheless, the traditional data deduplication models that support public auditing can endure the enormous waste of storage and computation resources induced through data redundancy and repeated audit work by multiple tenants on trusted third-party auditor (TPA). In this work, we introduce blockchain-based secure decentralized public auditing in a decentralized cloud storage with an efficient deduplication model. We employ blockchain to take on the task of centralized TPA, which also mitigates the implications of malicious blockchain miners by using the concept of a decentralized autonomous organization (DAO). Specifically, we employ the idea of redactability for blockchain to handle often neglected security issues that would adversely affect the integrity of stored auditing records on blockchain in decentralized auditing models. However, the proposed model also employs an efficient deduplication scheme to attain adequate storage savings while preserving the users from data loss due to duplicate faking attacks. Moreover, the detailed concrete security analysis demonstrates the computational infeasibility of the proposed model against proof-of-ownership, duplicate faking attack (DFA), collusion attack, storage free-riding attack, data privacy, and forgery attack with high efficiency. Finally, the comprehensive performance analysis shows the scalability and feasibility of the proposed model.

Data deduplication is a valuable technique for compressing and minimizing data duplication during data transfers, especially in cloud environments. By eliminating redundant data, it optimizes transmission capacity and reduces memory usage. To ensure the integrity of sensitive data, encryption is applied throughout the deduplication process. The SHA algorithm is commonly used for storing text data during deduplication. It generates security bits by padding the text and computes a hash consisting of hexadecimal, string, and integer data. Hash-based deduplication involves hashing the entire file and treating the hash values of text data as unique identifiers. This allows clients to identify duplicate data within the cloud. The Proof-of-Work (PoW) algorithm is widely utilized in blockchain networks like Bitcoin and Ethereum. Its primary function is to verify transactions through a process called mining. Miners engage in competition to solve complex mathematical problems, and the first one to find a solution is granted the right to add a new block to the blockchain. PoW relies on cryptographic hashing, such as the SHA-256 hashing function, to validate and secure transactions. On the other hand, the Proof of Retrievability (PoR) algorithm finds application in cloud computing systems. It serves as a consensus mechanism, ensuring that cloud storage providers store and retrieve data accurately. PoR enables cloud providers to demonstrate to consumers that their files can be fully recovered. This algorithm incorporates cryptographic proofs that validate the integrity and availability of the stored data. In cloud storage, deduplication is often implemented using the Memory mapping technique (MPT). This allows multiple data owners to store the same data in a single copy, enhancing storage efficiency. To maintain data security, encryption is applied both before and during the deduplication process, ensuring that sensitive information remains protected. In summary, data deduplication is a powerful method for compressing data, minimizing duplication, and optimizing transmission capacity. With encryption techniques and hash-based identification, it provides secure and efficient deduplication in cloud environments, while memory deduplication and MPT support further enhance performance and storage efficiency.

In today's digital era, the exponential growth of data necessitates effective storage and management solutions. The cloud has vast storage possibilities to store huge amounts of data. Public access to the cloud leads to duplicate copies of data stored in the storage. Maintaining a single copy of data in the cloud is most important for efficient data storage management. This paper introduces a groundbreaking strategy for improving the efficacy of cloud storage through innovative data deduplication techniques at the block levels. The block-level duplication verification efficiently identifies the duplicate data in the storage. It helps to protect the duplicate storage in the cloud data storage infrastructure. The block-level deduplication technique uses variable-length blocks based on the duplicate content of the block. Initially, A file is divided into a number of blocks with a size of 5kb. According to the proposed method, If any block is partially matched with a block already stored in the cloud, then that block is further divided into smaller blocks based on the matching percentage. The smaller blocks help to deduplicate the data more effectively. The work is implemented in a live cloud setting with a C# application hosted on MyASP.NET. The proposed methodology's effectiveness is validated against existing deduplication techniques. The results reveal a marked improvement in storage utilization and data management, affirming the potential of the approach to revolutionize cloud storage efficiency.

A storage system in cloud is well thought-out as a very big scale storage system that has independent storage servers. The service that cloud storage provides is, that can store user's data from remote through network and other authenticated users can access the data easily. Hadoop distributed file system is used to store large files consistently and to retrieve those files at very high bandwidth to user applications. Hadoop splits the files into large blocks and distributes them amongst the nodes in the cluster. When we retrieve data from the cloud, it is very important that the computation and communication overhead should be reduced. To reduce the communication overhead the server should send only the top-n files based on the keyword when the user asks for the data files. Since the owner need not maintain the copy of the files, it is all the more necessary to make check on the files available and also check the originality of the files stored in the server periodically. In HDFS the computation is done in parallel so that the execution time is drastically reduced. In the proposed system for retrieving top-n files we use Hadoop Distributed File System, so that the search time and the communication overhead is greatly reduced.

Recently, cloud computing is increasingly becoming the focus of attention at home and abroad. When the big data storage is become the core of cloud computing system, cloud computing system would evolve into a cloud storage system. With rapidly development of cloud computing, cloud storage has become one of most popular research field. HDFS (Hadoop Distributed File System) has a powerful data storage capacity for cloud storage system. However, it must be improved before applied to large scale since it exists some drawbacks during the design process. This paper researches the cloud storage model based on HDFS, solves uneven replica distribution.

Hadoop distributed file system (HDFS) was originally designed for streaming access large files, but the access and storage efficiency is low for the mass small files. This paper presents an access optimization approach for HDFS small file based on MapFile: TLB-MapFile. TLB-MapFile merges massive small files into large files by MapFile mechanism to reduce NameNode memory consumption and add fast table structure (TLB) in DataNode, and to improve retrieval efficiency of small files. First, according to MapFile mechanism, small files are merged into large files and stored in HDFS. Second, the access frequency and the ordered queue of small files (per unit time) can be obtained through accessing system audit logs in HDFS, and the mapping information between block and small files are stored in the TLB table with regularly being updated. TLB-MapFile improves access efficiency of small files through the prefetching of priori strategies based on TLB table. Experiment results show that this method can effectively reduce NameNode memory consumption and improve the reading speed of small files.

In order to improve storage efficiency of small files in the cloud storage systems based on HDFS (Hadoop Distributed File System), this paper proposed a merging process approach based on a two-dimensional packing algorithm, called TDPHDFS (two-dimensional packing for HDFS). In it the correlations between file size and arrival time are comprehensively considered to assist the small files to be merged into large ones. The simulation results demonstrate that the storage efficiency of small files is improved, while the stability remains the same, yet less resource is consumed. The TDPHDFS algorithm can effectively reduce the performance penalty in both storage space and memory consuming while managing massive small files.

Cloud security has gained increasingly emphasis in the research community, with much focus primary concentrated on how to secure the operation system and virtual machine on which cloud system runs on. We take an alternative perspective to consider the problem of building a secure cloud storage service on top of a public cloud infrastructure where the service provider is not completely trusted by the customer. So, it is necessary to put cipher text into the public cloud. We describe an architecture based on Trusted Platform Module and the client of cloud storage system to help manage the symmetric keys used for encrypting data in the public cloud and the asymmetric keys used for encrypting symmetric keys. The key management mechanism includes how to store keys, how to backup keys, and how to share keys. Based on the HDFS (Hadoop Distributed File System), we put a way of key management into practice, and survey the benefits that such an infrastructure will provide to cloud users and providers, and we also survey the time cost it will bring to us.

Cloud computing becomes a popular trend in recent years. In the meanwhile, content delivery network has been used for many years to distribute content all over the world. The relation between cloud computing and content delivery network is very interesting. In this paper, we proposed a cloud based content delivery network. This architecture integrates cloud infrastructure and content delivery network system together. It provides a low cost and elastic web serving cloud platform. Besides, it also provides easy way to manage data for web content developers, and accelerate the speed for users to access web sites. This system uses Hadoop Distributed File System as a cloud infrastructure service, and creates many data clusters around the world. Therefore, content providers could place content to closest node from end users. The result shows a high performance and flexible solution to build a personal content delivery network over cloud.

Cloud Computing is the delivery of computing as a service, which is specifically involved with Storage of data, enabling ubiquitous, convenient access to shared resources that are provided to computers and other devices as a utility over a network. Storage, which is considered to be the key attribute, is hindered by the presence of redundant copies of data. Data Deduplication is a specialized technique for data compression and duplicate detection for eliminating duplicate copies of data to make storage utilization efficient. Cloud Service Providers currently employ Hashing technique so as to avoid the presence of redundant copies. Apparently, there are a few major pitfalls which can be vanquished through the employment of a Nature - Inspired, Genetic Programming Approach, for deduplication. Genetic Programming is a systematic, domain - independent programming model making use of the ideologies of biological evolution so as to handle a complicated problem. A Sequence Matching Algorithm and Levenshtein's Algorithm are used for Text Comparison and then Genetic Programming concepts are utilized to detect the closest match. The performance of these three algorithms and hashing technique are compared. Since bio-inspired concepts, systems and algorithms are found to be more efficient, a Nature-Inspired Approach for data deduplication in cloud storage is implemented.

Cloud computing has empowered the individual user by providing seemingly unlimited storage space and availability and accessibility of data anytime and anywhere. Cloud service providers are able to maximize data storage space by incorporating data deduplication into cloud storage. Although data deduplication removes data redundancy and data replication, it also introduces major data privacy and security issues for the user. In this paper, a new privacy-preserving framework that addresses this issue is proposed. Our framework uses an efficient deduplication algorithm to divide a given file into smaller units. These units are then encrypted by the user using the combination of a secure hash function and a block encryption algorithm. An index tree of hash values of these units is also generated and encrypted using an asymmetric search encryption scheme by the user. This index tree will enable the cloud service provider to search through the index and return the requested units. We will show that our proposed framework will allow cloud service and storage providers to employ data deduplication techniques without giving them access to either the users' plaintexts or the users' decryption keys.

With the rapidly growing amount of remote sensing data, data management needs to adopt a new architecture. Some improvements have been made in some areas. For example, the Hadoop distributed file system (HDFS) can be utilised for large files. However, small files will be produced when storing remote sensing images in the pyramid-based structure. In this paper, we propose a method based on the Hadoop system with MapFile. The method is an improved storage model for efficient storage and allows for access of the small files on the HDFS. The proposed method combines small files into the MapFile serialised container, and it reduces the number of small files. The metadata sensor information is stored in the index to improve the search speed. Results demonstrate the efficiency of remote sensing image processing using the pyramid model and the reduced execution time compared with the HDFS, HDWebGIS and Hadoop archives.

Replica selection strategy have been used in Cloud Storage to reduce data access latency, eliminate bottlenecks of hot spots and achieve load balance. This paper studies the replica selection strategy and replication strategy. By concerning real-time bandwidth, performance of the peer, and replica's history request, following the principle that high valued replica has high proportionality to be select, a proportional matching based self-convergent replica selection strategy (PMC) based on HDFS (Hadoop Distributed File System) cloud storage environment is presented. The strategy has global optimization, effectively reduces access delay, balances load, eliminates hot bottlenecks. The simulation results show that the optimal selection strategy offers better performance by shortening the response time of replica selection.

This study compares data dispersion and encryption techniques with an emphasis on Hadoop Distributed File System (HDFS) and Advanced Encryption Standard (AES) in secure cloud storage systems. The study uses 29,502 samples using ClincalC with G-power values (α=0.05, β=0.2). Data dispersion was analyzed using synthetic datasets, AES encryption was verified in a controlled cloud environment, and HDFS functioning was assessed using Hadoop clusters. AES was shown through statistical analysis in SPSS to be a successful data security solution that guarantees confidentiality. Scalable and fault-tolerant storage was made possible by Hadoop (HDFS) (82.14% vs. 61.54%, p<0.02), showing strong performance but necessitating more security precautions for complete data protection.

We are stepping frontward for an epoch of zeta bytes from Giga/Tera/Peta/Exa bytes in this era of computer science. The data storage in clouds is not the only preference as big data technology is obtainable for processing of both structured and unstructured data. Today a colossal sum of data is engendered by mobile phones (Smartphone) of both the composition types. For the sake of faster processing and elegant data utilization for gigantic quantity of data CDroid operating systems can be implemented with big data. Big data architecture is realized with mobile cloud for paramount deployment of wherewithal. The faster execution can be ended feasible for the make use of a new data centric architecture of MapReduce technology where as HDFS (Hadoop Distributed File System) also plays a big accountability in using data with dissimilar structures. As time advances the degree of data and information generated from smartphone augments and faster execution is call for the same. As per our research and development the only resolution for this mammoth amount of data is to put into practice Big data with CDroid scheme for best use of it. We believe this effort will budge a step ahead on the road to healthier civilization in close proximity to expectations.