Image Classification Based on Layered Gradient Clipping Under Differential Privacy

The report presents an algorithm for constructing a convolutional hierarchical neural network classifier, which is a modification of the algorithm for constructing hierarchical neural network classifiers suggested before. The original algorithm was designed to exploit intrinsic class hierarchy to build a class tree with a neural network in each node classifying groups of initial classes (in a non-terminal node) or a subset of original classes (in a terminal node). The convolutional modification utilizes convolutional neural networks instead of regular fully connected networks in order to apply the model to image classification tasks. Use of class hierarchy for image classification should reduce the number of adjusted neural network parameters compared to deep convolutional neural networks, and therefore it should reduce training and inference time. In this context the algorithm may be compared with some pruning techniques. The convolutional hierarchical neural network classifier inherits some hyperparameters of a conventional hierarchical neural network classifier, like the activation threshold and the threshold by the share of voting patterns. The goal of this study was to explore different strategies of choosing these hyperparameters. To test these strategies, we used the CIFAR-10 dataset. Also, for demonstration purposes we apply the convolutional hierarchical neural network classifier to the CIFAR-100 dataset.

Differential privacy (DP) is introduced into many fields of AI to preserve privacy. However, introducing DP into AI models is extremely error-prone. To verify whether DP AI models can provide privacy guarantee (quantified by privacy budget) as these models claim, existing methods utilize attack methods to audit whether privacy budget of these models is the same as these models claim. To further improve precision of privacy budget auditing, we propose a brand new way to audit privacy budget, namely directly utilizing the parameters of DP AI models to audit privacy budget. In particular, our method utilizes statistical characteristics variance of the output distribution of DP mechanism to audit privacy budget of DP mechanism. DP AI models are regarded as data samples from output distribution of DP AI model training method and are utilized to approximate the variance of output distribution. The approximated variance is leveraged to estimate the variance of noise distribution of DP mechanism and through the relationship between noise variance and privacy budget, our method calculates the audited privacy budget through estimated noise variance. In addition, to reduce computation overhead, our method constructs parameter selection strategy to identify position whose parameter is suitable for privacy budget auditing. Comprehensive experiments are conducted to verify the effectiveness of our auditing method. Comparison results of five competitive auditing methods demonstrate that our method decreases MAE by 18.29% and decreases MSE by 23.17% on experiment datasets.

Federated learning is a distributed machine learning technique that allows multiple devices to collaborate on learning a shared model without exchanging data. It can be used to improve model accuracy while protecting user privacy. However, traditional federated learning is vulnerable to attacks from generative adversarial networks (GANs). As a new privacy protection method, differential privacy enhances privacy protection capabilities by sacrificing some data accuracy. To optimize the privacy budget allocation scheme in traditional differential privacy, we propose a differential privacy method called ADP-FL, which dynamically adjusts the privacy budget based on Newton’s Law of Cooling. While maintaining the overall privacy budget, it dynamically tunes adaptive parameters to improve training accuracy. Additionally, we propose an asynchronous federated learning aggregation scheme that combines privacy budget with data freshness, thereby reducing the impact of differential privacy on accuracy. We conducted extensive experiments on differential privacy algorithms based on Gaussian mechanisms and Laplace mechanisms. The experimental results show that, under the same privacy budget, our algorithm achieves higher accuracy and lower communication overhead compared to the baseline algorithm.

This paper demonstrates image classification using deep learning. Deep learning has the inherent ability to automatically discover and extract meaningful features for specific applications. Among the popular techniques in deep learning, the convolutional neural network (CNN) stands out. CNN consists of an input layer, hidden layers, and an output layer, where meaningful features are automatically extracted from input images. This paper presents the performance and identifies the most effective CNN architectures for accurately classifying images in the CIFAR-10 dataset. Five CNN architectures were implemented, namely [Architecture 1], [Architecture 2], [Architecture 3], [Architecture 4], and [Architecture 5] using the CIFAR-10 dataset. The architectures were selected based on the need to explore variations in convolutional filter sizes, dense layers, and batch normalization to assess their impact on CIFAR-10 image classification performance. Each architecture was trained on a standard training set and evaluated on a validation set. We used specific details on data preprocessing and training settings for a consistent and fair comparison. After training and evaluation, we have obtained the following results for each architecture. Architecture 1 has a training accuracy of 74.7% and validation accuracy of 76.6%, Architecture 2 has 96.09% and 86.08%, Architecture 3 has 77% and 78.1%, Architecture 4 has 67.91% and 69.54%, and Architecture 5 has 94.64% and 87.34% of training accuracy and validation accuracy respectively. After conducting a comparative analysis, we found that Architecture 5 has achieved the highest validation accuracy in classifying images in the CIFAR-10 dataset. These findings suggest that Architecture 5 is a promising choice for image classification tasks involving the CIFAR-10 dataset.

Recently, machine learning has been frequently applied in multiple areas, like image classification and object recognition. However, some machine learning applications train their models on sensitive data that could leak personal information, leading to potential privacy issues. Many privacy attacks against deep learning models have been proposed. A membership inference attack was demonstrated to be a type of effective attack and a real threat to privacy in machine learning. Here, we trained three different convolutional neural networks for image classification with the CIFAR10 dataset and examined the effects of a membership inference attack on these models. We also explore different defense strategies, including differential privacy against membership inference attack. We showed that membership inference attack was more effective against overfitting models and defense strategies like differential privacy and regularization can significantly lower the attack accuracy.

Differential privacy is often used in location privacy protection because of its strict reasoning and proof privacy guarantee. When users make continuous location query, it will cause noise superposition, which leads to the decline of query accuracy. At present, although differential privacy based on rule tree structure can reduce the query error, it will generate a lot of invalid zero nodes. The data structure is too large, and more improvement in the query accuracy can be further investigated. In this paper, we proposed a differential privacy location privacy protection method based on generative adversary network. Firstly, the definition of location data privacy protection under differential privacy mechanism is given, and then resume density aware network under differential privacy mechanism. Based on density aware network, the privacy protection problem of location data can be transformed into the distribution of fitting trajectory length. Finally, we use Markov chain to generate a new trajectory, and introduce the generative adversarial network to construct a set differential privacy protection method. Compared with other methods to improve the accuracy of differential privacy query, this method can effectively reduce the problem of query accuracy decline caused by noise superposition in continuous query, and can adapt to lbs location query service in different density environments.KeywordsGenerative adversarial networkLocation differencePrivacy protectionMarkov chainDensity aware network

The paper investigates the impact of preliminary image restoration using a denoising autoencoder (DAE) on the classification accuracy of a convolutional neural network (CNN) under various types of noise. The relevance of the topic is due to the fact that in real conditions, optical images often contain distortions caused by changes in lighting, vibrations, camera movement, and other factors, which significantly complicates the task of object recognition. Traditional filters do not always provide sufficient cleaning quality and can lead to the loss of important structural features. In this regard, the use of deep neural networks, in particular autoencoders, is a promising direction for improving the robustness of computer vision algorithms to noise of various nature. The study uses the CIFAR-10 dataset and implements a two-component model: an autoencoder for preliminary cleaning and a CNN for classification. The trained autoencoder restores the image structure after exposure to Gaussian, impulse, Poisson, and speckle noise. Three series of experiments were conducted: classification of clean images, classification of noisy data without cleaning, and classification after preliminary restoration by the autoencoder. The results showed that CNN demonstrates an accuracy of 70.37% on clean data, but when noise is introduced, the accuracy drops to 30–59% depending on the type of distortion. After applying the autoencoder, classification accuracy increased to 56–60% for all types of noise, with the greatest improvement observed for Gaussian noise with high dispersion. The results confirm that using an autoencoder as a preliminary restoration step is an effective method for improving classification accuracy and reducing CNN vulnerability to noise. This approach provides better generalization and stability of the system, which is especially important for real-time applications—in particular, in dynamic systems, robotics, autonomous transport, and navigation systems, where the quality of optical data is often unstable. The study demonstrates the promise of integrating restoration and classification models into a single structure to improve the performance of computer vision systems in challenging conditions.

With the rapid development of the Internet, privacy protection in big data has become an important research hotspot in the field of information security. Among them, differential privacy protection is the most widely used big data privacy protection technology today. However, the existing differential privacy protection technology cannot fully and effectively deal with the problem of high-dimensional privacy big data. Although the private data publishing method based on a Bayesian network can effectively deal with the conversion problem of high-dimensional data sets to low-dimensional data sets, this method also has some defects and deficiencies. Therefore, this paper proposes a large data source differential privacy protection algorithm (NIE-PrivBayes algorithm) based on normalized information entropy Bayesian network and proposes three improvements to the original PrivBayes algorithm. First, the original data set is protected by differential privacy on the client side. Secondly, the Bayesian network is constructed by the joint probability distribution obtained by the EM algorithm. Finally, the existing Bayesian network construction algorithm is optimized in the aspects of normalized information entropy. Based on significantly reducing the computational overhead, the effectiveness and availability of publishing high-dimensional data sets are realized under the premise of differential privacy protection.

In response to the privacy leakage risks inherent in the big data processing of power user personas, propose a collaborative optimization‐based joint privacy protection mechanism for Differential Privacy (DP) and Secret Sharing (SS). This method reveals the inherent correlation between DP noise parameters and SS sharding strategy and designs an adaptive noise sharding mapping mechanism to achieve joint optimization of privacy protection and communication overhead. Based on this mechanism, a specific privacy protection scheme for power user personas is designed. Experimental results demonstrate that the DP + SS privacy protection method designed in this study achieves an accuracy rate of 93.14% on the MNIST dataset and maintains an accuracy rate of 92.06% across multiple datasets, indicating that the scheme effectively quantifies the level of privacy protection. The privacy protection level of the scheme is quantitatively defined through differential privacy budget ε ∈ [0.11,0.15], secret sharing ( k = 3, n = 5) threshold mechanism, and privacy leakage probability δ = 10 −5 . The privacy budget is dynamically allocated through the correlation between differential privacy noise variance and the minimum reconstruction threshold of secret sharing, ensuring that privacy risks are controllable and quantifiable. At the same time, the average training time for servers is about 186.4 s, which refers to the total end‐to‐end training time in a four‐server cluster environment. Compared with a single server encryption scheme, the efficiency is improved by 48.6%. The communication overhead has increased to 10.6GB, which is the total chip transmission overhead per 10,000 users. However, the calculation stability can be maintained through load balancing, indicating that the scheme has certain feasibility. The aforementioned research results not only provide a viable applied cryptographic technology solution for privacy protection in power big data but also offer new insights into balancing privacy protection strength and data availability. More importantly, the joint privacy protection mechanism proposed by the research institute can serve as a fundamental enabling technology for the infrastructure of a secure smart grid, providing quantifiable and trustworthy privacy and security guarantees for advanced data‐driven applications such as precise load prediction, demand‐side response, and collaborative optimization of distributed energy, thereby supporting the reliable and intelligent development of the smart grid.

PurposeThe purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets.Design/methodology/approachThis research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where “n” denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths.FindingsThe findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks.Originality/valueThis study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

Privacy budget management plays an important role when applying differential privacy, as it sets an upper limit in the ability to utilise the private database. In this paper, we explore the possibility of extending the total allocated privacy budget, taking into consideration the data consumer characteristics and the data utilisation context. To this end, we first study the problem of privacy budget distribution in adaptive multi-data consumer differential privacy use cases. Then, we present an extension of the classic differential privacy formal model that allows taking into consideration data consumers’ information disclosure risk when distributing the privacy budget among them. Finally, we define a method that allows to optimally distribute a given privacy budget among a private database’s data consumers.

Data mining technology can be used to dig out potential and valuable information from massive data, and support vector machine (SVM) is one of the most widely used and most efficient methods in the field of data mining classification. However, the training set data often contains sensitive attributes, and the traditional training method of SVM reveals the individual privacy information. In view of the low prediction accuracy and poor versatility of the existing SVM classifiers with privacy protection, this paper proposed a new SVM training method for differential privacy protection. The algorithm first solved the dual problem of SVM by using SMO method and the difference $E_{i}$ between the estimated value and the real value for each support vector was recorded. Then the ratio of the $E_{i}$ of each support vector to the sum of the $E_{i}$ of all the support vectors was calculated. Next, different levels of Laplace random noise were added to the corresponding dual variables $\alpha _{i}$ of each support vector to be released, according to the ratio of each support vector. According to the principle of differential privacy protection, the algorithm meets $\epsilon $ -differential privacy which can be used to effectively protect individual privacy. Experimental results on real datasets showed that the algorithm proposed in this paper could be used for classification prediction under a reasonable privacy budget.

The convolution neural networks have revolutionized the computer vision domain. It has proven to be a dominant technology to carry out tasks such as image classification, semantic segmentation, and object detection. The convolution neural networks surpass the performance of the existing algorithms such as SIFT, HOG, etcetera. Where, instead of manually engineering the features, supervised learning help to learn the essential low-level and high-level features necessary for classifications. The convolution neural networks have become a popular tool to counter computer vision problems. However, it is computationally, and memory intensive to train and deploy the network because of the model size of a deep convolution neural networks. However, the research in the field of design space exploration (DSE) of neural networks and compression techniques to develop compact architectures, have made convolution neural networks memory and computationally efficient. These techniques have also improved the feasibility of convolution neural network for deployment on embedded targets. The paper explores the concept of compact convolution filters to reduce the number of parameters in a convolution neural network. The intuition behind the approach is that replacing convolution filters with a stack of compact convolution filters helps in developing a compact architecture with competitive accuracy. This paper explores the fire module a compact convolution filter and proposes a method of recreating a state-of-the-art architecture VGG-16 using the fire modules to develop a compact architecture, which is further trained on the CIFAR-10 dataset and deployed on a real-time embedded platform known as Bluebox 2.0 by NXP using RTMaps software framework.

The Stochastic gradient descent algorithm (SGD) is a classical algorithm for model optimization in machine learning. Introducing a differential privacy model to avoid privacy leakages in the optimization iteration process can achieve the balance of training accuracy and data availability. In addition, a fixed number of iterations are chosen in a conventional implementation scheme. At each iteration, parameters are updated with a noisy gradient. However, the privacy budget is mostly split evenly to each iteration without taking into account the difference in the privacy leakage risk under optimal processing. In this paper, we improve the SGD-based algorithms by appropriately allocating the privacy budget for each iteration. Intuitively, the gradient value is inversely proportional to the number of iterations. The closer the parameter is to its optimal objective value, the smaller the gradient is, and hence the gradients need to be measured more accurately. We propose an adaptive “noise reduction” algorithm that can be applied to private SGD-based empirical risk minimization (ERM) algorithms, meets the accuracy constraint simultaneously. We apply our approach to the backpropagation (BP) neural network. In the experiment, we show and validate that the proposed noise parameter configuration method provides sufficient privacy protection and improves the accuracy of data utility.

Personal data privacy is a great concern by governments across the world as citizens generate huge amount of data continuously and industries using this for betterment of user centric services. There must be a reasonable balance between data privacy and utility of data. Differential privacy is a promise by data collector to the customer’s personal privacy. Centralised Differential Privacy (CDP) is performing output perturbation of user’s data by applying required privacy budget. This promises the inclusion or exclusion of individual’s data in data set not going to create significant change for a statistical query output and it offers -Differential privacy guarantee. CDP is holding a strong belief on trusted data collector and applying global sensitivity of the data. Local Differential Privacy (LDP) helps user to locally perturb his data and there by guaranteeing privacy even with untrusted data collector. Many differential privacy algorithms handles parameters like privacy budget, sensitivity and data utility in different ways and mostly trying to keep trade-off between privacy and utility of data. This paper evaluates differential privacy algorithms in regard to the privacy support it offers according to the sensitivity of the data. Generalized application of privacy budget is found ineffective in comparison to the sensitivity based usage of privacy budget.