Locally Differentially Private Frequency Estimation via Joint Randomized Response

Personal data privacy is a great concern by governments across the world as citizens generate huge amount of data continuously and industries using this for betterment of user centric services. There must be a reasonable balance between data privacy and utility of data. Differential privacy is a promise by data collector to the customer’s personal privacy. Centralised Differential Privacy (CDP) is performing output perturbation of user’s data by applying required privacy budget. This promises the inclusion or exclusion of individual’s data in data set not going to create significant change for a statistical query output and it offers -Differential privacy guarantee. CDP is holding a strong belief on trusted data collector and applying global sensitivity of the data. Local Differential Privacy (LDP) helps user to locally perturb his data and there by guaranteeing privacy even with untrusted data collector. Many differential privacy algorithms handles parameters like privacy budget, sensitivity and data utility in different ways and mostly trying to keep trade-off between privacy and utility of data. This paper evaluates differential privacy algorithms in regard to the privacy support it offers according to the sensitivity of the data. Generalized application of privacy budget is found ineffective in comparison to the sensitivity based usage of privacy budget.

У статті здійснено когнітивне моделювання розвитку загроз приватності особистості та безпеці країни. Для моделювання обрано такі фактори: «Приватність особистості», «Безпека розвитку країни», «Розробка систем захисту», «Онлайн-присутність», «Кібер-атаки», «Втрати від кіберзлочинів», «Витрати на захист», «Сталий розвиток». Результати показали, що всі фактори в системі є дестабілізаційними й чутливими до змін. Найбільш чутливим є фактор «Онлайн-присутність». Такий характер системи свідчить про те, що будь-які зміни можуть дестабілізувати систему, привести до змін усі інші фактори. При цьому система сама сприятиме таким змінам, оскільки є нестійкою. Отже, уся система забезпечення захисту приватності особистості та безпеки країни потребує постійної уваги й моніторингу змін усіх факторів. Фактори «Приватність особистості» та «Безпека розвитку країни» є взаємозалежними, оскільки демонструють майже однаковий вплив на них усіх інших факторів системи. Так, ці фактори підсилюватимуться при зростанні факторів «Розробка систем захисту», «Онлайн-присутність», «Витрати на захист» і «Сталий розвиток». Зростання рівня захисту приватності особистості сприятиме зростанню рівня безпеки країни й навпаки. При цьому «Приватність особистості» сильніше впливає на зміцнення рівня безпеки країни, порівняно з впливом безпеки держави на захист приватності особистості при її присутності в мережі Інтернет. Лише два фактори – «Кібератаки» та «Втрати від кіберзлочинів» зменшуватимуть рівень захисту приватностій безпеки країни при їх зростанні. Рівень захисту приватності є більш чутливим до впливу на неї з боку кібератак, порівняно з чутливістю рівня безпеки країни. На рівень безпеки розвитку держави найбільше впливає рівень захисту приватності особистості, і цей вплив на 14 % більш потужний, порівняно з протилежним впливом факторів. Аналіз засвідчив, що основним пріоритетом під час розвитку мережевих технологій, упровадженні ІКТ у всі сфери суспільного життя та розвитку інформаційної економіки повинно бути забезпечення захисту приватності особистості, її персональних даних. Забезпечення безпеки розвитку країни потребуватиме більшого рівня розвитку систем захисту, порівняно із захистом приватності особистості, при цьому на 4 % менше впливатиме на сталий розвиток.

With the advent of the era of big data, privacy issues have been becoming a hot topic in public. Local differential privacy (LDP) is a state-of-the-art privacy preservation technique that allows to perform big data analysis (e.g., statistical estimation, statistical learning, and data mining) while guaranteeing each individual participant’s privacy. In this paper, we present a comprehensive survey of LDP. We first give an overview on the fundamental knowledge of LDP and its frameworks. We then introduce the mainstream privatization mechanisms and methods in detail from the perspective of frequency oracle and give insights into recent studied on private basic statistical estimation (e.g., frequency estimation and mean estimation) and complex statistical estimation (e.g., multivariate distribution estimation and private estimation over complex data) under LDP. Furthermore, we present current research circumstances on LDP including the private statistical learning/inferencing, private statistical data analysis, privacy amplification techniques for LDP, and some application fields under LDP. Finally, we identify future research directions and open challenges for LDP. This survey can serve as a good reference source for the research of LDP to deal with various privacy-related scenarios to be encountered in practice.

In local differential privacy (LDP), each user perturbs her data locally before sending the noisy data to a data collector. The latter then analyzes the data to obtain useful statistics. Unlike the setting of centralized differential privacy, in LDP the data collector never gains access to the exact values of sensitive data, which protects not only the privacy of data contributors but also the collector itself against the risk of potential data leakage. Existing LDP solutions in the literature are mostly limited to the case that each user possesses a tuple of numeric or categorical values, and the data collector computes basic statistics such as counts or mean values. To the best of our knowledge, no existing work tackles more complex data mining tasks such as heavy hitter discovery over set-valued data. In this paper, we present a systematic study of heavy hitter mining under LDP. We first review existing solutions, extend them to the heavy hitter estimation, and explain why their effectiveness is limited. We then propose LDPMiner, a two-phase mechanism for obtaining accurate heavy hitters with LDP. The main idea is to first gather a candidate set of heavy hitters using a portion of the privacy budget, and focus the remaining budget on refining the candidate set in a second phase, which is much more efficient budget-wise than obtaining the heavy hitters directly from the whole dataset. We provide both in-depth theoretical analysis and extensive experiments to compare LDPMiner against adaptations of previous solutions. The results show that LDPMiner significantly improves over existing methods. More importantly, LDPMiner successfully identifies the majority true heavy hitters in practical settings.

Local differential privacy (LDP) provides a way for an untrusted data collector to aggregate users' data without violating their privacy. Various privacy-preserving data analysis tasks have been studied under the protection of LDP, such as frequency estimation, frequent itemset mining, and machine learning. Despite its privacy-preserving properties, recent research has demonstrated the vulnerability of certain LDP protocols to data poisoning attacks. However, existing data poisoning attacks are focused on basic statistics under LDP, such as frequency estimation and mean/variance estimation. As an important data analysis task, the security of LDP frequent itemset mining has yet to be thoroughly examined. In this paper, we aim to address this issue by presenting novel and practical data poisoning attacks against LDP frequent itemset mining protocols. By introducing a unified attack framework with composable attack operations, our data poisoning attack can successfully manipulate the state-of-the-art LDP frequent itemset mining protocols and has the potential to be adapted to other protocols with similar structures. We conduct extensive experiments on three datasets to compare the proposed attack with four baseline attacks. The results demonstrate the severity of the threat and the effectiveness of the proposed attack.

In the context of the Internet of Things (IoT), the large-scale generation and collection of data can greatly improve the quality of service provided, but they also raise significant concerns about privacy breaches. However, existing privacy-preserving data collection solutions based on local differential privacy (LDP) often struggle to balance security and accuracy when handling composite data types. To address this challenge, in this paper, we propose CSKV, a high-precision and privacy-preserving key-value data collection scheme. Specifically, we first design a padding and sampling protocol to improve data utility. Then, we propose two randomized response mechanisms to safely perturb keys and values in a cohesive and segmented manner. After that, by leveraging the sampling protocol and key-value correlation perturbation, we demonstrate that CSKV can provide secondary privacy amplification. Detailed theoretical analysis verifies the security and effectiveness of CSKV. In addition, extensive performance evaluations are conducted on synthetic and real-world datasets, and the results indicate that our proposed scheme outperforms existing schemes in terms of hit rate and estimation variance.

Frequency estimation mechanisms are widely applied in domains such as machine learning and cloud computing, where it is desirable to provide statistical information. As a fundamental operation in these domains, frequency estimation utilizes personal data which contains sensitive information while it is necessary to protect sensitive information from others. Motivated by this, we preserve user’s privacy with local differential privacy by obfuscating personal data on the user side. In this paper, we propose frequency estimation mechanisms under utility-optimized local differential privacy (ULDP), which allow the data collector to obtain some non-sensitive values to improve data utility while protecting sensitive values from leaking sensitive information. We propose three frequency estimation mechanisms under <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(\epsilon ,\delta )$</tex-math></inline-formula> -ULDP (uRFM-GRR, uRFM-RAPPOR, uRFM-OLH) to preserve user’s sensitive information. Our proposed mechanisms protect sensitive data with the same privacy guarantee and they are suitable for different scenarios. Besides, in theory, we compare the estimation errors of our proposed mechanisms with existing LDP based mechanisms and show that ours are lower than theirs. Finally, we conduct experiments on synthetic and real-world datasets to evaluate the performance of the three mechanisms. The experimental results demonstrate that our proposed mechanisms are better than the existing LDP based solutions over the same privacy level, while uRFM-OLH frequently performs the best.

Local differential privacy federated learning has attracted wide attention because it can solve the problem of data islands without damaging data privacy, but it faces data heterogeneity problems on the client side. Existing solutions commonly cluster clients with similar data distributions into the same training groups by leveraging clients’ model parameters, thus mitigating the impact of data heterogeneity on federated learning performance. However, model parameters, as an indirect representation for client data, often fail to accurately reflect the true data distribution, resulting in inaccurate client groupings. Furthermore, these solutions perturb high-dimensional parameter vectors dimension-by-dimension to protect client data privacy, which introduces substantial LDP noise that significantly further compromises the client grouping accuracy. To tackle these challenges, we propose PCFed-LDP, a privacy-preserving clustered federated learning framework that improves the federated learning performance while protecting client data and satisfying LDP in heterogeneous environments. Specifically, we introduce a client clustering method based on geometric properties of client data subspaces, which conducts label grouping-based principal angle analysis on client data subspaces to accurately capture the similarities in client data distributions, thereby enabling precise client grouping. To reduce the amount of introduced LDP noise, we design an adaptive noise addition method that utilizes the Haar wavelet technique to decouple the relationship between the noise amount and vector dimensionality, and employs noise error minimization strategy-based vector segmentation to inject LDP noise with finer granularity. Theoretical analysis and experiments on real datasets demonstrate that our solution not only satisfies the constraints of local differential privacy but also outperforms state-of-the-art methods.

Federated learning (FL) allows to train a massive amount of data privately due to its decentralized structure. Stochastic gradient descent (SGD) is commonly used for FL due to its good empirical performance, but sensitive user information can still be inferred from weight updates shared during FL iterations. We consider Gaussian mechanisms to preserve local differential privacy (LDP) of user data in the FL model with SGD. The trade-offs between user privacy, global utility, and transmission rate are proved by defining appropriate metrics for FL with LDP. Compared to existing results, the query sensitivity used in LDP is defined as a variable, and a tighter privacy accounting method is applied. The proposed utility bound allows heterogeneous parameters over all users. Our bounds characterize how much utility decreases and transmission rate increases if a stronger privacy regime is targeted. Furthermore, given a target privacy level, our results guarantee a significantly larger utility and a smaller transmission rate as compared to existing privacy accounting methods.

Federated learning (FL), as a type of collaborative machine learning framework, is capable of preserving private data from mobile terminals (MTs) while training the data into useful models. Nevertheless, from a viewpoint of information theory, it is still possible for a curious server to infer private information from the shared models uploaded by MTs. To address this problem, we first make use of the concept of local differential privacy (LDP), and propose a user-level differential privacy (UDP) algorithm by adding artificial noise to the shared models before uploading them to servers. According to our analysis, the UDP framework can realize <inline-formula><tex-math notation="LaTeX">$(\epsilon _{i}, \delta _{i})$</tex-math></inline-formula> -LDP for the <inline-formula><tex-math notation="LaTeX">$i$</tex-math></inline-formula> th MT with adjustable privacy protection levels by varying the variances of the artificial noise processes. We then derive a theoretical convergence upper-bound for the UDP algorithm. It reveals that there exists an optimal number of communication rounds to achieve the best learning performance. More importantly, we propose a communication rounds discounting (CRD) method. Compared with the heuristic search method, the proposed CRD method can achieve a much better trade-off between the computational complexity of searching and the convergence performance. Extensive experiments indicate that our UDP algorithm using the proposed CRD method can effectively improve both the training efficiency and model quality for the given privacy protection levels.

Local Differential Privacy (LDP) provides a formal guarantee of privacy that enables the collection and analysis of sensitive data without revealing any individual's data. While LDP methods have been extensively studied, there is a lack of a systematic and empirical comparison of LDP methods for descriptive statistics. In this paper, we first provide a systematization of LDP methods for descriptive statistics, comparing their properties and requirements. We demonstrate that several mean estimation methods based on sampling from a Bernoulli distribution are equivalent in the one-dimensional case and introduce methods for variance estimation. We then empirically compare methods for mean, variance, and frequency estimation. Finally, we provide recommendations for the use of LDP methods for descriptive statistics and discuss their limitations and open questions.

We propose a novel framework for the differentially private ERM, input perturbation. Existing differentially private ERM implicitly assumed that the data contributors submit their private data to a database expecting that the database invokes a differentially private mechanism for publication of the learned model. In input perturbation, each data contributor independently randomizes her/his data by itself and submits the perturbed data to the database. We show that the input perturbation framework theoretically guarantees that the model learned with the randomized data eventually satisfies differential privacy with the prescribed privacy parameters. At the same time, input perturbation guarantees that local differential privacy is guaranteed to the server. We also show that the excess risk bound of the model learned with input perturbation is $O(1/n)$ under a certain condition, where $n$ is the sample size. This is the same as the excess risk bound of the state-of-the-art.

Local differential privacy (LDP) is a promising privacy model for data collection that protects sensitive information of individuals. However, applying LDP to top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> estimation in set-valued data (e.g., identifying most frequent <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> items) may yield poor results for small and sparse datasets due to high sensitivity and heavy perturbation. To address this, we propose an adaptive approach that frames the problem as a multi-armed bandit (MAB) problem, in which the decision-maker selects actions based on information collected from previous rounds to maximize the total reward over time. Inspired by this, we present two adaptive sampling schemes based on MAB: <b>ARBS</b> for identifying top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> items and <b>ARBSF</b> for both top-<inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> item discovery and frequency estimation on these items. Furthermore, to address the potential long delay of multi-round collection, we propose an optimization technique to reduce the time complexity. Both theoretical and empirical results show that our adaptive sampling schemes significantly outperform existing alternatives.

Local Differential Privacy protocols are stochastic protocols used in data aggregation when individual users do not trust the data aggregator with their private data. In such protocols there is a fundamental tradeoff between user privacy and aggregator utility. In the setting of frequency estimation, established bounds on this tradeoff are either nonquantitative, or far from what is known to be attainable. In this paper, we use information-theoretical methods to significantly improve established bounds. We also show that the new bounds are attainable for binary inputs. Furthermore, our methods lead to improved frequency estimators, which we experimentally show to outperform state-of-the-art methods.

Privacy concerns hinder the sharing and utilization of trajectory data collected from Internet of Things (IoT) devices. While local differential privacy mitigates leakage risks by perturbing data on mobile devices, existing methods primarily focus on enhancing global-level statistical usability of perturbed trajectories (such as spatial distributions and mobility transition patterns), neglecting individual-level semantic usability (such as road network consistency and traffic rule compliance). The resulting trajectories frequently contain unrealistic behaviors that violate traffic rules, severely limiting their applicability. To overcome this challenge, we propose TPIS, a trajectory protection approach with individual semantic utility under local differential privacy. First, we design a semantic-enhanced hierarchical modeling method (SEHM) that leverages traffic rule semantic information and geospatial information to model the road network as a weighted graph, supported by an R-tree index for efficient coarse-grained and fine-grained trajectory matching. Second, we propose a cascaded perturbation method combining stochastic sampling guided by the hierarchical graph model (SHG) and multi-feature fusion perturbation mechanism based on δ-location set (MFPD). SHG strategically constrains the perturbation space to preserve global-level statistical usability while ensuring rigorous privacy, and MFPD incorporates a comprehensive trajectory difference metric and δ -location set definition to generate perturbed trajectories that maintain individual-level structural integrity of real-world movement patterns. TPIS efficiently balances privacy and utility and ensures high computational efficiency. Theoretical analysis confirms its differential privacy guarantee and upper bound on perturbation error. Experiments conducted on four real-world datasets demonstrate significant improvements over existing methods, with up to 8× higher utility and 14× greater computational efficiency.