PiPar: Pipeline parallelism for collaborative machine learning

This paper explores the problem of data collaboration in health care, which is the one of the critical infrastructure sectors designated by the Department of Home Security. Limitations to data sharing in health care obstruct the development of a new generation of medical technology powered by artificial intelligence (AI). Collaborative machine learning helps to overcome these limitations through training models on distributed data sets without data sharing. Among other approaches to collaborative machine learning, federated learning in recent years has demonstrated multiple advantages. However, it had been developed and tested in a highly distributed data environment, which is different from the typical cases of health care data collaboration. The objective of this paper is to validate the known advantages of federated learning and to assess possible risks in a small multiparty setting. The experiments show that federated learning can be successfully applied in a multiparty collaboration setting. However, with a small number of parties, it becomes easier to overfit to each local data so that the averaging steps have to occur more frequently. In addition, for the first time, the risks of a membership inference attack were assessed for different methods of collaborative machine learning.

Collaborative machine learning is a promising paradigm that allows multiple participants to jointly train a machine learning model without exposing their private datasets to other parties. Although collaborative machine learning is more privacy-friendly compared with conventional

Collaborative machine learning (CML) provides a promising paradigm for democratizing advanced technologies by enabling cost-sharing among participants. However, the potential for rent-seeking behaviors among parties can undermine such collaborations. Contract theory presents a viable solution by rewarding participants with models of varying accuracy based on their contributions. However, unlike monetary compensation, using models as rewards introduces unique challenges, particularly due to the stochastic nature of these rewards when contribution costs are privately held information. This paper formalizes the optimal contracting problem within CML and proposes a transformation that simplifies the non-convex optimization problem into one that can be solved through convex optimization algorithms. We conduct a detailed analysis of the properties that an optimal contract must satisfy when models serve as the rewards, and we explore the potential benefits and welfare implications of these contract-driven CML schemes through numerical experiments.

In this study, we aim to bridge the gap between the theoretical understanding of attacks against collaborative machine learning workflows and their practical ramifications by considering the effects of model architecture, learning setting and hyperparameters on the resilience against attacks. We refer to such mitigations asmodel adaptation. Through extensive experimentation on both, benchmark and real-life datasets, we establish a more practical threat model for collaborative learning scenarios. In particular, we evaluate the impact of model adaptation by implementing a range of attacks belonging to the broader categories of model inversion and membership inference. Our experiments yield two noteworthy outcomes: they demonstrate the difficulty of actually conducting successful attacks under realistic settings when model adaptation is employed and they highlight the challenge inherent in successfully combining model adaptation and formal privacy-preserving techniques to retain the optimal balance between model utility and attack resilience.

Data analytics frameworks shift towards larger degrees of parallelism. Efficient scheduling of data-parallel jobs (tasks) is critical for improving job performance such as response time, and resource utilization. It is an important challenge for large scale data analytics frameworks in which jobs are more complex and have diverse characteristics (e.g., diverse resource requirements). Prior work on scheduling cannot achieve low response time and high resource utilization simultaneously because they cannot accurately estimate the durations of tasks in the queue of a worker machine by using sampling-based approach (including sampling with late binding) for task placement, and thus they fail to place tasks at the best possible worker machine. Also, they do not sufficiently consider the diverse resource requirements of jobs (tasks) for placing tasks on worker machines. To address this challenge, we propose a Dependency-aware and Resource-efficient Scheduling (DRS) to achieve low response time and high resource utilization. DRS takes into account task dependency and assigns tasks that are independent of each other to different worker machines. Also, DRS considers tasks' resource requirements and packs complementary tasks whose resource demands on multiple resources are complementary to each other to increase the resource utilization. In addition, DRS uses the mutual reinforcement learning to estimate the task's waiting time (the duration of tasks in the queue of a worker), and assigns tasks to workers with the consideration of tasks' waiting time to reduce the response time. Extensive experimental results based on a real cluster and experiments using real-world Amazon EC2 cloud service show that DRS achieves low response time and high resource utilization compared to previous strategies.

In recent years, the enhanced sensing and computation capabilities of Internet-of-Things (IoT) devices have opened the doors to several mobile crowdsensing applications. In mobile crowdsensing, a model owner announces a sensing task following which interested workers collect the required data. However, in some cases, a model owner may have insufficient data samples to build an effective machine learning model. To this end, we propose a federated learning (FL)-based privacy-preserving approach to facilitate collaborative machine learning among multiple model owners in mobile crowdsensing. Our system model allows collaborative machine learning without compromising data privacy given that only the model parameters instead of the raw data are exchanged within the federation. However, there are two main challenges of incentive mismatches between workers and model owners, as well as among model owners. For the former, we leverage on the self-revealing mechanism in the contract theory under information asymmetry. For the latter, to ensure the stability of a federation through preventing free-riding attacks, we use the coalitional game theory approach that rewards model owners based on their marginal contributions. Considering the inherent hierarchical structure of the involved entities, we propose a hierarchical incentive mechanism framework. Using the backward induction, we first solve the contract formulation and then proceed to solve the coalitional game with the merge and split algorithm. The numerical results validate the performance efficiency of our proposed hierarchical incentive mechanism design, in terms of incentive compatibility of our contract design and fair payoffs of model owners in stable federation formation.

Federated Learning (FL) is a transformative paradigm that enables collaborative machine learning across multiple decentralized data sources without requiring direct data sharing. This approach has emerged as a critical solution for privacy-preserving machine learning, particularly in sensitive domains such as healthcare, finance, and IoT systems. FL frameworks employ advanced techniques such as secure aggregation, differential privacy, and homomorphic encryption to protect user data while facilitating model convergence. These frameworks address key challenges, including communication efficiency, data heterogeneity, and robustness against malicious actors. This paper reviews the state-of-the-art FL frameworks, exploring their architecture, privacy-preserving techniques, and application scenarios. Furthermore, it highlights recent advancements in secure federated learning protocols, multi-party computation strategies, and scalability improvements. The discussion extends to open challenges, such as ensuring fairness, optimizing resource consumption, and maintaining security guarantees in adversarial settings. By advancing the development of privacy-preserving FL frameworks, researchers and practitioners can unlock the potential of collaborative machine learning while upholding stringent data privacy standards.

BackgroundEffective triage systems are crucial for prioritizing patients based on urgency and optimizing resource utilization. An ideal triage system is expected to have low resource utilization, hospitalization and mortality among patients classified at low urgency levels. Furthermore, it should exhibit an increase in the risk of hospitalization and mortality as urgency levels increase, ensuring the most critically ill patients receive priority care first. However, it is unclear which triage system performs best.ObjectiveTo compare the performance of the Manchester Triage System (MTS), the Emergency Severity Index (ESI), and the Netherlands Triage Standard (NTS) by investigating the association between urgency levels and resource utilization, hospitalization and in-hospital mortality in Emergency Department (ED) patients.MethodsObservational multicenter cohort study using data from the Netherlands Emergency department Evaluation Database, comprising seven representative EDs in six Dutch hospitals. All consecutive ED patients with a registered urgency level were included. Resource utilization, hospitalization and mortality were measured across all urgency levels. In each triage system, multivariable logistic regression was used to assess the association between urgency level and in-hospital mortality and hospitalization, adjusting for age, sex, presenting complaints and hospital type.ResultsA total of 696,518 ED visits (MTS 320,406 (46.1%), ESI 214,267 (30.8%), NTS 161,845 (23.3%) patients) were included. Resource utilization was substantially lower in the lowest urgency level of the ESI compared to the MTS and NTS. Hospitalization to a regular ward, cardiac, medium or intensive care unit in the least urgent level was 3.9% in the ESI, considerably lower than in the MTS (23.1%) and NTS (34.3%) (P < 0.05). Mortality in the lowest urgency level of the ESI was 0.8%, while in the MTS and NTS this was 6.3% and 12.4%, respectively (P < 0.05). In the ESI, the risk (Adjusted Odds Ratios) for hospitalization and mortality increased much more with increasing urgency levels compared to the MTS and NTS.ConclusionThis study suggests that the ESI may be more effective in distinguishing between patients with low and high urgency, with a reduced risk of undertriage when compared to the MTS and NTS.

This paper discusses the proposed model of the collaborative virtual learning system for the introductory computer programming course which uses one of the collaborative learning techniques known as the “Think-Pair-Share”. The main objective of this study is to design a model for an online learning system that facilitates the collaborative learning activities in a virtual environment such as online communications and pair or small group discussions. In order to model the virtual learning environment, the RUP methodology has been used where it involves the data collection phase and the analysis and design phase. Fifty respondents have been randomly selected to participate in the data collection phase to investigate the students’ interest and learning styles as well as their learning preferences. The results have shown the needs for the development of online small group discussions that can be used as an alternative learning style for programming courses. The proposed design of the virtual learning system named as the Online Collaborative Learning System or OCLS is being depicted using the objectoriented models which are the use-case model and class diagram in order to show the concise processes of virtual “Think-Pair-Share” collaborative activities. The “Think-PairShare” collaborative learning technique that is being used in this model has been chosen because of its simplicity and relatively low-risk. This paper also presents the proposed model of the system’s architecture that will become the guidelines for the physical development of OCLS using the web-based applications.

Federated Learning (FL) is a collaborative machine learning (ML) framework that combines on-device training and server-based aggregation to train a common ML model among distributed agents. In this work, we propose an asynchronous FL design with periodic aggregation to tackle the straggler issue in FL systems. Considering limited wireless communication resources, we investigate the effect of different scheduling policies and aggregation designs on the convergence performance. Driven by the importance of reducing the bias and variance of the aggregated model updates, we propose a scheduling policy that jointly considers the channel quality and training data representation of user devices. The effectiveness of our channel-aware data-importance-based scheduling policy, compared with state-of-the-art methods proposed for synchronous FL, is validated through simulations. Moreover, we show that an ``age-aware'' aggregation weighting design can significantly improve the learning performance in an asynchronous FL setting.

Efficient collaboration between collaborative machine learning and wireless communication technology, forming a Federated Edge Learning (FEEL), has spawned a series of next-generation intelligent applications. However, due to the openness of network connections, the FEEL framework generally involves hundreds of remote devices (or clients), resulting in expensive communication costs, which is not friendly to resource-constrained FEEL. To address this issue, we propose a distributed approximate Newton-type algorithm with fast convergence speed to alleviate the problem of FEEL resource (in terms of communication resources) constraints. Specifically, the proposed algorithm is improved based on distributed L-BFGS algorithm and allows each client to approximate the high-cost Hessian matrix by computing the low-cost Fisher matrix in a distributed manner to find a “better” descent direction, thereby speeding up convergence. Second, we prove that the proposed algorithm has linear convergence in strongly convex and non-convex cases and analyze its computational and communication complexity. Similarly, due to the heterogeneity of the connected remote devices, FEEL faces the challenge of heterogeneous data and non-IID (Independent and Identically Distributed) data. To this end, we design a simple but elegant training scheme, namely FedOVA (Federated One-vs-All), to solve the heterogeneous statistical challenge brought by heterogeneous data. In this way, FedOVA first decomposes a multi-class classification problem into more straightforward binary classification problems and then combines their respective outputs using ensemble learning. In particular, the scheme can be well integrated with our communication efficient algorithm to serve FEEL. Numerical results verify the effectiveness and superiority of the proposed algorithm.

Growing concerns about centralized mining of personal data threatens to stifle further proliferation of machine learning (ML) applications. Consequently, a recent trend in ML training advocates for a paradigm shift – moving the computation of ML models from a centralized server to a federation of edge devices owned by the users whose data is to be mined. Though such decentralization aims to alleviate concerns related to raw data sharing, it introduces a set of challenges due to the hardware heterogeneity among the devices possessing the data. The heterogeneity may, in the most extreme cases, impede the participation of low-end devices in the training or even prevent the deployment of the ML model to such devices. Recent research in distributed collaborative machine learning (DCML) promises to address the issue of ML model training over heterogeneous devices. However, the actual extent to which the issue is solved remains unclear, especially as an independent investigation of the proposed methods’ performance in realistic settings is missing. In this paper, we present a detailed survey and an evaluation of algorithms that aim to enable collaborative model training across diverse devices. We explore approaches that harness three major strategies for DCML, namely Knowledge Distillation, Split Learning, and Partial Training, and we conduct a thorough experimental evaluation of these approaches on a real-world testbed of 14 heterogeneous devices. Our analysis compares algorithms based on the resulting model accuracy, memory consumption, CPU utilization, network activity, and other relevant metrics, and provides guidelines for practitioners as well as pointers for future research in DCML.

Resource Sharing in Mobile Cloud-computing with Coap

In order to solve problems such as low resources utilization of data center backbone bandwidth and expensive daily expenses caused by the tidal effect in the transition of large-scale network data, the new optimization method of bandwidth resource for bulk data transfer is proposed. This method finishes the bulk data transfer by taking full advantage of the idle channel bandwidth. Firstly, the dynamic idle network bandwidth resource was switched into the static flow network; and then the max-min fair multi-commodity flow model is built on the basis of static network; finally, the iterative technique is used to solve the programming model for getting the maximum transport flows and the corresponding transmission path of bulk data transfer. The simulation method was used to test this optimization project. The results show that the optimization method proposed by the paper can significantly improve the utilization of data center backbone bandwidth resource, and the maximum amount of data transmission and the speed of data transfer have been obviously increased

SummaryIn recent years, Docker container technology is being applied in the field of cloud computing at an explosive speed. The scheduling of Docker container resources has gradually become a research hotspot. Existing big data computing and storage platforms apply with traditional virtual machine technology, which often results in low resource utilization, a long time for flexible scaling and expanding clusters. In this paper, we propose an improved container scheduling algorithm for big data applications named Kubernetes‐based particle swarm optimization(K‐PSO). Experimental results show that the proposed K‐PSO algorithm converges faster than the basic PSO algorithm, and the running time of the algorithm is cut in about half. The K‐PSO container scheduling algorithm and algorithm experiment for big data applications are implemented in the Kubernetes container cloud system. Our experimental results show that the node resource utilization rate of the improved scheduling strategy based on K‐PSO algorithm is about 20% higher than that of the Kube‐scheduler default strategy, balanced QoS priority strategy, ESS strategy, and PSO strategy, while the average I/O performance and average computing performance of Hadoop cluster are not degraded.