Research and Model Library Construction in Teacher-Student Learning Architectures for Knowledge Transfer

Multiagent Reinforcement Learning (MARL) has been well adopted due to its exceptional ability to solve multiagent decision-making problems. To further enhance learning efficiency, knowledge transfer algorithms have been developed, among which experience-sharing-based and action-advising-based transfer strategies share the mainstream. However, it is notable that, although there exist many successful applications of both strategies, they are not flawless. For the long-developed action-advising-based methods (namely KT-AA, short for knowledge transfer based on action advising), their data efficiency and scalability are not satisfactory. As for the newly proposed experience-sharing-based knowledge transfer methods (KT-ES), although the shortcomings of KT-AA have been partially overcome, they are incompetent to correct specific bad decisions in the later learning stage. To leverage the superiority of both KT-AA and KT-ES, this study proposes KT-Hybrid, a hybrid knowledge transfer approach. In the early learning phase, KT-ES methods are employed, expecting better data efficiency from KT-ES to enhance the policy to a basic level as soon as possible. Later, we focus on correcting specific errors made by the basic policy, trying to use KT-AA methods to further improve the performance. Simulations demonstrate that the proposed KT-Hybrid outperforms well-received action-advising- and experience-sharing-based methods.

We present Self-Guidance, a simple way to train deep neural networks via knowledge distillation. The basic idea is to train sub-network to match the prediction of the full network, so-called "Self-Guidance". Under the "teacher-student" framework, we construct both teacher and student within the same target network. Student network is the sub-networks that randomly skip some portions of the full network. The teacher network is the full network, can be considered as the ensemble of all possible student networks. The training process is performed in a closed-loop: (1) Forward prediction contains two passes that generate student and teacher predictions. (2) Backward distillation allows knowledge transfer from the teacher back to students. Comprehensive evaluations show that our approach improves the generalization ability of deep neural networks to a significant margin. The results prove our superior performance in both image classification on CIFAR10, CIFAR100, and facial expression recognition on FER-2013 and RAF.

PurposeThis study reviews strategies organizational leaders and human resource practitioners can deploy to develop knowledge transfer and retention succession planning strategies for older employees to mitigate generational organizational knowledge loss prior to retirement.Design/methodology/approachThis study used a questionnaire with 28 baby boomer employees and leaders of baby boomers at a large federal agency. Purposive and snowball sampling techniques were used. Questions asked participants about knowledge transfer, retention strategies and how challenges to counter baby boomer knowledge loss are addressed in the workplace.FindingsDevelop succession plans using standard operating procedures and job aids to reduce knowledge loss and enhance retention. Deploy hands-on training to share historical knowledge, enhance relationship building, mentoring programs, cross-training opportunities, retention incentives and document process improvement. The strategies are supported by organizational learning and knowledge management theories.Originality/valueThis study contributes to organizational leaders’ and human resource practitioners’ knowledge transfer and retention succession planning strategies to counter generational knowledge loss.

Knowledge transfer has become very popular in recent years, and it is either based on a one-way transfer method used with knowledge distillation or based on a two-way knowledge transfer implemented by deep mutual learning, while both of them adopt a teacher–student paradigm. A one-way based method is more simple and compact because it only involves an untrained low-capacity student and a high-capacity teacher network in the knowledge transfer process. In contrast, a two-way based method requires more training costs because it involves two or more low-cost network capacities from scratch simultaneously to obtain better accuracy results for each network. In this paper, we propose two new approaches, namely full deep distillation mutual learning (FDDML) and half deep distillation mutual learning (HDDML), and improve convolutional neural network performance. These approaches work with three losses by using variations of existing network architectures, and the experiments have been conducted on three public benchmark datasets. We test our method on some existing KT task methods, showing its performance over related methods.

The well-being of residents is a top priority for megacities, which is why urban design and sustainable development are crucial topics. Quality of Life (QoL) is used as an effective key performance index (KPI) to measure the efficiency of a city plan’s quantity and quality factors. For city dwellers, QoL for pedestrians is also significant. The walkability concept evaluates and analyzes the QoL in a walking scene. However, the traditional questionnaire survey approach is costly, time-consuming, and limited in its evaluation area. To overcome these limitations, the paper proposes using artificial intelligence (AI) technology to evaluate walkability data collected through a questionnaire survey using virtual reality (VR) tools. The proposed method involves knowledge extraction using deep convolutional neural networks (DCNNs) for information extraction and deep learning (DL) models to infer QoL scores. Knowledge distillation (KD) is also applied to reduce the model size and improve real-time performance. The experiment results demonstrate that the proposed approach is practical and can be considered an alternative method for acquiring QoL.

Surrogate-assisted evolutionary algorithms provide an effective means for complex and computationally expensive optimization problems. However, due to the scarcity of training samples, the prediction accuracy of frequently-used regression surrogate models can hardly be guaranteed as the difficulty of the problem increases, resulting in performance degradation of the whole algorithm. Since real-world problems rarely exist in isolation, it is expected to alleviate the above issue by properly exploiting the knowledge shared across different problems. In this context, this study proposes a novel evolutionary multitasking optimization algorithm assisted by a classifier rather than a regression model for expensive multitasking problems, where the accuracy of the classifier is boosted by knowledge transfer. To be specific, a support vector classifier (SVC) is first developed and integrated into a classic evolutionary algorithm, i.e., covariance matrix adaptation evolution strategy (CMA-ES). With a low computational cost, it helps CMA-ES to prescreen parent solutions from the current population. Following that, a knowledge transfer strategy is designed to enrich the training samples for each task-oriented classifier by sharing high-quality solutions among different tasks, where a PCA-based subspace alignment technique is employed. Extensive experiments indicate that the SVC-assisted CMA-ES gains significant superiority over general CMA-ES in terms of both robustness and scalability, and the knowledge transfer strategy further helps it earn a competitive edge over some state-of-the-art algorithms on expensive multitasking optimization problems.

In software development, requirements changes cannot be avoided. Despite the importance of requirements change management, few studies in the literature concern requirements change knowledge management(RCKM). This study explored knowledge transfer and sharing of RCKM based on change cause analysis and the design of a knowledge management system(KMS) for change management. In this paper, previous research is discussed in three aspects: factors affecting knowledge transfer and sharing, methods to implement knowledge transfer and sharing, and the design of a KMS. After that, the results of a recent questionnaire are analyzed. Finally, this paper discusses the sources of tacit knowledge in the personnel management aspects of software development; the knowledge and the methods of knowledge transfer and sharing in the requirements change management process based on change cause analysis; the design of a data flowchart of repositories, and a prototype structure, logical model, and functional modules of RCKM systems. In conclusion, further study is suggested.

Recently, video analysis technology through deep learning has been developing at a very rapid pace, and most of the technology related to improving recognition performance in server environment is being developed. However, in addition to video analysis technology in the existing server environment, the demand of object detection in visual image analysis have been increasing recently in embedded boards of low specification and mobile environments such as smartphones, drones, and industrial boards. Despite the significant improvement in the accuracy of existing object detectors, image processing for real- time applications often requires a lot of runtime. Therefore, many studies are being conducted on lightweight object detection technology, and knowledge distillation is one of the solutions. Efforts such as model compression use fewer parameters, but there is a problem that accuracy is significantly reduced. In this paper, we propose method to improve the performance of lightweight mobilenet-SSD models in object detection by using knowledge transfer methods. We conduct evaluation with PASCAL VOC dataset. Our results show detection accuracy improvement in object detection.

This article investigates the impact of knowledge transfer (Goh 2002) from founder firms to the corporate foundations (CFs) on the CFs’ effectiveness. Starting from a typology of CFs’ effectiveness (Ostrower 2006a), we conducted a survey addressed to a sample of Italian CFs to address the impact of different knowledge transfer methods (KTMs) on three dimensions of CFs’ orientation to effectiveness: proactive orientation, social advocacy, and capacity building. The research identified four different KTMs and, using a linear regression, pointed out that the methods adopted by founder firms have a significant influence on proactivity, competences, and on social advocacy of CFs.

Need for a strategic approach to knowledge transfer and exchangeLate-phase clinical trials and systematic reviews find results that have the potential to improve health outcomes for people. However, there are often delays in these results influencing clinical practice. We developed a knowledge transfer and exchange strategy to support research teams, aiming to identify activities along the research process to maximise and accelerate the research impact.Our knowledge transfer and exchange strategyThe strategy has five strands of activity across the life-course of our research:1. Partnerships with external stakeholders (including patient and public involvement, charities, policymakers, healthcare professionals, professional bodies, regulators and industry)2. Communication (including the development of research impact strategies and use of communication tools and channels)3. Maximising the scientific value of our studies (including open access, data and sample sharing, and incorporating multi-disciplinary components within studies)4. Strengthening capacity (including building internal and partner capacity to communicate effectively, and strengthening the capacity of external stakeholders to understand and apply our research).5. Learning and sharing (evaluating the impact of research, sharing lessons learnt internally and externally)The strategy has helped trial teams think systematically about impact and was easy to use.ConclusionsOur strategy helps researchers systematically identify activities which may improve the usefulness and uptake of their study results. While developed in a single trials unit, we think it may be of use to others.

This study employed a qualitative methodology to explore small family farm owners' strategiesto transfer knowledge and preserve critical expertise, skills, and organizational insights. Given the challenges inherent in small family farms, including the lack of standardized processes for knowledge transfer, qualitative research allows for an in-depth exploration of these complexities. The study aimed to investigate knowledge transfer strategies used by retired small family farm owners in the northeastern U.S. A total of 13 were interviewed using a semi-structured format, focusing on strategies employed to facilitate knowledge retention. Data was analyzed through coding and theme identification to provide comprehensive insights into the barriers, facilitators, and situational factors affecting knowledge transfer. Findings contribute valuable knowledge to the existing literature on knowledge transfer and offer practical recommendations for improving knowledge retention in small family farms.

Sequential recommendation aims to capture users' dynamic interest and predicts the next item of users' preference. Most sequential recommendation methods use a deep neural network as sequence encoder to generate user and item representations. Existing works mainly center upon designing a stronger sequence encoder. However, few attempts have been made with training an ensemble of networks as sequence encoders, which is more powerful than a single network because an ensemble of parallel networks can yield diverse prediction results and hence better accuracy. In this paper, we present Ensemble Modeling with contrastive Knowledge Distillation for sequential recommendation (EMKD). Our framework adopts multiple parallel networks as an ensemble of sequence encoders and recommends items based on the output distributions of all these networks. To facilitate knowledge transfer between parallel networks, we propose a novel contrastive knowledge distillation approach, which performs knowledge transfer from the representation level via Intra-network Contrastive Learning (ICL) and Cross-network Contrastive Learning (CCL), as well as Knowledge Distillation (KD) from the logits level via minimizing the Kullback-Leibler divergence between the output distributions of the teacher network and the student network. To leverage contextual information, we train the primary masked item prediction task alongside the auxiliary attribute prediction task as a multi-task learning scheme. Extensive experiments on public benchmark datasets show that EMKD achieves a significant improvement compared with the state-of-the-art methods. Besides, we demonstrate that our ensemble method is a generalized approach that can also improve the performance of other sequential recommenders. Our code is available at this link: https://github.com/hw-du/EMKD.

In recent years, deep neural network has achieved high accuracy in the field of image recognition, but the consumption of resource is high. Inspired by collaborative learning approaches and the knowledge distillation technique, this study proposes a knowledge transfer method for semantic segmentation based on feature aggregation. In this research, the original student network is applied to generate the auxiliary teacher network, and share the information learned from the network by establishing dense feature connections between the two networks, which are trained simultaneously. Any one of these networks has access to information that is not available to the individual networks. In addition, in order to increase the degree of collaboration, this paper proposes two methods for establishing connections between the teacher network and the student network. The first method is a dense feature connection between networks of the same layer, and the second method is a dense feature connections between multi-layer networks. The approaches proposed above are validated on the train of Unet, and the experimental results show that the knowledge transfer approach of shared feature aggregation has better performance than the traditional single network.

In recent years, deep learning revolutionized the field of machine learning. While many applications of deep learning are observed in computer vision, other domains like natural language processing (NLP) or speech recognition also benefited from advances in deep learning research. More recently, the field of time series analysis and more especially time series classification (TSC) also witnessed the emergence of deep neural networks providing competitive results. Through the years, the proposed network architectures became deeper and deeper pushing the performance higher. While these very deep models achieve impressive accuracy, their training and deployment became challenging. Indeed, a large number of GPUs is often required to train state-of-the-art networks and obtain high performances. While the requirements needed for the training step can be acceptable, deploying very deep neural networks can be difficult especially in embedded systems (e.g. robots) or devices with limited resources (e.g. web browsers, smartphones). In this context, knowledge distillation is a machine learning task consisting in transferring knowledge from a large model to a smaller one with fewer parameters. The goal is to create a lighter model mimicking the predictions of a larger one in order to obtain similar performances with a fraction of the computational cost. In this paper, we introduce and explore the concept of knowledge distillation for the specific task of TSC. We also present a first experimental study showing promising results on several datasets of the UCR time series archive. As current state-of-the-art models for TSC are deep and sometimes ensemble of models, we believe that knowledge distillation could become an important research area in the coming years.

PurposeRecent work on public private partnerships (PPP)/private finance initiatives (PFI) and on the construction industry has emphasised the importance of knowledge transfer, learning and innovation to improve the working processes and procedures of the industry. However, until recently all three have been researched separately. It is on this basis the research presents a theoretical proposition that “the integration of alliance principles through the formation of collaborative network; innovation, knowledge transfer and organisational learning can be enabled where there is coordination, cohesion, commitment, teamwork, clear vision, monitoring and governance structure for control, risks/reward sharing, efficient and effective communication strategy, learning and knowledge transfer strategy”.Design/methodology/approachThe theory of social capital, social networks and inter‐organisational network was applied based on a network perspective to explain and aid specific attributes of PPP and PFI as a collaborative network to achieve the research objective. A case study research methodology was adopted to investigate the theoretical proposition.FindingsThe findings of this research illustrates that collaborative relationship is a crucial factor in contractual arrangements to enable cost savings, time savings and project quality. In addition, the challenges clients and contractors encounter in integrating innovation are as a result of project scope, project duration, affordability and project procurement strategy. Furthermore, the difficulties of knowledge transfer and learning is as a result of the disconnection between the team that closes the PFI deal and the team that implements the contract.Originality/valueThe paper suggests a network perspective to improve work relationships that helps to achieve predetermined common goals and to enhance project performance.