A Review of Text Style Transfer Using Deep Learning

Existing text style transfer (TST) methods rely on style classifiers to disentangle the text’s content and style attributes for text style transfer. While the style classifier plays a critical role in existing TST methods, there is no known investigation on its effect on the TST methods. In this paper, we conduct an empirical study on the limitations of the style classifiers used in existing TST methods. We demonstrated that the existing style classifiers cannot learn sentence syntax effectively and ultimately worsen existing TST models’ performance. To address this issue, we propose a novel Syntax-Aware Controllable Generation (SACG) model, which includes a syntax-aware style classifier that ensures learned style latent representations effectively capture the sentence structure for TST. Through extensive experiments on two popular text style transfer tasks, we show that our proposed method significantly outperforms twelve state-of-the-art methods. Our case studies have also demonstrated SACG’s ability to generate fluent target-style sentences that preserved the original content.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

With the development of deep neural network, especially the emergence of large scale pre-trained language model, methods based on deep learning have substantially advanced the state of the art across a variety of subtasks in natural language processing. Nevertheless, in order to get better performance, the increasing model parameters and exponential growth of training time become problems that can not be ignored. In this paper, inspired by the application of deep hashing in large-scale image retrieval, text representation learning based on deep hashing technology has been extensively explored. And we use three common subtasks in natural language processing to evaluate the proposed method. Experimental results show that deep hashing can greatly reduce the physical space cost by text representation under the condition of limited performance loss, which is of great significance in large-scale text representation, and is worthy of further exploration.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Abstract. Large language models (LLMs) have revolutionized the field of natural language processing (NLP), demonstrating remarkable capabilities in understanding, generating, and manipulating human language. This comprehensive review explores the development, applications, optimizations, and challenges of LLMs. This paper begin by tracing the evolution of these models and their foundational architectures, such as the Transformer, GPT, and BERT. We then delve into the applications of LLMs in natural language understanding tasks, including sentiment analysis, named entity recognition, question answering, and text summarization, highlighting real-world use cases. Next, we examine the role of LLMs in natural language generation, covering areas such as content creation, language translation, personalized recommendations, and automated responses. We further discuss LLM applications in other NLP tasks like text style transfer, text correction, and language model pre-training. Subsequently, we explore techniques for optimizing and improving LLMs, including model compression, explainability, robustness, and security. Finally, we address the challenges posed by the significant computational requirements, sample inefficiency, and ethical considerations surrounding LLMs. We conclude by discussing potential future research directions, such as efficient architectures, few-shot learning, bias mitigation, and privacy-preserving techniques, which will shape the ongoing development and responsible deployment of LLMs in NLP.

Recently, deep neural networks (DNNs) have been widely studied and achieved tremendous success in a variety of real-world applications, such as computer vision, natural language processing, medical diagnosis, and machine translation. Deep reinforcement learning (DRL), as an emerging powerful deep learning technique, combines DNNs with reinforcement learning into an interactive system that enables the agent to learn the best policy from the environment to optimize their rewards. DRL opens up many new applications in domains such as healthcare, robotics, and smart grids. With the rapid evolution of IT infrastructures, cloud computing has been witnessed as the most common computing paradigm. The underlying infrastructure of cloud computing relies on a large amount of data centers. Nowadays, with the fast development of communication technologies, such as 5G, cloud computing is now evolving into yet another new paradigm called edge computing, providing for emerging applications, especially in the Internet-of-Things. The energy efficiency issue from both "cloud" and "edge" are becoming more crucial and calls for more attention. To solve the real-world energy efficiency problems, we take advantage of the deep learning and deep reinforcement learning techniques for efficient modeling of both "edge" and "cloud" applications. On the "edge" side, wearable technologies and the Internet of Things (IoT) are becoming increasingly popular in both personal and commercial applications with the employment of the embedded system technologies. Non-Volatile Processors (NVPs) have been proposed for the intermittent power supply problem, to ensure the instant on/off for embedded systems under the unstable power supply. We proposed neural network-based prediction techniques to derive the best power extraction policy and converter parameters optimization technique for NVPs. In cloud computing, we formulated a novel optimal power management problem of data centers to minimize the total cost by proposing the long short term memory (LSTM) neural network-based method for the prediction of future data center power consumption, and further we presented a novel DRL-based hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. Furthermore, we explored the effectiveness of the DRL-based framework for high dimensional action and state spaces problem-dynamic treatment regimes, to model real-life complexity in heterogeneous disease progression and treatment choices, to provide doctors and patients the data-driven personalized decision recommendations. Last but not least, we explored the efficiency of deep neural networks. Structured weight pruning is a representative model compression technique of DNNs to reduce both the storage and computation requirements and meanwhile accelerate inference. An automatic hyperparameter determination process is necessary due to a large number of flexible hyperparameters. We propose an automatic structured pruning framework to compress the DNNs. Our framework outperforms the prior works on automatic model compression by up to 33× in pruning rate (120× reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the proposed framework on actual measurements on the smartphone.

Review question/objective The objective of this systematic review is to provide a narrative summary synthesis of the quality of existing Husserlian phenomenological research studies in the health sciences. As a methodological systematic review, only studies that focus on adult patients’ experiences of preventive, screening, diagnosis, treatment or rehabilitation interventions/procedures will be considered to serve as a paradigm “case” of the use of the Husserlian phenomenological approach within the broad field of health. The review question is: What is the quality of existing Husserlian phenomenological research studies in the health sciences? The motivation for this systematic review is not to fault authors of existing Husserlian phenomenological research, but to explore the problems encountered in existing research literature, if any, and to provide guidance to authors for future health research guided by this specific philosophical perspective. In order to avoid repetition, the term ‘Husserlian phenomenological approach’ as used in the text refers to the approach described by Edmund Husserl in English translations of his own published work; or as described in English language commentaries by internationally recognized scholars; or from Husserlian phenomenological scholars including the structured framework presented by Spiegelberg. Each time the ‘content of Husserlian phenomenological research studies’ is used in the text it should be understood to mean ‘content of Husserlian phenomenological research studies as reflected in the stated philosophical perspective, research methodology, data collection methods, data analysis methods, presentation and interpretation of the results, and the conclusions’. Definition of terms: 1. Husserlian phenomenological research studies refers to primary qualitative research studies that are explicitly grounded in the work of Husserl reflected in explicit statements or descriptions by the authors. 2. Quality refers to the extent to which there is congruence between the Husserlian phenomenological approach and the content of Husserlian phenomenological research studies. These stipulative definitions are intended to preclude the need to cite the scientific literature as there is no need to demonstrate their ‘validity’ or ‘acceptability’ by the communities of scholars. TRUNCATED AT 350 WORDS.

Deep neural networks (DNNs) have achieved unprecedented success across many scientific and engineering fields in the last decades. Despite its empirical success, unfortunately, recent studies have shown that there are various failure modes and blindspots in DNN models which may result in unexpected serious failures and potential harms, e.g. the existence of adversarial examples and small perturbations. This is not acceptable especially for safety critical and high stakes applications in the real-world, including healthcare, self-driving cars, aircraft control systems, hiring and malware detection protocols. Moreover, it has been challenging to understand why and when DNNs will fail due to their complicated structures and black-box behaviors. Lacking interpretability is one critical issue that may seriously hinder the deployment of DNNs in high-stake applications, which need interpretability to trust the prediction, to understand potential failures, and to be able to mitigate harms and eliminate biases in the model. To make DNNs trustworthy and reliable for deployment, it is necessary and urgent to develop methods and tools that can (i) quantify and improve their robustness against adversarial and natural perturbations, and (ii) understand their underlying behaviors and further correct errors to prevent injuries and damages. These are the important first steps to enable Trustworthy AI and Trustworthy Machine Learning. In this talk, I will survey a series of research efforts in my lab contributed to tackling the grand challenges in (i) and (ii). In the first part of my talk, I will overview our research effort in Robust Machine Learning since 2017, where we have proposed the first attack-agnostic robustness evaluation metric, the first efficient robustness certification algorithms for various types of perturbations, and efficient robust learning algorithms across supervised learning to deep reinforcement learning. In the second part of my talk, I will survey a series of exciting results in my lab on accelerating interpretable machine learning and explainable AI. Specifically, I will show how we could bring interpretability into deep learning by leveraging recent advances in multi-modal models. I'll present recent works in our group on automatically dissecting neural networks with open vocabulary concepts, designing interpretable neural networks without concept labels, and briefly overview our recent efforts on demystifying black-box DNN training process, automated neuron explanations for Large Language Models and the first robustness evaluation of a family of neuron-level interpretation techniques.

In a previous paper, we introduced a deep learning neural network that should be able to detect the existence of very shallow periodic planetary transits in the presence of red noise. The network in that feasibility study would not provide any further details about the detected transits. The current paper completes this missing part. We present a neural network that tags samples that were obtained during transits. This is essentially similar to the task of identifying the semantic context of each pixel in an image—an important task in computer vision, called “semantic segmentation,” which is often performed by deep neural networks. The neural network we present makes use of novel deep learning concepts such as U-Nets, Generative Adversarial Networks, and adversarial loss. The resulting segmentation should allow further studies of the light curves that are tagged as containing transits. This approach toward the detection and study of very shallow transits is bound to play a significant role in future space-based transit surveys such as PLATO, which are specifically aimed to detect those extremely difficult cases of long-period shallow transits. Our segmentation network also adds to the growing toolbox of deep learning approaches that are being increasingly used in the study of exoplanets; but, so far mainly for vetting transits, rather than their initial detection.

Recently, in a massive multiple-input multipleoutput (MIMO) system, deep learning (DL)-based beamforming method has been proposed for reducing the overhead associated with downlink training and uplink feedback. However, the DL-based approach is sensitive to the variation of the communication environment and requires a huge number of training data to ensure a certain level of performance. To reduce the number of required channel data for training a deep neural network (DNN), we introduce deep transfer learning (DTL), which exploits the information from the pre-trained DNN for training other DNNs to find the beamforming vector in the specific channel. Through DTL, DNN can be trained suitably for the communication environment at each BS with fewer channel data. Moreover, we propose ‘step-by-step’ DTL to flexibly apply DTL considering the uncertainties of the realistic system. Simulation results show that DTL has better performance than the conventional DLapproaches even with a small amount number of channel data. Therefore, the DTL-based approach can be a good framework to train DNN when high overhead occurs or designing the beamformer is complicated such as a massive MIMO system.

Time Series Classification (TSC) is an important and challenging problem in data mining. With the increase of time series data availability, hundreds of TSC algorithms have been proposed. Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task. This is surprising as deep learning has seen very successful applications in the last years. DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks. Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition. In this article, we study the current state-of-the-art performance of deep learning algorithms for TSC by presenting an empirical study of the most recent DNN architectures for TSC. We give an overview of the most successful deep learning applications in various time series domains under a unified taxonomy of DNNs for TSC. We also provide an open source deep learning framework to the TSC community where we implemented each of the compared approaches and evaluated them on a univariate TSC benchmark (the UCR/UEA archive) and 12 multivariate time series datasets. By training 8,730 deep learning models on 97 time series datasets, we propose the most exhaustive study of DNNs for TSC to date.

In modern engineering, predicting the fatigue life of materials is crucial for safety assessment. The relationship between fatigue life and its influencing factors is difficult to predict by traditional methods, and deep learning can achieve great power and flexibility through nested hierarchies of concepts. Taking the low cycle fatigue life of 316 austenitic stainless steel as an example, a method for predicting the low cycle fatigue life of austenitic stainless steel by deep learning is established based on the limited ability of traditional neural network model and genetic algorithm optimization model. The deep neural network model is introduced to predict the fatigue life of the material. The results show that the prediction correlation coefficient R of the deep neural network prediction model with three hidden layers is 0.991, and the deep neural network learning model has better prediction ability.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Deep neural networks, inspired by information processing in the brain, can achieve human-like performance for various tasks. However, research efforts to use these networks as models of the brain have primarily focused on modeling healthy brain function so far. In this work, we propose a paradigm for modeling neural diseases in silico with deep learning and demonstrate its use in modeling posterior cortical atrophy (PCA), an atypical form of Alzheimer’s disease affecting the visual cortex. We simulated PCA in deep convolutional neural networks (DCNNs) trained for visual object recognition by randomly injuring connections between artificial neurons. Results showed that injured networks progressively lost their object recognition capability. Simulated PCA impacted learned representations hierarchically, as networks lost object-level representations before category-level representations. Incorporating this paradigm in computational neuroscience will be essential for developing in silico models of the brain and neurological diseases. The paradigm can be expanded to incorporate elements of neural plasticity and to other cognitive domains such as motor control, auditory cognition, language processing, and decision making.