Learning Parameter Analysis for Machine Reading Comprehension

Socio-economic indicators are important statistical data used to monitor and evaluate the development of economy and society. They are of great value to policy makers of government, organizations and enterprises. Previous methods to measure socio-economic conditions are through data collecting or large-scale surveys by manpower. Over recent years, with the quick development of Internet, a huge amount of online text becomes available and contains clues of socio-economic indicators. As a result, rule-based methods and traditional machine learning methods are emerging to extract indicators more effectively but they require too much prior domain knowledge and feature engineering. Recently, deep learning approaches are dominating in Natural Language Processing (NLP) and formulating NLP tasks as machine reading comprehension (MRC) problems has been proven to be effective. Therefore, we propose a RoBERTa-based model that treats indicator extraction as a question answering (QA) task. By answering questions customized for indicator elements which are the components of the indicator, it transforms extracting indicator elements to identifying and extracting answer spans from the given contexts. Experiments show that it outperforms other deep learning baselines.

Machine Reading Comprehension (MRC) is a challenging task in Natural Language Processing (NLP) field. It is a fast evolving field in the recent years because of the development of large-scale datasets and the deep learning technology. Even though, numerous MRC models have been developed for Question Answer (QA) system, the existing MRC models are not on par with reading comprehension. In order to assess and improve the performance of the projected datasets, an advanced level of evaluation metrics have been introduced. Therefore, this work has conducted a survey over the existing MRC models and evaluation metrics utilized in them to address the lack of MRC with human reading.

Conversational Machine Comprehension (CMC), a research track in\nconversational AI, expects the machine to understand an open-domain natural\nlanguage text and thereafter engage in a multi-turn conversation to answer\nquestions related to the text. While most of the research in Machine Reading\nComprehension (MRC) revolves around single-turn question answering (QA),\nmulti-turn CMC has recently gained prominence, thanks to the advancement in\nnatural language understanding via neural language models such as BERT and the\nintroduction of large-scale conversational datasets such as CoQA and QuAC. The\nrise in interest has, however, led to a flurry of concurrent publications, each\nwith a different yet structurally similar modeling approach and an inconsistent\nview of the surrounding literature. With the volume of model submissions to\nconversational datasets increasing every year, there exists a need to\nconsolidate the scattered knowledge in this domain to streamline future\nresearch. This literature review attempts at providing a holistic overview of\nCMC with an emphasis on the common trends across recently published models,\nspecifically in their approach to tackling conversational history. The review\nsynthesizes a generic framework for CMC models while highlighting the\ndifferences in recent approaches and intends to serve as a compendium of CMC\nfor future researchers.\n

A semi-supervised deep learning approach for cropped image detection

Recently, deep neural networks (DNNs) have been widely studied and achieved tremendous success in a variety of real-world applications, such as computer vision, natural language processing, medical diagnosis, and machine translation. Deep reinforcement learning (DRL), as an emerging powerful deep learning technique, combines DNNs with reinforcement learning into an interactive system that enables the agent to learn the best policy from the environment to optimize their rewards. DRL opens up many new applications in domains such as healthcare, robotics, and smart grids. With the rapid evolution of IT infrastructures, cloud computing has been witnessed as the most common computing paradigm. The underlying infrastructure of cloud computing relies on a large amount of data centers. Nowadays, with the fast development of communication technologies, such as 5G, cloud computing is now evolving into yet another new paradigm called edge computing, providing for emerging applications, especially in the Internet-of-Things. The energy efficiency issue from both "cloud" and "edge" are becoming more crucial and calls for more attention. To solve the real-world energy efficiency problems, we take advantage of the deep learning and deep reinforcement learning techniques for efficient modeling of both "edge" and "cloud" applications. On the "edge" side, wearable technologies and the Internet of Things (IoT) are becoming increasingly popular in both personal and commercial applications with the employment of the embedded system technologies. Non-Volatile Processors (NVPs) have been proposed for the intermittent power supply problem, to ensure the instant on/off for embedded systems under the unstable power supply. We proposed neural network-based prediction techniques to derive the best power extraction policy and converter parameters optimization technique for NVPs. In cloud computing, we formulated a novel optimal power management problem of data centers to minimize the total cost by proposing the long short term memory (LSTM) neural network-based method for the prediction of future data center power consumption, and further we presented a novel DRL-based hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. Furthermore, we explored the effectiveness of the DRL-based framework for high dimensional action and state spaces problem-dynamic treatment regimes, to model real-life complexity in heterogeneous disease progression and treatment choices, to provide doctors and patients the data-driven personalized decision recommendations. Last but not least, we explored the efficiency of deep neural networks. Structured weight pruning is a representative model compression technique of DNNs to reduce both the storage and computation requirements and meanwhile accelerate inference. An automatic hyperparameter determination process is necessary due to a large number of flexible hyperparameters. We propose an automatic structured pruning framework to compress the DNNs. Our framework outperforms the prior works on automatic model compression by up to 33× in pruning rate (120× reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the proposed framework on actual measurements on the smartphone.

Purpose. The digital transformation of business in green smart cities is crucial due to the evolving needs of residents and businesses, as well as the increasing demands for enhancing efficiency, improving the quality of urban life, and protecting the environment. The aim of this publication is to characterize the directions of development in artificial intelligence (AI) and cloud computing that support digital transformation and sustainable urban development. It is worth noting that there is a gap in the literature, as clear ideas are lacking on how to effectively utilize deep learning based on artificial neural networks (ANN) in the cloud. Methods. Basic research methods include critical analysis of the literature on the subject. In addition, modeling was used to simulate the use of deep learning and cloud computing in smart city management systems. Intensive computational experiments were carried out to analyze the quality of the solutions, which were determined by the proposed deep learning methods using ANN based on Long Short Term Memory (LSTM). Results. The results of theoretical research and numerical experiments confirmed significantly contributing AI and cloud computing in the increasing the efficiency of the city and improving the quality of resident’s life and protecting the natural environment.

Content-Based Image Retrieval (CBIR) has become a critical technology for efficiently searching and retrieving images from large datasets based on their visual content. Traditional CBIR systems, which rely on low-level features like color, texture, and shape, often struggle with semantic gaps and scalability issues. With the rapid advancements in deep learning and cloud computing, there is a growing need to enhance CBIR performance for real-world applications. In order to tackle the issues, an enhanced CBIR process leveraging advanced neural networks, particularly Convolutional Neural Networks (CNNs), Siamese Networks, and attention mechanisms is proposed. It integrates multimodal data, including text and audio, to improve retrieval accuracy. Additionally, cloud-based infrastructure is employed to support large-scale image processing, enabling faster retrieval times and real-time performance. Edge computing techniques are also incorporated to reduce latency in applications requiring immediate responses. The proposed model demonstrates significant improvements in retrieval accuracy and efficiency compared to traditional CBIR methods. Deep learning models, particularly CNNs with transfer learning and attention mechanisms, effectively capture high-level semantic features. The integration of cloud infrastructure enhances scalability and real-time processing capabilities, while multimodal retrieval improves search relevance. The use of explainable AI techniques adds transparency to the decision-making process, increasing user trust. Hence, the advanced neural networks, coupled with cloud and edge computing, can significantly optimize CBIR systems, making them more robust, scalable, and applicable to a wide range of industries such as healthcare, security, and e-commerce.

Deep (machine) learning in recent years has significantly increased the predictive modeling strengths for applications in the areas of computer vision, speech recognition, and natural language processing. Being heavily data-driven, conventional general-purpose deep learning methods mainly rely on training data to learn task-specific data representations. In this thesis, we focus on leveraging novel learning algorithms in deep learning to tackle visual recognition challenges at the three key levels -- namely pixel, object, and image levels. Throughout the thesis, we demonstrate how each and every of our proposed methods can address a specific visual recognition problem. Firstly, we apply deep learning to improve the saliency detection performance of deep networks. A coarse saliency map of the input image's salient object is initially predicted by a convolutional-deconvolutional network (or equivalently a fully convolutional network). Following that, the coarse saliency map is sequentially refined by a separate network called Recurrent Attentional Convolutional-Deconvolutional Network (RACDNN) that shares its network weights across its sequential steps. At every sequential (temporal) step, the same RACDNN is applied to a newly selected/attended image sub-region and it performs saliency map refinement for that particular sub-region. Despite having many learnable network parameters and no access to additional training data, RADCNN can be trained well and it outperforms the "single-step" (coarse saliency detection) baseline and state-of-the-art saliency detection methods by large margins, on several saliency detection benchmark datasets. Secondly, we introduce auto-encoder weight transfer networks (AE-WTN+), a deep model for scaling up object detection. Conventional WTN transfers object class knowledge from the classification weights of a pretrained large-scale image classification network, to the classification weights of an object detection network. However, they suffer from under-fitting and are not able to preserve the rich object class information of novel classes. In this work, we propose a new WTN method -- AE-WTN+ which makes use of normalization techniques to ease network optimization. Inputs and intermediate features are normalized to improve trainability for reduced under-fitting. More importantly, a novel auto-encoding (reconstruction) loss is proposed to encourage information preservation of all classes in the weight predictions of AE-WTN+. This enables the predicted weights of AE-WTN+ to maintain the class discriminability and neighbourhood relationships contained in the pre-trained large-scale classification weights. Experiments on large-scale detection datasets validate the effectiveness of AE-WTN+ on both seen and novel classes. Thirdly, we improve the efficiency of cross-layer network connections for very deep convolutional network in the context of image classification. This gives rise to a new convolutional network architecture called Deluge Networks (DelugeNets) which pass the output (feature activations) of any preceding composite network layer as direct inputs to all succeeding composite layers. For the incoming feature activations of preceding composite layers, the succeeding composite layers first apply rather lightweight cross-layer depthwise convolutional weights on them for feature aggregation. The heavy lifting is carried out by the last convolutional weight layer in each preceding composite layer that performs cross-channel convolutions. The proposed technique enables unimpeded information flow among many layers in very deep DelugeNets with great efficiency. Experiments on small-scale and large-scale datasets show that DelugeNets perform competitively to state-of-the-art deep convolutional networks at lower computational and parameter costs. Fourthly, we propose Stochastic Downsampling (SDPoint), a method to overcome a major limitation suffered by conventional deep convolutional networks -- their computational costs during inference are fixed and cannot be changed. SDPoint downsamples the feature maps at a randomly selected point (layer index) in the network hierarchy during training. The many downsampling configurations known as SDPoint instances each entail a unique downsampling point and downsampling ratio. The same convolutional network weights are shared across the different SDPoint instances that downsample feature maps differently. During inference, the computational cost of a network trained with SDPoint can be instantaneously adjusted to fit a given computational budget, by manually selecting an appropriate SDPoint instance. Additionally, sharing weights between different feature map scales provides significant regularization benefits, making the networks trained with SDPoint less sensitive to scales and more competent. In image classification experiments, SDPoint demonstrates significant cost-accuracy and computational advantages, over independently-trained (non-weight-shared) baselines with various inference costs.

In this article, the method of text classification using a convolutional neural network is presented. The problem of text classification is formulated, the architecture and the parameters of a convolutional neural network for solving the problem are described, the steps of the solution and the results of classification are given. The convolutional network which was used was trained to classify the texts of the news messages of Internet information portals. The semantic preprocessing of the text and the translation of words into attribute vectors are generated using the open word2vec model. The analysis of the dependence of the classification quality on the parameters of the neural network is presented. The using of the network allowed obtaining a classification accuracy of about 84%. In the estimation of the accuracy of the classification, the texts were checked to belong to the group of semantically similar classes. This approach allowed analyzing news messages in cases where the text themes and the number of classification classes in the training and control samples do not equal.

The question generation task can automatically generate large-scale questions to provide training data for reading comprehension tasks and QA systems, which are crucial for low-resource languages such as Tibetan. At present, due to the emergence of large-scale datasets and pre-trained language models in the Chinese and English domains, the task of question generation in the Chinese and English domains has been well developed, while the research on question generation in the Tibetan is still in its infancy. The main reason is the lack of datasets and the relatively backward development of various models in Tibetan. To solve the above questions, this paper constructs a Tibetan pre-trained language model TiBERT to provide a basis for the development of various downstream tasks, to expand the datasets of Tibetan machine reading comprehension, this paper proposes a Tibetan question generation model named TQGR. The model consists of two parts, the question generation and question quality assessment. The question generation adopts the classic seq2seq architecture to generate questions, and the question quality assessment improves the quality of generated questions by evaluating the fluency reward score, word repetition rate reward score and interrogative words classification auxiliary task. Finally, the experimental results show that our model has higher performance than baseline models, and ablation experiments demonstrate the effectiveness of the three mechanism.

Cloud computing and big data: Technologies and applications

By moving a depth sensor around a room, we compute a 3D CAD model of the environment, capturing the room shape and contents such as chairs, desks, sofas, and tables. Rather than reconstructing geometry, we match, place, and align each object in the scene to thousands of CAD models of objects. In addition to the fully automatic system, the key technical contribution is a novel approach for aligning CAD models to 3D scans, based on deep reinforcement learning. This approach, which we call Learning-based ICP, outperforms prior ICP methods in the literature, by learning the best points to match and conditioning on object viewpoint. LICP learns to align using only synthetic data and does not require ground truth annotation of object pose or keypoint pair matching in real scene scans. While LICP is trained on synthetic data and without 3D real scene annotations, it outperforms both learned local deep feature matching and geometric based alignment methods in real scenes. The proposed method is evaluated on real scenes datasets of SceneNN and ScanNet as well as synthetic scenes of SUNCG. High quality results are demonstrated on a range of real world scenes, with robustness to clutter, viewpoint, and occlusion.

Purpose:There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.Methods:A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines and micro-RNAs) in a population of 106 non-small cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. 22 patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint-architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM) and multi-layer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique (SMOTE) was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic (ROC) curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.Results:An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761–0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737–0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805–0.863) with 22 features (p-value = 0.000642 compared with handcrafted features only (Case A) and p-value=0.000453 compared to VAE alone (Case C)) with an MLP classifier.Conclusion:The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.

In order to adapt to the development and application of power dispatch control systems in the future cloud computing environment, this study proposes to make full use of the elastic characteristics of cloud computing to construct a new power dispatch control system architecture. First, we analyze the shortcomings of the traditional power dispatch control system architecture, from the perspective of architecture complexity, operation and maintenance, universality and disaster tolerance. Secondly, combining the characteristics of the existing dispatch control system, we propose a new power dispatch control system architecture that conforms to the cloud computing platform. Finally, we emphatically analyzed the key technologies for key breakthroughs needed to realize the power dispatching control system architecture. Through the comparison of traditional and new power dispatch control system architectures, it can be considered that the new power dispatching control system architecture has greatly improved reliability, flexibility, safety and economy, and has certain advantages.

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.