Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs

Code completion is an important feature in Integrated Development Environments (IDEs). These years, researchers have been making efforts for intelligent code completion. However, existing work on intelligent code completion either only considered production code, or did not distinguish between production code and test code. It is unclear how effective existing completion models are for test code completion, nor whether we can further improve it. In this work, we focus on the completion of test code. We first find through experiments that completion models for production code are suboptimal for test code completion. Then we analyze the specific characteristics of test code, and observe that test code has inter- and intra-project similarities, and a strong relationship with its focal class and other production classes depending on the focal class (i.e., focal-related code). By incorporating test code from other projects to fine-tune existing models, we leverage the inter-project similarity of test code to improve the completion of tokens specific to test code. By introducing a local component and constructing existing test code as well as the focal-related code in the project as references, we enhance existing code completion models with the intra-project similarity and the focal-related code of test code. Experiments show that each characteristic of test code we exploit can bring substantial improvement to test code completion and our integrated framework outperforms other baseline frameworks. Compared to the base completion model, on token-level completion, our optimal model for test code completion relatively improves all-token and identifier completion accuracy by 7.68% and 19.96%, respectively; on line-level completion, it relatively improves edit-distance similarity and exact-match metrics by 8.89% and 22.82%, respectively. Moreover, we perform error analysis and point out potential directions for future work.

Code completion, one of the most useful features in the Integrated Development Environments (IDEs), can accelerate software development by suggesting the libraries, APIs, and method names in real-time. Recent studies have shown that statistical language models can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from three major drawbacks: a) The hierarchical structural information of the programs is not fully utilized in the program's representation; b) In programs, the semantic relationships can be very long. Existing recurrent neural networks based language models are not sufficient to model the long-term dependency. c) Existing approaches perform a specific task in one model, which leads to the underuse of the information from related tasks. To address these challenges, in this paper, we propose a self-attentional neural architecture for code completion with multi-task learning. To utilize the hierarchical structural information of the programs, we present a novel method that considers the path from the predicting node to the root node. To capture the long-term dependency in the input programs, we adopt a self-attentional architecture based network as the base language model. To enable the knowledge sharing between related tasks, we creatively propose a Multi-Task Learning (MTL) framework to learn two related tasks in code completion jointly. Experiments on three real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.

Code completion is one of the most useful features in the Integrated Development Environments (IDEs), which can accelerate software development by suggesting the next probable token based on the contextual code in real-time. Recent studies have shown that statistical language modeling techniques can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from two major drawbacks: a) Existing research uses static embeddings, which map a word to the same vector regardless of its context. The differences in the meaning of a token in varying contexts are lost when each token is associated with a single representation; b) Existing language model based code completion models perform poor on completing identifiers, and the type information of the identifiers is ignored in most of these models. To address these challenges, in this paper, we develop a multi-task learning based pre-trained language model for code understanding and code generation with a Transformer-based neural architecture. We pre-train it with hybrid objective functions that incorporate both code understanding and code generation tasks. Then we fine-tune the pre-trained model on code completion. During the completion, our model does not directly predict the next token. Instead, we adopt multi-task learning to predict the token and its type jointly and utilize the predicted type to assist the token prediction. Experiments results on two real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.

Intelligent code completion has become an area of interest to accelerate software development. In this study, novel methods for code completion are proposed and the performance of these methods on code completion is demonstrated. Deep Learning models are applied over large datasets to achieve code completion in dynamically written programming languages. In this study, an LSTM model that performs next word prediction is trained on the JavaScript programming language. The code completion system consists of two phases. In the first phase, multiple completion suggestions are generated from the LSTM model. In the second phase, these suggestions are checked by the JavaScript Parser and the best results are achieved. This system is an alternative solution where the model can be trained without preprocessing the training set with a Parser. The proposed method has been tested on the test dataset and the success of the code completion is shown in a quantitative way.

Integrated Development Environments (IDEs) are pivotal in enhancing productivity with features like code completion in modern software development. Recent advancements in Natural Language Processing (NLP) have empowered neural language models for code completion. In this study, we present an extensive investigation of the impact of open and closed vocabulary systems on the task of code completion. Specifically, we compare open and closed vocabulary systems with various vocabulary sizes to observe their impact on code completion performance. We experiment with three different open vocabulary systems: byte pair encoding (BPE), WordPiece and Unigram to compare them with closed-vocabulary systems to analyze their modeling performance. We also conduct experiments with different context sizes to study their impact on code completion performance. We have experimented using various prominent language models, including one from recurrent neural networks and five from transformers. Our results indicate that vocabulary size significantly impacts modeling performance and can artificially boost the accuracy of code completion models, especially in the case of a closed-vocabulary system. Moreover, we find that different vocabulary systems have varying impacts on token coverage, whereas open-vocabulary systems exhibit better token coverage. Our findings offer valuable insights for building effective code completion models, aiding researchers and practitioners in this field.

In current software development, code completion is necessary to enhance productivity of our programming tasks. However, how developers use code completion tools on integrated development environments is still not elucidated completely. Aiming to improve such tools, we performed an investigation in terms of code completion use. We investigated developers' operation histories on an integrated development environment and found that code completion operations inserting the same text tend to be repetitively performed in a short time period. We also propose new code completion strategies to reduce such repetitive code completion.

ADMET property prediction via multi-task graph learning under adaptive auxiliary task selection

The suggestions made by current IDE's code completion features are based exclusively on static type system of the programming language. As a result, often proposals are made which are irrelevant for a particular working context. Also, these suggestions are ordered alphabetically rather than by their relevance in a particular context. In this paper, we present intelligent code completion systems that learn from existing code repositories. We have implemented three such systems, each using the information contained in repositories in a different way. We perform a large-scale quantitative evaluation of these systems, integrate the best performing one into Eclipse, and evaluate the latter also by a user study. Our experiments give evidence that intelligent code completion systems which learn from examples significantly outperform mainstream code completion systems in terms of the relevance of their suggestions and thus have the potential to enhance developers' productivity.

Code completion is an essential feature of IDEs, yet current autocompleters are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant drawbacks: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand the semantics of the programming language and the developer's code context. In this work, we present CodeFill, a language model for autocompletion that combines learned structure and naming information. Using a parallel Transformer architecture and multi-task learning, CodeFill consumes sequences of source code token names and their equivalent AST token types. Uniquely, CodeFill is trained both for single-token and multi-token (statement) prediction, which enables it to learn long-range dependencies among grammatical and naming elements. We train CodeFill on two datasets, consisting of 29M and 425M lines of code, respectively. To make the evaluation more realistic, we develop a method to automatically infer points in the source code at which completion matters. We compare CodeFill against four baselines and two state-of-the-art models, GPT-C and TravTrans+.CodeFill surpasses all baselines in single token prediction (MRR: 70.9% vs. 66.2% and 67.8%) and outperforms the state of the art for multi-token prediction (ROUGE-L: 63.7% vs. 52.4% and 59.2%, for n=4 tokens). We publicly release our source code and datasets.

Multi-task feature learning has been proposed to improve the generalization performance by learning the shared features among multiple related tasks and it has been successfully applied to many real-world problems in machine learning, data mining, computer vision and bioinformatics. Most existing multi-task feature learning models simply assume a common noise level for all tasks, which may not be the case in real applications. Recently, a Calibrated Multivariate Regression (CMR) model has been proposed, which calibrates different tasks with respect to their noise levels and achieves superior prediction performance over the non-calibrated one. A major challenge is how to solve the CMR model efficiently as it is formulated as a composite optimization problem consisting of two non-smooth terms. In this paper, we propose a variant of the calibrated multi-task feature learning formulation by including a squared norm regularizer. We show that the dual problem of the proposed formulation is a smooth optimization problem with a piecewise sphere constraint. The simplicity of the dual problem enables us to develop fast dual optimization algorithms with low per-iteration cost. We also provide a detailed convergence analysis for the proposed dual optimization algorithm. Empirical studies demonstrate that, the dual optimization algorithm quickly converges and it is much more efficient than the primal optimization algorithm. Moreover, the calibrated multi-task feature learning algorithms with and without the squared norm regularizer achieve similar prediction performance and both outperform the non-calibrated ones. Thus, the proposed variant not only enables us to develop fast optimization algorithms, but also keeps the superior prediction performance of the calibrated multi-task feature learning over the non-calibrated one.

In order to effectively improve energy efficiency, integrated energy system has recently become an important form of energy utilization. Multi-load forecasting of integrated energy system with superior performance is very important to the planning and operation of power grid planning. Considering the coupling of energy sources and the randomness and uncertainty of load forecasting in integrated energy system, a short-term multi-load interval forecasting method for integrated energy system based on Multi-task learning is proposed in this paper. Firstly, multi-task learning is adopted to establish a close relationship between multiple energy sources. Secondly, the RELM optimized by PSO is used to establish the multivariate interval prediction model. Finally, the actual data of an integrated energy system are used to verify the effectiveness of the proposed method.

Large Language Models (LLMs) have achieved remarkable success in code completion, as evidenced by their essential roles in developing code assistant services such as Copilot. Being trained on in-file contexts, current LLMs are quite effective in completing code for single source files. However, it is challenging for them to conduct repository-level code completion for large software projects that require cross-file information. Existing research on LLM-based repository-level code completion identifies and integrates cross-file contexts, but it suffers from low accuracy and limited context length of LLMs. In this paper, we argue that Integrated Development Environments (IDEs) can provide direct, accurate and real-time cross-file information for repository-level code completion. We propose IDECoder, a practical framework that leverages IDE native static contexts for cross-context construction and diagnosis results for self-refinement. IDECoder utilizes the rich cross-context information available in IDEs to enhance the capabilities of LLMs of repository-level code completion. We conducted preliminary experiments to validate the performance of IDECoder and observed that this synergy represents a promising trend for future exploration.

Code completion is one of the most widely used features of modern integrated development environments (IDEs). While deep learning has made significant progress in the statistical prediction of source code, state-of-the-art neural network models consume hundreds of megabytes of memory, bloating the development environment. We address this in two steps: first we present a modular neural framework for code completion. This allows us to explore the design space and evaluate different techniques. Second, within this framework we design a novel reranking neural completion model that combines static analysis with granular token encodings. The best neural reranking model consumes just 6 MB of RAM, - 19x less than previous models - computes a single completion in 8 ms, and achieves 90% accuracy in its top five suggestions.

In the field of code intelligence, pretrained models exhibit impressive performance. However, it is imperative to modify all model parameters and maintain full copies for various tasks. Moreover, the effectiveness of fine-tuning a pretrained model depends on the availability of data, which can be constrained in practical settings. Prefix-tuning, a novel approach in NLP for addressing the aforementioned issue, has demonstrated promising results in numerous tasks. This paper aims to investigate the potential of prefix-tuning in the field of code intelligence, an area that has been relatively understudied. We also acknowledge the critical role of initialization in the performance of prefixtuning, which has been neglected in prior research. To address this gap, we introduce a novel method termed "adaptive prefixtuning". This approach entails training the prefix parameters on adapting tasks, followed by subsequent tuning on downstream tasks. We perform prefix-tuning and adaptive prefix-tuning using well-known pretrained models, CodeT5 and CodeGPT, and conduct experiments on four code intelligence tasks, namely, defect detection, code completion, code summarization, and code translation. Our experiments revealed that, in situations with restricted data availability, adaptive prefix-tuning yields substantial performance enhancements compared to fine-tuning, whereas prefix-tuning does not exhibit evident advantages. Furthermore, adaptive prefix-tuning has demonstrated superior performance relative to prefix-tuning in scenarios with comprehensive datasets. In certain cases, it has even outperformed traditional fine-tuning methods. Our findings indicate that within the field of code intelligence, adaptive prefix-tuning can serve as an effective substitute for fine-tuning, especially in situations with constrained data availability.

Early detection and timely surgical intervention are crucial in reducing mortality rates associated with clinically significant prostate cancer (csPCa). Currently, clinical diagnostics primarily depend on magnetic resonance imaging (MRI) and nuclear medicine, with the potential diagnostic value of abdominal computed tomography (CT) remaining underexplored. This study aims to evaluate the effectiveness of multi-task deep learning neural networks in identifying early-stage prostate cancer using CT scans. In this study, we enrolled 539 patients from the Department of Radiology (N=461) and Nuclear Medicine (N=78). We utilized a multi-task deep learning network model (MTDL), based on the 3DUnet architecture, to segment and analyze the collected abdominal plain CT images. The predictive performance of this model was compared with a radiomics model and a single-task deep learning model using ResNet18. A diagnostic nomogram was then developed using the multi-task deep learning approach, incorporating prediction results and PSAD, age. The diagnostic performance of the different models was evaluated using the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The 461 patients from the Department of Radiology were divided into training and test sets at a ratio of 6:4, while the patients from the Department of Nuclear Medicine formed the validation set. Our MTDL nomogram demonstrated AUCs of 0.941 (95% confidence interval [CI]: 0.905valceedi 0.912 (95% CI: 0.904valceedi and 0.932 (95% CI: 0.883valceed in the training, test, and validation cohorts, respectively. This study indicates that combining abdominal CT with a multi-task neural network model effectively diagnoses csPCa, offering superior diagnostic performance compared to clinical models. Additionally, the multi-task neural network model outperformed both the single-task neural network model and the radiomics model in diagnostic accuracy. Our study demonstrated that the MTDL nomogram can accurately predict the presence of prostate cancer using abdominal CT scans, offering significant value for the early diagnosis of prostate cancer.