Distilled GPT for source code summarization

Production of a natural language description for the function of a source code segment is commonly referred to as source code summarization. Useful comments in source code can facilitate more rapid onboarding of new engineers and contribute to decreased maintenance costs. Unfortunately, the documentation task can also be labor intensive. In this paper, we introduce a new model agnostic tool for AI driven automation of source code summarization at scale. The initial version of the adVanced AI Driven Enhancement to Readability for Spurce Code (VADER-SC) software offers numerous options for customization and the ability to leverage a variety of AI models to enable experimentation in resource constrained environments, while also scaling up to benefit from larger models in contexts with increased compute resources. It further supports private cloud, self-hosted, and air-gapped network configurations for environments with strict intellectual property protections or processing of sensitive or controlled data. Qualitative and quantitative results suggest model selection, fine-tuning, and multi-shot tailoring significantly impact the quality of generated comments. VADER-SC could be an enabler for practitioners to explore large-scale automation of AI driven source code summarization and researchers may find it enables studies with larger volumes of disparate data across a diversity of AI model options and target programming languages.

Source code summarization is the task of creating readable summaries that describe the functionality of software. Source code summarization is a critical component of documentation generation, for example as Javadocs formed from short paragraphs attached to each method in a Java program. At present, a majority of source code summarization is manual, in that the paragraphs are written by human experts. However, new automated technologies are becoming feasible. These automated techniques have been shown to be effective in select situations, though a key weakness is that they do not explain the source code's context. That is, they can describe the behavior of a Java method, but not why the method exists or what role it plays in the software. In this paper, we propose a source code summarization technique that writes English descriptions of Java methods by analyzing how those methods are invoked. We then performed two user studies to evaluate our approach. First, we compared our generated summaries to summaries written manually by experts. Then, we compared our summaries to summaries written by a state-of-the-art automatic summarization tool. We found that while our approach does not reach the quality of human-written summaries, we do improve over the state-of-the-art summarization tool in several dimensions by a statistically-significant margin.

Code summarization generates brief natural language description given a source code snippet, while code retrieval fetches relevant source code given a natural language query. Since both tasks aim to model the association between natural language and programming language, recent studies have combined these two tasks to improve their performance. However, researchers have yet been able to effectively leverage the intrinsic connection between the two tasks as they train these tasks in a separate or pipeline manner, which means their performance can not be well balanced. In this paper, we propose a novel end-to-end model for the two tasks by introducing an additional code generation task. More specifically, we explicitly exploit the probabilistic correlation between code summarization and code generation with dual learning, and utilize the two encoders for code summarization and code generation to train the code retrieval task via multi-task learning. We have carried out extensive experiments on an existing dataset of SQL and Python, and results show that our model can significantly improve the results of the code retrieval task over the-state-of-art models, as well as achieve competitive performance in terms of BLEU score for the code summarization task.

Source code summarization aims at generating concise and clear natural language descriptions for programming languages. Well-written code summaries are beneficial for programmers to participate in the software development and maintenance process. To learn the semantic representations of source code, recent efforts focus on incorporating the syntax structure of code into neural networks such as Transformer. Such Transformer-based approaches can better capture the long-range dependencies than other neural networks including Recurrent Neural Networks (RNNs), however, most of them do not consider the structural relative correlations between tokens, e.g., relative positions in Abstract Syntax Trees (ASTs), which is beneficial for code semantics learning. To model the structural dependency, we propose a StruCtural RelatIve Position guided Transformer, named SCRIPT. SCRIPT first obtains the structural relative positions between tokens via parsing the ASTs of source code, and then passes them into two types of Transformer encoders. One Transformer directly adjusts the input according to the structural relative distance; and the other Transformer encodes the structural relative positions during computing the self-attention scores. Finally, we stack these two types of Transformer encoders to learn representations of source code. Experimental results show that the proposed SCRIPT outperforms the state-of-the-art methods by at least 1.6%, 1.4% and 2.8% with respect to BLEU, ROUGE-L and METEOR on benchmark datasets, respectively. We further show that how the proposed SCRIPT captures the structural relative dependencies.

Automatic source code summarization system aims to generate a valuable natural language description for a program, which can facilitate software development and maintenance, code categorization, and retrieval. However, previous sequence-based research did not consider the long-distance dependence and highly structured characteristics of source code simultaneously. In this article, we present a Transformer-based Graph-Augmented Source Code Summarization (GA-SCS), which can effectively incorporate inherent structural and textual features of source code to generate an effective code description. Specifically, we develop a graph-based structure feature extraction scheme leveraging abstract syntax tree and graph attention networks to mine global syntactic information. And then, to take full advantage of the lexical and syntactic information of code snippets, we extend the original attention to a syntax-informed self-attention mechanism in our encoder. In the training process, we also adopt a reinforcement learning strategy to enhance the readability and informativity of generated code summaries. We utilize the Java dataset and Python dataset to evaluate the performance of different models. Experimental results demonstrate that our GA-SCS model outperforms all competitive methods on BLEU, METEOR, ROUGE, and human evaluations.

With the fast development of large software projects, automatic code summarization techniques, which summarize the main functionalities of a piece of code using natural languages as comments, play essential roles in helping developers understand and maintain large software projects. Many research efforts have been devoted to building automatic code summarization approaches. Typical code summarization approaches are based on deep learning models. They transform the task into a sequence-to-sequence task, which inputs source code and outputs summarizations in natural languages. All code summarization models impose different input size limits, such as 50 to 10,000, for the input source code. However, how the input size limit affects the performance of code summarization models still remains under-explored. In this article, we first conduct an empirical study to investigate the impacts of different input size limits on the quality of generated code comments. To our surprise, experiments on multiple models and datasets reveal that setting a low input size limit, such as 20, does not necessarily reduce the quality of generated comments. Based on this finding, we further propose to use function signatures instead of full source code to summarize the main functionalities first and then input the function signatures into code summarization models. Experiments and statistical results show that inputs with signatures are, on average, more than 2 percentage points better than inputs without signatures and thus demonstrate the effectiveness of involving function signatures in code summarization. We also invite programmers to do a questionnaire to evaluate the quality of code summaries generated by two inputs with different truncation levels. The results show that function signatures generate, on average, 9.2% more high-quality comments than full code.

Translation between natural language and source code can help software development by enabling developers to comprehend, ideate, search, and write computer programs in natural language. Despite growing interest from the industry and the research community, this task is often difficult due to the lack of large standard datasets suitable for training deep neural models, standard noise removal methods, and evaluation benchmarks. This leaves researchers to collect new small-scale datasets, resulting in inconsistencies across published works. In this study, we present CoDesc - a large parallel dataset composed of 4.2 million Java methods and natural language descriptions. With extensive analysis, we identify and remove prevailing noise patterns from the dataset. We demonstrate the proficiency of CoDesc in two complementary tasks for code--description pairs: code summarization and code search. We show that the dataset helps improve code search by up to 22\% and achieves the new state-of-the-art in code summarization. Furthermore, we show CoDesc's effectiveness in pre-training--fine-tuning setup, opening possibilities in building pretrained language models for Java. To facilitate future research, we release the dataset, a data processing tool, and a benchmark at \url{https://github.com/csebuetnlp/CoDesc}.

Several studies have suggested that comments describing source code can help mitigate the burden of program understanding. However, software systems usually lack adequate comments and even when present, the comments may be obsolete or unhelpful. Researchers have addressed this issue by automatically generating comments from source code, a task referred to as Code Summarization. In this technical presentation, we take a deeper look at some of the significant, recent works in the area of code summarization and how each of them attempts to take a new perspective of this task including methods leveraging RNNs, Transformers, Graph neural networks and Reinforcement learning.

Code summarization in software engineering refers to the task of automatically generating short natural language descriptions for source code, which has been addressed by the development of transformer-based models such as CodeBERT, CodeT5, and CodeSearchNet in deep learning. These approaches have played a significant role in enhancing the capability of automated code documentation and producing good summaries. This paper provides an in-depth review of the latest transformer-based models for code summarization in terms of architectures, performance metrics, and diverse applications. Through the analysis of the strengths and weaknesses of these models, we aim to give insights that could guide future research and development in the area of automated code documentation. Through this survey, we will evaluate the progress of transformer-based models and give a foundation for further advancements in code summarization technologies.

Enterprise software systems generate vast volumes of unstructured and semi-structured artifacts such as source code, build logs, configuration files, architectural diagrams, design documents, operational runbooks, and incident records which collectively encode critical institutional knowledge but are rarely maintained in a consistent or timely manner. In large-scale, distributed, and continuously evolving environments, manual documentation quickly becomes outdated, fragmented, and misaligned with system behavior, leading to increased onboarding time, operational risk, and reduced system transparency. Recent advances in Generative Artificial Intelligence (GenAI), enabled by transformer-based language models and retrieval-augmented architectures, offer a viable path toward addressing these challenges by enabling machines to interpret, synthesize, and generate human-readable knowledge directly from evolving system artifacts. This article presents a structured approach to automated documentation using GenAI agents that integrate semantic retrieval, neural language generation, and knowledge extraction pipelines to transform raw technical data into coherent, verifiable documentation. By grounding generative outputs in retrieved evidence and structured representations, such agents mitigate hallucination risks while supporting traceability, auditability, and continuous updates aligned with system changes. Drawing on prior research in transformers, open-domain question answering, information extraction, and code summarization, the study identifies architectural patterns, enabling technologies, and empirical findings that collectively demonstrate the practicality and scalability of GenAI-driven documentation systems prior to 2022.

The current landscape of binary code summarization predominantly focuses on generating a single summary, which limits its utility and understanding for reverse engineers. Existing approaches often fail to meet the diverse needs of users, such as providing detailed insights into usage patterns, implementation nuances, and design rationale, as observed in the field of source code summarization. This highlights the need for multi-intent binary code summarization to enhance the effectiveness of reverse engineering processes. To address this gap, we propose MiSum, a novel method that leverages multi-modality heterogeneous code graph alignment and learning to integrate both assembly code and pseudo-code. MiSum introduces a unified multi-modality heterogeneous code graph (MM-HCG) that aligns assembly code graphs with pseudo-code graphs, capturing both low-level execution details and high-level structural information. We further propose multi-modality heterogeneous graph learning with heterogeneous mutual attention and message passing, which highlights important code blocks and discovers inter-dependencies across different code forms. Additionally, an intent-aware summary generator with an intent-aware attention mechanism is introduced to produce customized summaries tailored to multiple intents. Extensive experiments, including evaluations across various architectures and optimization levels, demonstrate that MiSum outperforms state-of-the-art baselines in BLEU, METEOR, and ROUGE-L metrics. Human evaluations validate its capability to effectively support reverse engineers in understanding diverse binary code intents, marking a significant advancement in binary code analysis.

Converting source or unit test code to English has been shown to improve the maintainability, understandability, and analysis of software and tests. Code summarizers identify important statements in the source/tests and convert them to easily understood English sentences using static analysis and NLP techniques. However, current test summarization approaches handle only a subset of the variation and customization allowed in the JUnit assert API (a critical component of test cases) which may affect the accuracy of conversions. In this paper, we present our work towards improving JUnit test summarization with a detailed process for converting a total of 45 unique JUnit assertions to English, including 37 previously-unhandled variations of the assertThat method. This process has also been implemented and released as the AssertConvert tool. Initial evaluations have shown that this tool generates English conversions that accurately represent a wide variety of assertion statements which could be used for code summarization or other NLP analyses.

Software developers write a lot of source code and documentation during software development. Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them. To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models. REDCODER has a couple of uniqueness. First, it extends the state-of-the-art dense retrieval technique to search for relevant code or summaries. Second, it can work with retrieval databases that include unimodal (only code or natural language description) or bimodal instances (code-description pairs). We conduct experiments and extensive analysis on two benchmark datasets of code generation and summarization in Java and Python, and the promising results endorse the effectiveness of our proposed retrieval augmented framework.

We demonstrate how deep learning over programs is used to provide (preliminary) augmented programmer intelligence. In the first part, we show how to tackle tasks like code completion, code summarization, and captioning. We describe a general path-based representation of source code that can be used across programming languages and learning tasks, and discuss how this representation enables different learning algorithms. In the second part, we describe techniques for extracting interpretable representations from deep models, shedding light on what has actually been learned in various tasks.

Modern software developers tend to engage in social coding platforms to reuse code snippets to expedite the development process, while the codes on such platforms are often suffering from comments being mismatched, missing or outdated. This puts the code search and comprehension in difficulty, and increases the burden of maintenance for software building upon these codes. As summarizing code is beneficial yet it is very expensive for manual operation, in this paper, we elaborate an automatic and effective code summarization paradigm to address this laborious challenge. We represent a given code snippet as an abstract syntax tree (AST), and generate a set of compositional root-to-leaf paths to make the AST accessible regarding code context and structure in a less complex yet expressive way. Accordingly, we design a tree-based transformer model, called TreeXFMR, on these paths to summarize source code in a hierarchical attention operation. This yields two advantages on code representation learning: (1) attention mechanisms at token-and path-level attend the semantics and interactions of source code from different aspects; (2) bi-level positional encodings introduced reveal the intra- and inter-path structure of AST and improve the unambiguity of the representations. During decoding, TreeXFMR attends such learned representations to produce each output of natural language word. We further pre-train the transformer to achieve faster and better training convergence results. Extensive experiments on the code collection from GitHub demonstrate the effectiveness of TreeXFMR, which significantly outperforms state-of-the-art baselines.