Deep learning similarities from different representations of source code

On the impact of multiple source code representations on software engineering tasks — An empirical study

Context:Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge.Objective:Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors:(1) reproducibility—whether the reported experimental results can be obtained by other researchers using authors’ artifacts (i.e., source code and datasets) with the same experimental setup; and(2) replicability—whether the reported experimental result can be obtained by other researchers using their re-implemented artifacts with a different experimental setup. We observed that DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process, unlike classical supervised machine learning (ML) methods (e.g., random forest). This study aims to investigate the urgency and importance of reproducibility and replicability for DL studies on SE tasks.Method:In this study, we conducted a literature review on 147 DL studies recently published in 20 SE venues and 20 AI (Artificial Intelligence) venues to investigate these issues. We also re-ran four representative DL models in SE to investigate important factors that may strongly affect the reproducibility and replicability of a study.Results:Our statistics show the urgency of investigating these two factors in SE, where only 10.2% of the studies investigate any research question to show that their models can address at least one issue of replicability and/or reproducibility. More than 62.6% of the studies do not even share high-quality source code or complete data to support the reproducibility of their complex models. Meanwhile, our experimental results show the importance of reproducibility and replicability, where the reported performance of a DL model could not be reproduced for an unstable optimization process. Replicability could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data.Conclusion:It is urgent for the SE community to provide a long-lasting link to a high-quality reproduction package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.

Engineering of the compiled software is an important task in software engineering. One of the stages of the engineering technologies is a construction and analysis of a control flow graphs, which reflect a general structure of algorithms. The paper presents a technique for analyzing and visualizing the control flow graph of a compiled software. The analysis is based on semantically equivalent transformations of the original graph resulting in an abstract node that contains a hierarchy of isolated regions, which then associated with predefined templates resulting a graphical representation. The templates and additional signature data allow one to recognize the high-level programming structures and user statements to construct a flow-chart notation of the original program.

Software clone detection, aiming at identifying out code fragments with similar functionalities, has played an important role in software maintenance and evolution. Many clone detection approaches have been proposed. However, most of them represent source codes with hand-crafted features using lexical or syntactical information, or unsupervised deep features, which makes it difficult to detect the functional clone pairs, i.e., pieces of codes with similar functionality but differing in both syntactical and lexical level. In this paper, we address the software functional clone detection problem by learning supervised deep features. We formulate the clone detection as a supervised learning to hash problem and propose an end-to-end deep feature learning framework called CDLH for functional clone detection. Such framework learns hash codes by exploiting the lexical and syntactical information for fast computation of functional similarity between code fragments. Experiments on software clone detection benchmarks indicate that the CDLH approach is effective and outperforms the state-of-the-art approaches in software functional clone detection.

PathPair2Vec: An AST path pair-based code representation method for defect prediction

Source code summarization aims at generating concise and clear natural language descriptions for programming languages. Well-written code summaries are beneficial for programmers to participate in the software development and maintenance process. To learn the semantic representations of source code, recent efforts focus on incorporating the syntax structure of code into neural networks such as Transformer. Such Transformer-based approaches can better capture the long-range dependencies than other neural networks including Recurrent Neural Networks (RNNs), however, most of them do not consider the structural relative correlations between tokens, e.g., relative positions in Abstract Syntax Trees (ASTs), which is beneficial for code semantics learning. To model the structural dependency, we propose a StruCtural RelatIve Position guided Transformer, named SCRIPT. SCRIPT first obtains the structural relative positions between tokens via parsing the ASTs of source code, and then passes them into two types of Transformer encoders. One Transformer directly adjusts the input according to the structural relative distance; and the other Transformer encodes the structural relative positions during computing the self-attention scores. Finally, we stack these two types of Transformer encoders to learn representations of source code. Experimental results show that the proposed SCRIPT outperforms the state-of-the-art methods by at least 1.6%, 1.4% and 2.8% with respect to BLEU, ROUGE-L and METEOR on benchmark datasets, respectively. We further show that how the proposed SCRIPT captures the structural relative dependencies.

Pre-trained models (PTMs) have succeeded in various software engineering (SE) tasks following the “pre-train then fine-tune” paradigm. As fully fine-tuning all parameters of PTMs can be computationally expensive, a potential solution is parameter-efficient fine-tuning (PEFT), which freezes PTMs while introducing extra parameters. Although PEFT methods have been applied to SE tasks, researchers often focus on specific scenarios and lack a comprehensive comparison of PTMs from different aspects such as field, size, and architecture. To fill this gap, we have conducted an empirical study on six PEFT methods, eight PTMs, and four SE tasks. The experimental results reveal several noteworthy findings. For example, model architecture has little impact on PTM performance when using PEFT methods. Additionally, we provide a comprehensive discussion of PEFT methods from three perspectives. First, we analyze the effectiveness and efficiency of PEFT methods. Second, we explore the impact of the scaling factor hyperparameter. Finally, we investigate the application of PEFT methods on the latest open-source large language model, Llama 3.2. These findings provide valuable insights to guide future researchers in effectively applying PEFT methods to SE tasks.

Duplicate code is an example of bad smells, which are usually been refactored after the detection to improve the quality of programs. Locate the duplicate code at the programming phase may reduce the cost of maintenance, but the challenge is it need to detect duplicate code between an incomplete code fragment with complete files, which the existing tools are hard to be applied to this scenario. In this paper, we propose an AST-sequence-based duplicate code detection approach for onsite programming code. The abstract syntax tree (AST) is extracted from source code and then is transformed into an encoded sequence. A local sequence alignment algorithm is used to find highly similar subsequences. After the post-processing, similar regions will be found between two code fragments according to the subsequences. We have developed a prototype tool as a plugin for Visual Studio Code. Experimental results indicate that our approach is effective in finding highly similar regions between cross-granularity code fragments, which can facilitate duplicate code detection for incomplete onsite programming code.

Software engineering tasks like modeling and programming in mechanical engineering are influenced by different factors. This includes knowledge and abilities in the specific profession but also characteristics of the person and gender aspects are necessary. Those personal factors on the hand are cognitive abilities and knowledge acquired through prior experience and on the other hand, personal characteristics like personality structure, self-efficacy, motivation and mental workload. It also includes the consideration of gender aspects and the relationship of gender specific characteristics of personal factors to performance in software engineering tasks. The aim of the research approach is to reveal characteristics of the person associated with performance in software engineering tasks. An empirical study examined the gender differences in the personal factors. Furthermore, females' personal factors relationship to software engineering performance was examined. Finally, an additional plant programming course was introduced containing the work with real hardware. The exam (behavior and structure modeling) at the end of the course showed that especially females benefit from the more practical learning method.

Survey of source code vulnerability analysis based on deep learning

Approaches that support software maintenance need to be evaluated and compared against existing ones, in order to demonstrate their usefulness in practice. However, oftentimes the lack of well-established sets of benchmarks leads to situations where these approaches are evaluated using different datasets, which results in biased comparisons. In this data paper we describe and make publicly available a set of benchmarks from six Java applications, which can be used in the evaluation of various software engineering (SE) tasks, such as feature location and impact analysis. These datasets consist of textual description of change requests, the locations in the source code where they were implemented, and execution traces. Four of the benchmarks were already used in several SE research papers, and two of them are new. In addition, we describe in detail the methodology used for generating these benchmarks and provide a suite of tools in order to encourage other researchers to validate our datasets and generate new benchmarks for other subject software systems. Our online appendix: http://www.cs.wm.edu/semeru/data/msr13/.

Code classification and code clone detection are crucial for understanding and maintaining large software systems. Although deep learning surpasses traditional techniques in capturing the features of source code, existing models suffer from low processing power and high complexity. We propose a novel source code representation method based on the multi-head attention mechanism (SCRMHA). SCRMHA captures the vector representation of entire code segments, enabling it to focus on different positions of the input sequence, capture richer semantic information, and simultaneously process different aspects and relationships of the sequence. Moreover, it can calculate multiple attention heads in parallel, speeding up the computational process. We evaluate SCRMHA on both the standard dataset and an actual industrial dataset, and analyze the differences between these two datasets. Experiment results in code classification and clone detection tasks show that SCRMHA consumes less time and reduces complexity by about one-third compared with traditional source code feature representation methods. The results demonstrate that SCRMHA reduces the computational complexity and time consumption of the model while maintaining accuracy.

Theory-oriented software engineering

Context: The identification of code smells is one of the most subjective tasks in software engineering. A key reason is the influence of collective aspects of communities working on this task, such as their beliefs regarding the relevance of certain smells. However, collective aspects are often neglected in the context of smell identification. For this purpose, we can use the social representations theory. Social representations comprise the set of values, behaviors and practices of communities associated with a social object, such as the task of identifying smells. Aim: To characterize the social representations behind smell identification. Method: We conducted a preliminary study on the social representations of smell identification by two communities. One community is composed of postgraduate students involved in various investigations related to code smells. The other community is composed of practitioners from industry, with experience in code reviews. We analyzed the associations made by the study participants about smell identification, i.e., what immediately comes to their minds when they think about this task. Results: One of the key findings is that only the community of practitioners strongly associates this task with semantic smells. This finding suggests research directions on code smells may be revisited, as they focus mostly on measurable or structural smells. Considering the novelty of using the social representations theory in software engineering, we also compiled a set of lessons learned. For instance, we observed some key challenges we faced in using the theory. These challenges include: (i) the predominance of associations with technical rather than non-technical concepts, and (ii) the fuzzy definitions of key concepts in our field. Conclusion: We found initial evidence that social representations analysis is a useful instrument to reveal discrepancies and commonalities on how different communities deal with a subjective task. Thus, we expect the experience reported in this paper may encourage and contribute to future studies of social representations in the field.

With the increasing popularity of Internet, more and more developers are collaborating together for software development. During the collaboration, a lot of information related to software development, including communication and coordination information of developers, can be recorded in software repositories. The information can be employed to construct Developer Social Networks (DSNs) for facilitating tasks in software engineering. In this paper, we survey recent advances of DSNs and examine three fundamental steps of DSNs, namely construction, analysis, and applications. We summarize the state-of-the-art methods in the three steps and investigate the relationships among them. Furthermore, we discuss the main issues and point out the future opportunities in the study of DSNs.