Using Hierarchical Latent Dirichlet Allocation to Construct Feature Tree for Program Comprehension

During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer’s time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers’ program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.’s approach to assign developers’ activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend $\sim$ 58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers’ experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.

This paper describes the architecture of an ongoing developed Flexible and Extensible Program comprehension Support System (FEPSS). Program comprehension is a process for supporting users to understand a software system and is required for forward, reverse engineering, and re-engineering. In a program comprehension process some artifacts such as control-flow graphs and data-flow graphs need to be generated to help users in understanding and some artifacts such as design rationale and testing cases also need to be recorded for further re-engineering. It is impossible to predict all the types of artifacts to be supported in a program comprehension process as at least no one makes a re-engineering plan when designing a new software system. Therefore a program comprehension system should support a great flexibility and extensibility in artifact modelling. The approach which FEPSS takes is to provide a support system where the users can add new artifacts even at run time. FEPSS uses a relational object model to link the artifacts in order to integrate various activities in a program comprehension process as a whole. To help users to understand the artifacts and their relationships FEPSS uses 3D visualization to display artifacts. As the number of the artifacts for a software system can be very large FEPSS provides a navigation tool for users to control what artifacts to display. Based on these mechanisms, FEPSS can support users to analyze, model, and explore the artifacts through pictures.

During software development and maintenance, developers spend a considerable amount of time on program comprehension. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions.

Program comprehension is a fundamental prerequisite before software engineers may engage in software maintenance or evolution activities and requires the study of large amounts of documentation - either developer documentation or reverse engineered. Very often, from this documentation is missing a short overview document pointing to the most important classes of the system, these who are essential for starting the understanding of the systems architecture. In this work we propose a recommender tool to automatically identify the most important classes of a system. Our approach relies on modeling the static dependencies structure of the system as a graph and applying a graph ranking algorithm. We empirically identify the optimal way of building the system graph, identifying how different dependency types should be taken into account. In experiments performed on a set of open source real life systems, we compare the sets of classes recommended by our tool with these included in the architectural overviews provided by the system developers.

Developers spend a substantial amount of their time with program comprehension. To improve their comprehension and refresh their memory, developers need to communicate with other developers, read the documentation, and analyze the source code. Many studies show that developers focus primarily on the source code and that small improvements can have a strong impact. As such, it is crucial to bring the code itself into a more comprehensible form. A particular technique for this purpose are explicit feature traces to easily identify a program’s functionalities. To improve our empirical understanding about the effects of feature traces, we report an online experiment with 49 professional software developers. We studied the impact of explicit feature traces, namely annotations and decomposition, on program comprehension and compared them to the same code without traces. Besides this experiment, we also asked our participants about their opinions in order to combine quantitative and qualitative data. Our results indicate that, as opposed to purely object-oriented code: (1) annotations can have positive effects on program comprehension; (2) decomposition can have a negative impact on bug localization; and (3) our participants perceive both techniques as beneficial. Moreover, none of the three code versions yields significant improvements on task completion time. Overall, our results indicate that lightweight traceability, such as using annotations, provides immediate benefits to developers during software development and maintenance without extensive training or tooling; and can improve current industrial practices that rely on heavyweight traceability tools (e.g., DOORS) and retroactive fulfillment of standards (e.g., ISO-26262, DO-178B).

The bag-of-words model used for some clustering methods is often unsatisfactory as it ignores the relationship between the important terms that do not cooccur literally. In this paper, a document clustering algorithm based on semi-constrained Hierarchical Latent Dirichlet Allocation (HLDA) is proposed, the frequent itemsets is considered as the input of this algorithm, some keywords are extracted as the prior knowledge from the original corpus and each keyword is associated with an internal node, which is thought as a constrained node and adding constraint to the path sampling processing. Experimental results show that the semi-constrained HLDA algorithm outperforms the LDA, HLDA and semi-constrained LDA algorithms.KeywordsDocument clusteringHLDAfrequent itemsetsPMIempirical likelihood

Topic models are widely used to discover the latent representation of a set of documents. The two canonical models are latent Dirichlet allocation, and Gaussian latent Dirichlet allocation, where the former uses multinomial distributions over words, and the latter uses multivariate Gaussian distributions over pre-trained word embedding vectors as the latent topic representations, respectively. Compared with latent Dirichlet allocation, Gaussian latent Dirichlet allocation is limited in the sense that it does not capture the polysemy of a word such as "bank." In this paper, we show that Gaussian latent Dirichlet allocation could recover the ability to capture polysemy by introducing a hierarchical structure in the set of topics that the model can use to represent a given document. Our Gaussian hierarchical latent Dirichlet allocation significantly improves polysemy detection compared with Gaussian-based models and provides more parsimonious topic representations compared with hierarchical latent Dirichlet allocation. Our extensive quantitative experiments show that our model also achieves better topic coherence and held-out document predictive accuracy over a wide range of corpus and word embedding vectors which significantly improves the capture of polysemy compared with GLDA and CGTM. Our model learns the underlying topic distribution and hierarchical structure among topics simultaneously, which can be further used to understand the correlation among topics. Moreover, the added flexibility of our model does not necessarily increase the time complexity compared with GLDA and CGTM, which makes our model a good competitor to GLDA.

Many organizations and open-source projects use the C preprocessor (CPP) to implement configurability in their software systems. Despite extensive research, existing studies on the effects of CPP use on program comprehension are still limited to experiences, opinions, and empirical studies with narrow scopes. So, it is unclear whether the CPP actually leads to what is sometimes referred to as "#ifdef hell." In this paper, we expand the existing evidence on program comprehension in the presence of CPP directives, but we also highlight a surprising dilemma. We conducted an empirical study, including an experiment and a questionnaire, on the impact of refactoring CPP directives with 521 experienced software developers. The results indicate that, in contrast to previous findings, comprehension performance slightly worsened in terms of correctness when our participants worked on code with refactored CPP directives. However, in alignment with previous findings, they preferred the refactored code, considering it more comprehensible and easier to work with. This dilemma of objective performance versus subjective preference is a surprising outcome that has not been found before. We argue that our work motivates the need for more studies to understand this dilemma-which may significantly impact common beliefs in research and practice.

Unit testing represents a key activity in software development and maintenance. Test suites with high internal quality facilitate maintenance activities, such as code comprehension and regression testing. Several guidelines have been proposed to help developers write good test suites. Unfortunately, such rules are not always followed resulting in the presence of bad test code smells (or simply test smells). Test smells have been defined as poorly designed tests and their presence may negatively affect the maintainability of test suites and production code. Despite the many studies that address code smells in general, until now there has been no empirical evidence regarding test smells (i) distribution in software systems nor (ii) their impact on the maintainability of software systems. This paper fills this gap by presenting two empirical studies. The first study is an exploratory analysis of 18 software systems (two industrial and 16 open source) aimed at analyzing the distribution of test smells in source code. The second study, a controlled experiment involving twenty master students, is aimed at analyzing whether the presence of test smells affects the comprehension of source code during software maintenance. The results show that (i) test smells are widely spread throughout the software systems studied and (ii) most of the test smells have a strong negative impact on the comprehensibility of test suites and production code.

Program comprehension is one of the most frequently performed activities during software maintenance and evolution. In order to facilitate program comprehension, a variety of graphical models have been proposed in software engineering community to construct relationships between program elements. These graphical models are mostly used for understanding the system based on structural syntax dependencies between program elements. However, these graphical models fail to extract the functional or semantic features of the system. Thus, developers still cannot effectively identify the functional part in source code fit for their needs. This paper tries to fill this gap, and proposes a novel representation, program feature network (PFN), to identify the semantic features of the program at class level. PFN is generated based on the relational topic model, a hierarchical probabilistic model of networks. Based on PFN, the semantic features and the links between pairs of two classes in the program can be clearly shown. In addition, PFN can predict the possible links between the newly change request in existing program feature network rather than reconstructing the representation from the start.

Program comprehension is an important activity in software maintenance, as software must be sufficiently understood before it can be properly modified. The study of a program's execution, known as dynamic analysis, has become a common technique in this respect and has received substantial attention from the research community, particularly over the last decade. These efforts have resulted in a large research body of which currently there exists no comprehensive overview. This paper reports on a systematic literature survey aimed at the identification and structuring of research on program comprehension through dynamic analysis. From a research body consisting of 4,795 articles published in 14 relevant venues between July 1999 and June 2008 and the references therein, we have systematically selected 176 articles and characterized them in terms of four main facets: activity, target, method, and evaluation. The resulting overview offers insight in what constitutes the main contributions of the field, supports the task of identifying gaps and opportunities, and has motivated our discussion of several important research directions that merit additional consideration in the near future.

Due to wide adoption of object-oriented programming in software development, there is always a requirement to produce well-designed software systems, so that the overall software maintenance cost is reduced and reusability of the component is increased. But, due to prolonged maintenance activities, the internal structure of software system deteriorates. In this situation, restructuring is a widely used solution to improve the overall internal structure of the system without changing its external behavior. As, it is known that, one technique to perform restructuring is to use refactoring on the existing source code to alter its internal structure without modifying its external functionality. However, the refactoring solely depends on our ability to identify various code smells present in the system. Refactoring aims at improving cohesion and reducing coupling in the software system. So, in this paper, a restructuring approach based on refactoring is proposed through improvement in cohesion. This paper focuses on improving the cohesion of different classes of object-oriented software using a newly proposed similarity metric based on Frequent Usage Patterns (FUP). The proposed similarity metric measure the relatedness among member functions of the classes. The metric makes use of FUPs used by member functions. The FUP consists of unordered sequences of member variables accessed by member function in performing its task. The usage pattern includes both direct and indirect usages based on sub-function calls within a member function. Based on the values of the similarity metric, we performed hierarchical agglomerative clustering using complete linkage strategy to cluster member functions. Finally, based on the clusters obtained, the source code of the software is refactored using proposed refactoring algorithm. The applicability of our proposed approach is tested using two java projects related to different domains of real life. The result obtained encourages the applicability of proposed approach in the restructuring of a software system.patterns, refactoring, hierarchical clustering, maintainability. usage patterns, refactoring, hierarchical clustering, maintainability.

LDA (Latent Dirichlet Allocation) topic model has been applied into many applications in recent years. But LDA has a shortcoming that it cannot deal with various changes of data set well, which has become a limitation for its applications. Hierarchical Latent Dirichlet Allocation (hLDA) is a generalization of LDA and it can adapt itself to the growing data set automatically. hLDA can mine latent topics from a large amount of discrete data and organize these topics into a hierarchy, in which the topics of higher level are more abstractive while the topics of lower level are more specific. This hierarchy could achieve a deeper semantic model which is similar with human mind. Given a set of documents, hLDA generates a prior distribution of Bayesian nonparametrics using a nested Chinese restaurant process (nCRP)[1]. The documents sharing similar topics are organized into a cluster of path. hLDA learns the distribution of topics using a method of Bayesian posterior inference. This paper tries to study hLDA model in details and apply it into the application of Chinese text clustering. Experiments have shown that hLDA is a very promising model for text clustering.

Pengembangan perangkat lunak adalah tugas kompleks dan membutuhkan adaptasi untuk mengakomodasi kebutuhan pengguna. Untuk membuat konsep dan perubahan perangkat lunak, dalam pemeliharaan, sekarang telah dikembangkan lebih mudah dalam pengembangan perangkat lunak, pola model-view-controller, yang merupakan arsitektur yang dapat membantu memfasilitasi dalam pengembangan dan pemeliharaan perangkat luna. Hal ini, karena dalam arsitektur model tiga-lapis, yaitu: tampilan dan pengontrolan dalam pembangunan dilakukan secara independen, sehingga dapat memberikan dahan dalam pengembangan dan pemeliharaan. Selain itu, arsitektur ini juga dapat melihat hal-hal yang sederhana dan menarik bagi pengguna. Software sistem on-line test adalah perangkat lunak yang memerlukan interaksi dengan pengguna, dan pemeliharaan perangkat adaptif. Karena sistem ujian on-line memerlukan pengembangan perangkat lunak untuk mengakomodasi kebutuhan ini berkembang dengan cepat. Makalah ini untuk menganalisis Model-View-Controller dan mencoba pembangunan, untuk menerapkannya dalam pengembangan perangkat lunak sistem pengujian on-line.

Research on multimodal data analysis such as annotated image analysis is becoming more important than ever due to the increase in the amount of data. One of the approaches to this problem is multimodal topic models as an extension of latent Dirichlet allocation (LDA). Symmetric correspondence topic models (SymCorrLDA) are state-of-the-art multimodal topic models that can appropriately model multimodal data considering inter-modal dependencies. Incidentally, hierarchically structured categories can help users find relevant data from a large amount of data collection. Hierarchical topic models such as hierarchical latent Dirichlet allocation (hLDA) can discover a tree-structured hierarchy of latent topics from a given unimodal data collection; however, no hierarchical topic models can appropriately handle multimodal data considering intermodal mutual dependencies. In this paper, we propose h-SymCorrLDA to discover latent topic hierarchies from multimodal data by combining the ideas of the two previously mentioned models: multimodal topic models and hierarchical topic models. We demonstrate the effectiveness of our model compared with several baseline models through experiments with two datasets of annotated images.