Labeling source code with information retrieval methods: an empirical study

This special issue of EMSE comprises four selected papers on program comprehension, an unusually wide-ranging research area within software engineering. It encompasses the human activities of comprehending software, the technologies for supporting the program comprehension, and their evaluation. It is notable for combining aspects of cognitive psychology, tool design, and empirical study of users, tools, and software artifacts. In “High-MCC Functions in the Linux Kernel”, the authors challenge the assumption that functions with high values of McCabe Cyclomatic Complexity (MCC) are likely to be problematic to understand and maintain. They explore the nuances of what a high MCC value might mean, and examine functions from the Linux kernel with high MCC values. They find that often the structure of these functions is found to be easy to understand by users, leading to the conclusion that syntactic measures alone may not be a good metric for understandability. In “Measuring and Modeling Programming Experience”, the authors describe a controlled experiment in assessing how the experience of software developers can best be measured. This is a particularly important problem in program comprehension research, as developer experience is often a key factor that must be explicitly taken into account when studying how programmers perform problem-solving tasks. In “An Empirical Study on the Impact of Static Typing on Software Maintainability”, the authors tackle one of the classic programming language debates: How does static typing affect human software development capabilities? The authors approach this question from an empirical perspective. In a carefully designed experiment, the authors assess the impact of static typing on understanding undocumented code, fixing type errors, and fixing semantic errors. The results show rigorous empirical evidence that static types are indeed beneficial to these activities, except when fixing semantic errors. In “Labeling Source Code with Information Retrieval Methods: An Empirical Study”, the authors address augmenting source code with explanatory labels. Automated techniques for this exist, which are based on, e.g., Vector Space Models, Latent Semantic Indexing (LSI), and latent Dirichlet allocation (LDA). The authors describe an experiment in which such Empir Software Eng (2014) 19:1259–1260 DOI 10.1007/s10664-014-9329-5

Topic modeling is the statistical model for discovering hidden topics or keywords in a collection of documents. Topic modeling is also considered a probabilistic model for learning, analyzing, and discovering topics from the document collection. The most popular techniques for topic modeling are latent semantic analysis (LSA), probabilistic latent semantic analysis (pLSA), latent Dirichlet allocation (LDA), and the recent deep learning-based lda2vec. LDA is most commonly used in extractive multi-document summarization to determine whether the extracted sentence reflects the concept of the input document. In this paper, we will try to explore various multi-document summarization techniques that use LDA as a topic modeling method for improving final summary coverage and to reduce redundancy. Finally, we compared LDA and LSA using the Genism toolkit, and our experiment results show that LDA outperforms LSA if we increase the number of features considered for sentence selection.KeywordsTopic modelingText summarizationLDALSA

Funding agencies such as the U.S. National Science Foundation (NSF), U.S. National Institutes of Health (NIH), and the Transportation Research Board (TRB) of The National Academies make their online grant databases publicly available which document a variety of information on grants that have been funded over the past few decades. In this paper, based on a quantitative analysis of the TRB’s Research In Progress (RIP) online database, we explore the feasibility of automatically estimating the appropriate funding level, given the textual description of a transportation research project. We use statistical Text Mining (TM) and Machine Learning (ML) technologies to build this model using the 14,000 or more records of the TRB’s RIP research grants big data. Several Natural Language Processing (NLP) based text representation models such as the Latent Dirichlet Allocation (LDA), Latent Semantic Indexing (LSI) and the Doc2Vec Machine Learning (ML) approach are used to vectorize the project descriptions and generate semantic vectors. Each of these representations is then used to train supervised regression models such as Random Forest (RF) regression. Out of the three latent feature generation models, we found LDA gives the least Mean Absolute Error (MAE) using 300 feature dimensions and RF regression model. However, based on the correlation coefficients, it was found that it is not very feasible to accurately predict the funding level directly from the unstructured project abstract, given the large variations in source agencies, subject areas, and funding levels. By using separate prediction models for different types of funding agencies, funding levels were better correlated with the project abstract.

A questionnaire text mining model is built based on latent semantic analysis (LSA). Take open questionnaire as the research object and improve Latent Dirichlet Allocation (LDA) model according to the text characteristics, a BM25-LDA model is built. By comparing the LDA model and BM25-LDA model on the open questionnaire text, we can verify that the performance of BM25-LDA mode is better than that of original LDA model.

The demand of scoring natural language responses has created a need for new computational tools that can be applied to automatically grade student essays. Systems for automatic essay assessment have been commercially available since 1990's. However, the progress in the field was obstructed by a lack of qualitative information regarding the effectiveness of such systems. Most of the research in automatic essay grading has been associated with English writing due to its widespread use and the availability of more learner collection and language processing software for the language. In addition, there is large number of commercial software for grading programming assignments automatically. In this work, we investigate document semantic similarity based on Latent Semantic Analysis (LSA) and on Latent Dirichlet Allocation (LDA). We use an open-source Python software, Gensim, to develop and implement an essay grading system able to compare an essay to an answer-key and assign to it a grade based on semantic similarity between the two. We test our tool on variable-size essays and conduct experimental tests to compare the results obtained from human grader (professor) and those obtained from the automatic grading system. Results show high correlation between the professor grades and the grades assigned by both modeling techniques. However, LSA-based modeling showed more promising results than the LDA-based method.

In the past decade, there have been many well-publicized cases of source code leaking from different well-known companies. These leaks pose a serious problem when the source code contains sensitive information encoded in its identifier names and comments. Unfortunately, redacting the sensitive information requires obfuscating the identifiers, which will quickly interfere with program comprehension. Program comprehension is key for programmers in understanding the source code, so sensitive information is often left unredacted. To address this problem, we offer a novel approach for REdacting Sensitive Information in Software arTifacts (RESIST). RESIST finds and replaces sensitive words in software artifacts in such a way to reduce the impact on program comprehension. We evaluated RESIST experimentally using 57 professional programmers from over a dozen different organizations. Our evaluation shows that RESIST effectively redacts software artifacts, thereby making it difficult for participants to infer sensitive information, while maintaining a desired level of comprehension.

The intelligent task of semantically assigning a paper to a reviewer with respect to his knowledge domain remains a challenging task in academic conferences. From literature, a number of automated reviewer assignment systems have been presented which are based on distributional semantic models such as Term Frequency-Inverse Document Frequency (TF-IDF), Latent Semantic Indexing (LSI) and Latent Dirichlet Allocation (LDA) have been used to capture semantics. Thus, this study presents the comparative study of the three models based on their derived suitability scores between a paper meant for review and a reviewer’s representation papers. From the experimental results obtained, it shows that TF-IDF outperformed the accuracy level of the other two models by a substantial margin.

We present Topic View, an application for visually comparing and exploring multiple models of text corpora. Topic View uses multiple linked views to visually analyze both the conceptual content and the document relationships in models generated using different algorithms. To illustrate Topic View, we apply it to models created using two standard approaches: Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). Conceptual content is compared through the combination of (i) a bipartite graph matching LSA concepts with LDA topics based on the cosine similarities of model factors and (ii) a table containing the terms for each LSA concept and LDA topic listed in decreasing order of importance. Document relationships are examined through the combination of (i) side-by-side document similarity graphs, (ii) a table listing the weights for each document's contribution to each concept/topic, and (iii) a full text reader for documents selected in either of the graphs or the table. We demonstrate the utility of Topic View's visual approach to model assessment by comparing LSA and LDA models of two example corpora.

Stack Overflow has undoubtedly become a big trend among the programmers and researchers. The problems or doubts faced by any programmer are put on this forum making it available to many knowledgeable people and hence giving out possibly correct solutions. This paper focuses on Tag based Answer Recommendation System on Stack Overflow. This model follows Topic Modeling methodology wherein two techniques were attempted, i.e., LSA(Latent semantic analysis) and LDA(Latent Dirichlet Allocation). These methods deal with various User Queries to find out the most similar questions from the database and hence, deliver the most valid answers related to the question as output. The results from both the Topic Modeling techniques were analyzed with the help of Cosine Similarity as metrics and hence, we found out that LDA performed better than LSA, thereby giving better and more accurate results.

Text representation, which is a fundamental and necessary process for text-based intelligent information processing, includes the tasks of determining the index terms for documents and producing the numeric vectors corresponding to the documents. In this paper, multi-word, which is regarded as containing more contextual semantics than individual word and possessing the favorable statistical characteristics, is proposed as an alternative index terms in vector space model for text representation with theoretical support. We investigate the traditional indexing methods as TF*IDF (term frequency inverse document frequency) and LSI (latent semantic indexing) for comparative study. The performances of TF*IDF, LSI and multi-word are examined on the tasks of text classification, which includes information retrieval (IR) and text categorization (TC), in Chinese and English document collection respectively. We also attempt to tune the rescaling factor of LSI and observe its effectiveness in text classification. The experimental results demonstrate that TF*IDF and multi-word are comparable when they are used for IR and TC and LSI is the poorest one of them. Moreover, the rescaling factor of LSI has an insignificant influence on its effectiveness on text classification for both Chinese and English text classification.

The literature reports that source code lexicon plays a paramount role in program comprehension, especially when software documentation is scarce, outdated or simply not available. In source code, a significant proportion of vocabulary can be either acronyms and-or abbreviations or concatenation of terms that can not be identified using consistent mechanisms such as naming conventions. It is, therefore, essential to disambiguate concepts conveyed by identifiers to support program comprehension and reap the full benefit of Information Retrieval-based techniques (e.g., feature location and traceability) whose linguistic information (i.e., source code identifiers and comments) used across all software artifacts (e.g., requirements, design, change requests, tests, and source code) must be consistent. To this aim, we propose source code vocabulary normalization approaches that exploit contextual information to align the vocabulary found in the source code with that found in other software artifacts. We were inspired in the choice of context levels by prior works and by our findings. Normalization consists of two tasks: splitting and expansion of source code identifiers. We also investigate the effect of source code vocabulary normalization approaches on software maintenance tasks. Results of our evaluation show that our contextual-aware techniques are accurate and efficient in terms of computation time than state of the art alternatives. In addition, our findings reveal that feature location techniques can benefit from vocabulary normalization approaches when no dynamic information is available.

Topmodpy: A Simple Python Script for Topic Modeling

Twitter is a popular platform for people to express their feelings on any subject or topic irrespective of place and time in the world. The view expressed by the Twitter community gives enormous information about them and the trend going on. To identify these trends and patterns, many data science techniques are used. However, using appropriate clustering technique always remains a challenge for researchers. The research concentrates on using hard clustering approaches like K-means and soft clustering approaches like Latent Dirichlet Allocation (LDA) and Latent Semantic Indexing (LSI). The proposed methodology uses K-means on numerical attributes, LDA, and LSI on textual attributes. The experiments are done using different Twitter datasets and tested on Sabarimala temple tweet dataset. The first time experimental study shows the promising correlation of the K-means cluster sentiments with topic sentiments. This helps in understanding the stance of the topics formed. The paper concludes by highlighting the relevance between the results of K-means clusters with the topics formed using LDA and LSI techniques.

People sourcing for a particular job role in any corporate venture is a painstaking, progress-impeding task, especially given the changing job market trends, quality of candidate profile and the sheer number of applicants. This paper elucidates an improvised topic modelling approach, where the semantic analysis stage is prepended with text summarization, for the procedure of people sourcing for job roles using topic modelling and machine learning techniques. The distinction between choosing the perfect candidate for the job and choosing a good candidate who is adept in certain domains but not relevant for the job and getting the candidate up to speed by providing on job training, and the downtime involved in the process is certainly a deciding factor in hiring a potential employee. The following paper aims to alleviate this issue by describing the algorithm which identifies the most suitable candidate in the applicants’ pool through a novel, robust approach which uses topic modelling techniques like Latent Semantic Indexing (LSA) and Latent Dirichlet Allocation (LDA). It is empirically found that the precision of profile matching task was enhanced by using text summarization (through TextRank model) for both LSA and LDA by 21.4% and 50% respectively, and LSA outperforming LDA with regard to precision, in both with and without text summarization cases by 23.8% and 51.57%, respectively.

Topic-based language model has attracted much attention as the propounding of semantic retrieval in recent years. Especially for the ASR text with errors, the topic representation is more reasonable than the exact term representation. Among these models, Latent Dirichlet Allocation(LDA) has been noted for its ability to discover the latent topic structure, and is broadly applied in many text-related tasks. But up to now its application in information retrieval(IR) is still limited to be a supplement to the standard document models, and furthermore, it has been pointed out that directly employing the basic LDA model will hurt retrieval performance. In this paper, we propose a lexicon-guided two-level LDA retrieval framework. It uses the HowNet to guide the first-level LDA model's parameter estimation, and further construct the second-level LDA models based on the first-level's inference results. We use TRECID 2005 ASR collection to evaluate it, and compare it with the vector space model(VSM) and latent semantic Indexing(LSI). Our experiments show the proposed method is very competitive.