Big Code Search: A Bibliography

Source code search tools are designed to help developers locate code relevant to their task. The effectiveness of a search technique often depends on properties of user queries, the code being searched, and the specific task at hand. Thus, new code search techniques should ideally be evaluated in realistic situations that closely reflect the complexity, purpose of use, and context encountered during actual search sessions. This paper explores what can be learned from using an online paired interleaving approach, originally used for evaluating internet search engines, to comparatively observe and assess the effectiveness of code search tools in the field. We present a case study in which we implemented online paired interleaving for code search, deployed the tool in an IDE for developers at multiple companies, and analyzed results from over 300 user queries during their daily software maintenance tasks. We leveraged the results to direct further improvement of a search technique, redeployed the tool and analyzed results from over 600 queries to validate that an improvement in search was achieved in the field. We also report on the characteristics of user queries collected during the study, which are significantly different than queries currently used in evaluations.

Over the years of software development, a vast amount of source code has been accumulated. Many code search tools were proposed to help programmers reuse previously-written code by performing free-text queries over a large-scale codebase. Our experience shows that the accuracy of these code search tools are often unsatisfactory. One major reason is that existing tools lack of query understanding ability. In this paper, we propose CodeHow, a code search technique that can recognize potential APIs a user query refers to. Having understood the potentially relevant APIs, CodeHow expands the query with the APIs and performs code retrieval by applying the Extended Boolean model, which considers the impact of both text similarity and potential APIs on code search. We deploy the backend of CodeHow as a Microsoft Azure service and implement the front-end as a Visual Studio extension. We evaluate CodeHow on a large-scale codebase consisting of 26K C# projects downloaded from GitHub. The experimental results show that when the top 1 results are inspected, CodeHow achieves a precision score of 0.794 (i.e., 79.4% of the first returned results are relevant code snippets). The results also show that CodeHow outperforms conventional code search tools. Furthermore, we perform a controlled experiment and a survey of Microsoft developers. The results confirm the usefulness and effectiveness of CodeHow in programming practices.

The software development process is characterized by an iterative cycle of continuous functionality implementation and debugging, essential for the enhancement of software quality and adaptability to changing requirements. This process incorporates two isolatedly studied tasks: Code Search (CS), which retrieves reference code from a code corpus to aid in code implementation, and Fault Localization (FL), which identifies code entities responsible for bugs within the software project to boost software debugging. The basic observation of this study is that these two tasks exhibit similarities since they both address search problems. Notably, CS techniques have demonstrated greater effectiveness than FL ones, possibly because of the precise semantic details of the required code offered by natural language queries, which are not readily accessible to FL methods. Drawing inspiration from this, we hypothesize that a fault localizer could achieve greater proficiency if semantic information about the buggy methods were made available. Based on this idea, we propose \(\mathtt{CosFL}\) , an FL approach that decomposes the FL task into two steps: query generation , which describes the functionality of the problematic code in natural language, and fault retrieval , which uses CS to find program elements semantically related to the query, allowing for finishing the FL task from a CS perspective. Specifically, to depict the buggy functionalities and generate high-quality queries, \(\mathtt{CosFL}\) extensively harnesses the code analysis, semantic comprehension, text generation, and decision-making capabilities of LLMs. Moreover, to enhance the accuracy of CS, \(\mathtt{CosFL}\) captures varying levels of context information and employs a multi-granularity code search strategy, which facilitates a more precise identification of buggy methods from a holistic view. The evaluation on 835 real bugs from 23 Java projects shows that \(\mathtt{CosFL}\) successfully localizes 324 bugs within Top-1, which significantly outperforms the state-of-the-art approaches by 26.6%-57.3%. The ablation study and sensitivity analysis further validate the importance of different components and the robustness of \(\mathtt{CosFL}\) across different backend models.

Modern Q&A websites, such as Stack Overflow, use social media to provide concise answers, and offer rich technical context with quality assessment capabilities. Although some of the answers may include executable code snippets, they are entangled in free text and are not easily extracted. Q&A websites are not designed for such direct code reuse. We present Example Overflow, a code search and recommendation tool which brings together social media and code recommendation systems. Example Overflow enables crowd-sourced software development by utilizing both textual and social information, which accompany source code on the Web. Its browsing mechanism minimizes the context switch associated with other code search tools. In this paper we describe the development of the tool, provide preliminary evaluation, and discuss its contribution to an example centric programming paradigm.

Despite the extensive adoption of crowdsourcing for the timely, cost-effective, and high-quality completion of software development tasks, a large number of crowdsourced challenges are not able to acquire a winning solution, on time, and within the desired cost and quality thresholds. A possible reason for this is that we currently lack a systematic approach that would aid software managers during the process of designing software development tasks that will be crowdsourced. This paper attempts to extend the current knowledge on designing crowdsourced software development tasks, by empirically answering the following management questions: (a) what type of projects should be crowdsourced; (b) why should one crowdsource-in terms of acquired benefits; (c) where should one crowdsource-in terms of application domain; (d) when to crowdsource-referring to the time period of the year; (e) who will win or participate in the contest; and (f) how to crowdsource (define contest duration, prize, type of contest etc.) to acquire the maximum benefits-depending on the goal of crowdsourcing. To answer the aforementioned questions, we have performed a case study on 2,209 software development tasks crowdsourced through TopCoder platform. The results suggest that there are significant differences in the level to which crowdsourcing goals are reached, across different software development activities. Based on this observation we suggest that software managers should prioritize the goals of crowdsourcing, decide carefully upon the activity to be crowdsourced and then define the settings of the task.

Code search techniques are well-known as one of the techniques that helps code reuse. If developers input queries that represent functionality that they want, the techniques suggest code fragments that are related to the query. Generally, code search techniques suggest code at the component level of programming language such as class or file. Due to this, developers occasionally need to search necessary code in the suggested area. As a countermeasure, there is a code search technique where code is suggested based on the past reuse. The technique ignores structural code blocks, so that developers need to add some code to the pasted code or remove some code from it. That is, the advantages and disadvantages of the former technique are disadvantages and advantages of the latter one, respectively. In this research, we have conducted a comparative study to reveal which level of code suggestion is more useful for code reuse. In the study, we also compared a hybrid technique of the two techniques with them. As a result, we revealed that component-level suggestions were able to provide reusable code more precisely. On the other hand, reuse-level suggestions were more helpful to reuse larger code.

Efficient code search techniques are crucial in accelerating software development by aiding developers in locating specific code snippets and understanding code functionalities. This study investigates code search methodologies, focusing on the emerging significance of semantic consistency in data augmentation techniques. While existing approaches predominantly enhance raw data, often requiring additional preprocessing and incurring higher training costs, this research introduces a pioneering method operating at the code and query representation levels. By bypassing the need for extensive data processing, this novel approach fosters an interactive alignment between code and query, augmenting the semantic coherence crucial for effective code search. An extensive empirical evaluation of a diverse dataset across multiple programming languages substantiates the efficacy of this approach in significantly enhancing code search model performance compared to traditional methodologies. The implementation is publicly available on GitHub, 1 offering an accessible resource for further exploration and application.

Most code search tools seem to yield semantically correct matches, but the search results rarely meet the demands of users perfectly. These results still have to be modified manually. One major reason is that existing tools lack the ability of intent predicting to guess what else a user might do after obtaining the results. In this paper, we propose an intent-enforced code search approach (IECS) that can predict the potential intents for a query before performing code retrieval. Then it expands the query with the intents and applies the Extended Boolean Model to retrieve the relevant results without any subsequent modification. We implement SnippetGen, a code search tool performing IECS. Compared with CodeHow and Google Code Search (CS), SnippetGen outperforms them by 28.5% with a precision score of 0.846 (i.e., 84.6% of the first returned results are relevant results) when we utilize these tools to perform 70 queries on a codebase consisting of 27K projects downloaded from GitHub. We also perform a controlled experiment by asking 20 participants to complete 3 tasks with SnippetGen and CodeHow. The results confirm the effectiveness of SnippetGen in programming practice.

Searching and reusing online code is a common activity in software development. Meanwhile, like many general-purposed searches, code search also faces the session search problem: in a code search session, the user needs to iteratively search for code snippets, exploring new code snippets that meet his/her needs and/or making some results highly ranked. This paper presents Cosoch, a reinforcement learning approach to session search of code documents (code snippets with textual explanations). Cosoch is aimed at generating a session that reveals user intentions, and correspondingly searching and reranking the resulting documents. More specifically, Cosoch casts a code search session into a Markov decision process, in which rewards measuring the relevances between the queries and the resulting code documents guide the whole session search. We have built a dataset, say CosoBe, from StackOverflow, containing 103 code search sessions with 378 pieces of user feedback. We have also evaluated Cosoch on CosoBe. The evaluation results show that Cosoch achieves an average NDCG@3 score of 0.7379, outperforming StackOverflow by 21.3%.

Code search is a fundamental part of program understanding and software maintenance and thus researchers have developed many techniques to improve its performance, such as corpora preprocessing and query reformulation. Unfortunately, to date, evaluations of code search techniques have largely been in lab settings, while scaling and transitioning to effective practical use demands more empirical feedback from the field. This paper addresses that need by studying metrics based on automatically-gathered anonymous field data from code searches to infer user satisfaction. We describe techniques for addressing important concerns, such as how privacy is retained and how the overhead on the interactive system is minimized. We perform controlled user and field studies which identify metrics that correlate with user satisfaction, enabling the future evaluation of search tools through anonymous usage data. In comparing our metrics to similar metrics used in Internet search we observe differences in the relationship of some of the metrics to user satisfaction. As we further explore the data, we also present a predictive multi-metric model that achieves accuracy of over 70% in determining query satisfaction.

Code search uses natural language queries to retrieve code snippets from a vast database, identifying those that are semantically similar to the query. This enables developers to reuse code and enhance software development efficiency. Most existing code search algorithms focus on capturing semantic and structural features by learning from both text and code graph structures. However, these algorithms often struggle to capture deeper semantic and structural features within these sources, leading to lower accuracy in code search results. To address this issue, this paper proposes a novel semantics-enhanced neural code search algorithm called SENCS, which employs graph serialization and a two-stage attention mechanism. First, the code program dependency graph is transformed into a unique serialized encoding, and a bidirectional long short-term memory (LSTM) model is used to learn the structural information of the code in the graph sequence to generate code vectors rich in structural features. Second, a two-stage attention mechanism enhances the embedded vectors by assigning different weight information to various code features during the code feature fusion phase, capturing significant feature information from different code feature sequences, resulting in code vectors rich in semantic and structural information. To validate the performance of the proposed code search algorithm, extensive experiments were conducted on two widely used code search datasets, CodeSearchNet and JavaNet. The experimental results show that the proposed SENCS algorithm improves the average code search accuracy metrics by 8.30 % (MRR) and 17.85% (DCG) and compared to the best baseline code search model in the literature, with an average improvement of 14.86% in the SR@1 metric. Experiments with two open-source datasets demonstrate SENCS achieves a better search effect than state of-the-art models.

MARVEL is a kernel for software development environments that enacts a processing modeling notation, called the MARVEL Strategy Language (MSL). The software process is defined in MSL as a collection of modular strategies, where different (sets of these) strategies are appropriate for different tool suites, user roles, phases of the lifecycle, management policies, etc. Each strategy specifies a partial object-oriented data model and a partial rule-based process model. The strategies are supplemented by a collection of envelopes, each of which describes how to apply one or more of the external tools used during software development activities; this includes the mapping from the inputs/outputs of these tools and the MARVEL objectbase and the command lines to invoke these tools. Multiple MSL strategies are merged to obtain the full data and process models for enaction by the MARVEL kernel.Each software development activity is described in MSL by a rule specifying the precondition and multiple postconditions of the activity. Both preconditions and postconditions are written in a subset of the first order predicate logic, stylized to incorporate directives for efficient queries and updates. A precondition is a logical statement that must be true before the activity is initiated; a postcondition is a logical statement asserted to be true when the activity completes. Multiple mutually exclusive postconditions reflect the multiple possible results of many activities. An activity is typically a particular combination of software development tool, options specific to that tool, and typed arguments. The notion of activity could be extended to include loading a new set of strategies to take over at this point in the process, to support both nesting of hierarchical activities and successive activity phases.Enaction of an MSL process model is implemented by what we call opportunistic processing, consisting primarily of forward and backward chaining among the rules. When a user initiates an activity, MARVEL applies backward chaining to attempt to satisfy its precondition. If this is not possible, MARVEL could use the rules in the style of an expert system to explain the problem. Once an activity has completed, MARVEL asserts one of its postconditions and uses forward chaining to automatically carry out other activities whose results are presumed to be needed in the near future.We have designed two facilities for controlling this processing, to prevent runaway automation. Hints are additional MSL preconditions that affect automatic but not user-initiated invocation of tools. Implicit queries represent a reflective capacity to keep tabs on estimated costs of automatic behavior and request user confirmation when certain user-selected thresholds are reached.MARVEL version 2.10 was recently completed, an upgrade over the version 2.01 that was demonstrated at the 3rd PSDE conference in November 1988. This latest version of MARVEL adds an X11 windows graphical user interface and new subsystems called Marvelizer and Organ. Marvelizer is a tool for immigrating existing software systems from the Unix file system or (a restricted class of) existing software development environment into a MARVEL objectbase. Organ is a set of commands for reorganizing MARVEL objectbases as the project and/or system structure evolves. Marvelizer has been used to immigrate Marvel itself, including about 35,000 lines of C code, into a C/Marvel objectbase. We would like to use the C/Marvel set of MSL strategies, which support activities using the user's favorite editor, lint, cc, ar, ranlib, less (and soon rcs, dbx and mail), for our own further development of MARVEL. Both MARVEL and C/Marvel were in use during the Spring 1989 semester by students in my E6123y Programming Environments and Software Tools course (admittedly a captive audience).We have discovered several difficulties with the MARVEL paradigm that seem likely to apply at least partially to enaction of other notations for software process modeling. First, constructing and debugging a process model is hard. We have run into many unanticipated interactions among rules, particularly among those defined in different strategies that happen to be loaded into the MARVEL kernel at the same time. We are working on debugging facilities that will uncover all direct and transitive dependencies by constructing a graphical dataflow graph of the currently loaded rules. It is feasible for a user to enable and disable individual rules, both during debugging to see how that affects the dataflow among rules and also during operation because certain behavior is not currently desired. This would be dangerous, however, if done by a naive user: if rules were turned off and on at arbitrary points, it would be possible for MARVEL to get into what is an inconsistent state with respect to the full set of rules in such a way that automatic recovery is impossible.Similar difficulties seem likely to arise for any enactable process modeling paradigm. The following questions need to be answered: How does an implementor go about writing, testing and debugging a process model? For non-trivial processes, the model is likely to be very large and hence should be modularized. What is the appropriate style of interface among process model modules and how is it enforced? Is the process model visible to the users and, if so, how is it presented? If the users can modify the process model during enaction, how does this affect continuing the process? Should (can) consistency be enforced or should (can) inconsistencies be detected and repaired?Second, constructing envelopes is even harder. We currently use the Shell languages (e.g., CShell) as notations for defining envelopes. Writing the Shell scripts requires an intimate knowledge of both the tool and the mapping of the MARVEL objectbase onto the file system. Although there is no way to avoid the former, we hope to minimize the latter by developing a higher level envelope language that understands the MARVEL objectbase implementation so the envelope writer doesn't have to. However, this is currently a relatively low priority due to lack of manpower.Again, similar difficulties seem inevitable for other enaction mechanisms intended to work with external tools. The questions: Is it possible (or desirable) to support existing tools, particularly tools where retrofitting is impractical? If only new tools are supported, can a relatively simple standard interface be developed and published so these tools can be developed separately by vendors? Would such an interface pre-empt too many design decisions, or would it be feasible for vendors to upgrade their current products?Third, supporting multiple users is hardest of all. The current MARVEL kernel does not provide any facilities for concurrency control. It would be easy to add locking of the entire objectbase, but a more flexible transaction model is needed. We are working on a number of alternative models that we think are promising for incorporation into the MARVEL kernel. Commit-serializability introduces two new transaction operations, split-transaction and join-transaction, that manipulate in-progress transactions to support coordination among multiple users. Participant transactions (AKA participation domains) distinguishes participants in group software development tasks from observers, and relaxes serializability as viewed by participants to permit cooperation.It is essential for enactable process models to support coordination and cooperation among the members of software development teams. There are two sets of questions: First, what are appropriate processes for programming-in-the-many? This is itself an active area of research, and there is certainly no consensus. One opinion, voiced at the previous workshop, is that the community should not be enacting process models until we understand the processes we are modeling. An opposing opinion is that enaction gives the community the capability to experiment with different processes without prematurely making a full scale commitment to a particular process. Assuming agreement with the second view, what process modeling notations and enaction mechanisms can support a reasonable range of cooperative work facilities?The database and operating system communities have agreed on atomic and serializable transactions, which can be implemented using a wide variety of technologies, as the basis for data consistency, failure recovery and concurrency control across their traditional application areas. Can the software engineering community develop an equally all-encompassing “transaction” model? Or is there some characteristic of the software development process such that coordination and cooperation policies will always require special support on a case-by-case basis? If so, what does this say about enaction of process models in general?In summary, MARVEL is implemented and in limited use, it works in some cases, it doesn't work in others, it raises a lot of questions, and both we the MARVEL group and we the software engineering community have a lot more work to do.

Leveraging the abundant community resources, e.g. campus students, programming fans, software amateurs, is promising to be a flexible and effective approach for relieving the enterprise's IT resource shortage. However, the concern about confidential knowledge exposure makes most enterprises hesitate to outsource software development to community resources. Unlike the traditional legal protection mechanism that prevents community resources from exposing the already known confidential knowledge, this paper presents a proactive confidential knowledge protection technology, which prevents the community resources from learning the confidential knowledge in software development activities. First, the client is facilitated on identifying confidential knowledge involved in the software development activity. Then, the risky software design elements that might expose confidential knowledge are identified in both qualitative and quantitative way. Based on the identification result, a holistic proactive protection solution is presented by orchestrating task reservation, task pretending and task partition activities. Finally, the processed software development tasks are dispatched to community developers without worrying about the confidential knowledge exposure. The technology has been preliminarily validated in two community-based software development cases.

Software development is a collaborative process in which programmers build software by integrating all the stages of the software development life cycle (SDLC). A software repository is a central file storage location where various software packages are stored, and these packages are retrieved and shared between all the software development team members at various locations. The software repositories are divided into various categories based on cooperation, coordination, and communication among the stakeholders as well as evolutionary changes to various software arti-facts such as source code repositories, software bug repositories, historical repositories, run-time repositories and requirement documents, and other documentation. The software bug repository is an essential repository among the entire repositories since the completion of the software is entirely dependent on the bug fixing mechanism associated with this repository in software development. Today's software systems are larger and more complex as they go through various stages from the requirement 98analysis phase to the maintenance phase. A variety of tasks and activities are carried out in each stage of software development, and these are expensive and vulnerable to errors. During software development, a large number of software bugs are continuously generated, and that has become the main reason for the delay in software completion. Hence, there is a vast demand for computational intelligent techniques to accomplish various tasks of software development. In recent years, fuzzy logic techniques emerged and played an important role in various fields of data mining and text mining. Since most of the content related to software bug repositories is text in nature, it is possible to effectively use fuzzy logic techniques to analyze these software bugs.

Program comprehension is a required activity for many software development and maintenance tasks. A common means of understanding software is though UML diagrams. UML diagrams model the design and implementation of an application, describing both its structure and behaviour. However with rapid software development life cycles, UML diagrams often become inconsistent with the implementation of an application. This limits their effectiveness in program comprehension activities. This paper presents a program analysis framework to reverse engineer sequence diagrams from application source code. Part of this framework is the ability to filter out irrelevant operations from a sequence diagram in order to simplify the representation of an application. This is achieved by identifying important data, and following its use in the application. The purpose of this framework is to aid in program comprehension activities by providing up to date representations of an application; while also enabling developers to identify the logical operation of a program without interference from irrelevant or supporting operations.