COSPEX: A Program Comprehension Tool for Novice Programmers

Software development practices rely extensively on reusing source code written by other programmers. One of the recurring questions about such practice is how much programmers, acting as users of somebody else’s code, really understand the source code that they inject it in their programs. The question is even more important for novices, who are trying to learn what programming is and how it should be practiced on a larger scale. In this paper we present the results of an ongoing research using a semiotic approach to investigate how novice programmers reuse source code, and how, through messages inscribed in the source code of the programs they write or reuse, they communicate, implicitly or explicitly, what such source code "means" to them and others. We carried out three studies with novice programmers, and results suggest that source code reuse may impact what programmers take their source code to mean.

This working group asserts that Program Comprehension (ProgComp) plays a critical part in the process of writing programs. For example, this paper is written from a basic draft that was edited and revised until it clearly presented our idea. Similarly, a program is written incrementally, with each step tested, debugged and extended until the program achieves its goal. Novice programmers should develop program comprehension skills as they learn to code so that they are able both to read and reason about code created by others, and to reflect on their code when writing, debugging or extending it. To foster such competencies our group identified two main goals: (g1) to collect and define learning activities that explicitly address key components of program comprehension and (g2) to define tentative theoretical learning trajectories that will guide teachers as they select and sequence those learning activities in their CS0/CS1/CS2 or K-12 courses.The WG has completed the first goal and laid down a strong foundation towards the second goal as presented in this report. After a thorough literature review, a detailed description of the Block Model is provided, as this model has been used with a dual purpose, to classify and present an extensive list of ProgComp tasks, and to describe a possible learning trajectory for a complex task, covering different cells of the Block Model matrix. The latter is intended to help instructors to decompose complex tasks and identify which aspects of ProgComp are being fostered.

The purpose of this research is to examine the influence of different methods of program navigation on the mental representation and comprehension of novice procedural programmers. As a programmer tries to comprehend a program, a particular navigation method may assist or inhibit the process by highlighting, or making more accessible, certain kinds of information. Presumably, a method of navigation that highlights a certain type of information will help the programmer to better comprehend that information. In this research we study the effect of sequential, control flow, and data flow navigation methods on novices programmers' overall comprehension of a program and on the ability to comprehend specific types of information. Our results indicate that novice comprehension is facilitated by a sequential or control flow view of the program and is inhibited by a data flow view.

Unit testing represents a key activity in software development and maintenance. Test suites with high internal quality facilitate maintenance activities, such as code comprehension and regression testing. Several guidelines have been proposed to help developers write good test suites. Unfortunately, such rules are not always followed resulting in the presence of bad test code smells (or simply test smells). Test smells have been defined as poorly designed tests and their presence may negatively affect the maintainability of test suites and production code. Despite the many studies that address code smells in general, until now there has been no empirical evidence regarding test smells (i) distribution in software systems nor (ii) their impact on the maintainability of software systems. This paper fills this gap by presenting two empirical studies. The first study is an exploratory analysis of 18 software systems (two industrial and 16 open source) aimed at analyzing the distribution of test smells in source code. The second study, a controlled experiment involving twenty master students, is aimed at analyzing whether the presence of test smells affects the comprehension of source code during software maintenance. The results show that (i) test smells are widely spread throughout the software systems studied and (ii) most of the test smells have a strong negative impact on the comprehensibility of test suites and production code.

Comprehension of computer programs involves identifying important program parts and inferring relationships between them. The ability to comprehend a computer program is a skill that begins its development in the novice programmer and reaches maturity in the expert programmer. This research examined the beginning of this process, that of comprehension of computer programs by novice programmers. The mental representations of the program text that novices form, which indicate the comprehension strategies being used, were examined. In the first study, 80 novice programmers were tested on their comprehension of short program segments. The results suggested that novices form detailed, concrete mental representations of the program text, supporting work that has previously been done with novice comprehension. Their mental representations were primarily procedural in nature, with little or no modeling using real‐world referents. In a second study, the upper and lower quartile comprehenders from Study 1 were test...

Developers spend a substantial amount of their time with program comprehension. To improve their comprehension and refresh their memory, developers need to communicate with other developers, read the documentation, and analyze the source code. Many studies show that developers focus primarily on the source code and that small improvements can have a strong impact. As such, it is crucial to bring the code itself into a more comprehensible form. A particular technique for this purpose are explicit feature traces to easily identify a program’s functionalities. To improve our empirical understanding about the effects of feature traces, we report an online experiment with 49 professional software developers. We studied the impact of explicit feature traces, namely annotations and decomposition, on program comprehension and compared them to the same code without traces. Besides this experiment, we also asked our participants about their opinions in order to combine quantitative and qualitative data. Our results indicate that, as opposed to purely object-oriented code: (1) annotations can have positive effects on program comprehension; (2) decomposition can have a negative impact on bug localization; and (3) our participants perceive both techniques as beneficial. Moreover, none of the three code versions yields significant improvements on task completion time. Overall, our results indicate that lightweight traceability, such as using annotations, provides immediate benefits to developers during software development and maintenance without extensive training or tooling; and can improve current industrial practices that rely on heavyweight traceability tools (e.g., DOORS) and retroactive fulfillment of standards (e.g., ISO-26262, DO-178B).

Program comprehension is an important activity in software maintenance, as software must be sufficiently understood before it can be properly modified. The study of a program's execution, known as dynamic analysis, has become a common technique in this respect and has received substantial attention from the research community, particularly over the last decade. These efforts have resulted in a large research body of which currently there exists no comprehensive overview. This paper reports on a systematic literature survey aimed at the identification and structuring of research on program comprehension through dynamic analysis. From a research body consisting of 4,795 articles published in 14 relevant venues between July 1999 and June 2008 and the references therein, we have systematically selected 176 articles and characterized them in terms of four main facets: activity, target, method, and evaluation. The resulting overview offers insight in what constitutes the main contributions of the field, supports the task of identifying gaps and opportunities, and has motivated our discussion of several important research directions that merit additional consideration in the near future.

In the following study, we deal with data-driven support for novice programmers – students, working in large groups. We introduce a virtual assistant, which aims to simplify communication and feedback, sending reports and advice during the programming course. We experiment with 134 students during one semester period, and we try to understand how the virtual assistant affects novices – students. In the introduction, we state 3 hypotheses, focusing on performance, behavior, and motivation. The experiment evaluation relies on data collected throughout the semester. The results reveal the impact of the virtual assistant on novice programmers in 2 of the studied aspects – performance and motivation. This can be perceived as potential towards efficient support in programming education, especially in large groups. We also believe the study is contributional to the areas of learning analytics and program comprehension.

During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer’s time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers’ program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.’s approach to assign developers’ activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend $\sim$ 58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers’ experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.

The literature reports that source code lexicon plays a paramount role in program comprehension, especially when software documentation is scarce, outdated or simply not available. In source code, a significant proportion of vocabulary can be either acronyms and-or abbreviations or concatenation of terms that can not be identified using consistent mechanisms such as naming conventions. It is, therefore, essential to disambiguate concepts conveyed by identifiers to support program comprehension and reap the full benefit of Information Retrieval-based techniques (e.g., feature location and traceability) whose linguistic information (i.e., source code identifiers and comments) used across all software artifacts (e.g., requirements, design, change requests, tests, and source code) must be consistent. To this aim, we propose source code vocabulary normalization approaches that exploit contextual information to align the vocabulary found in the source code with that found in other software artifacts. We were inspired in the choice of context levels by prior works and by our findings. Normalization consists of two tasks: splitting and expansion of source code identifiers. We also investigate the effect of source code vocabulary normalization approaches on software maintenance tasks. Results of our evaluation show that our contextual-aware techniques are accurate and efficient in terms of computation time than state of the art alternatives. In addition, our findings reveal that feature location techniques can benefit from vocabulary normalization approaches when no dynamic information is available.

Source code quality is undoubtedly very important, yet usually forgotten aspect of software development process. This study shows different approaches to analysis of student codes, mainly in Java. By comparing different widely-used tools and our solution, we are able to get precise results from such an analysis, and thus to improve the code quality. The analysis results can also be used to determine knowledge and skills of novice programmers (students). Analysis of source codes may be as simple as counting the occurrences of language constructs or as complex as design pattern detection. Different approaches and solutions, i.e. measurements of language construct occurrences, copy-pasted code and design pattern detection, or source code quality, are discussed and subjected to evaluation by being executing on source codes of students as novice programmers.

In the past decade, there have been many well-publicized cases of source code leaking from different well-known companies. These leaks pose a serious problem when the source code contains sensitive information encoded in its identifier names and comments. Unfortunately, redacting the sensitive information requires obfuscating the identifiers, which will quickly interfere with program comprehension. Program comprehension is key for programmers in understanding the source code, so sensitive information is often left unredacted. To address this problem, we offer a novel approach for REdacting Sensitive Information in Software arTifacts (RESIST). RESIST finds and replaces sensitive words in software artifacts in such a way to reduce the impact on program comprehension. We evaluated RESIST experimentally using 57 professional programmers from over a dozen different organizations. Our evaluation shows that RESIST effectively redacts software artifacts, thereby making it difficult for participants to infer sensitive information, while maintaining a desired level of comprehension.

To efficiently support novice programming learners feeling programming difficult, clarifying the cause of preventing programming understanding, and developing a new teaching method appropriate for their understanding degree would be necessary. The objective of this paper is to develop a learning support system with reading source codes. This paper also aims to evaluate the effectiveness of the developed system from the subjective viewpoint of learners. The developed system can automatically generate a source code of C programming language in which there is no particular meaning because the source codes as learning materials are generated randomly. The developed system was utilized in a programming lecture for novice programming learners. This paper obtained student responses from a questionnaire, after the students had completed one semester of the instruction in programming, and analyzed the data. From the analysis result, it turned out that different evaluation patterns existed depending on the learner's basic programming skill.

During software development and maintenance, developers spend a considerable amount of time on program comprehension. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions.

The ability to interpret error messages in order to find and fix bugs is an essential skill for novice programmers. Unfortunately, the technical language of most error messages can hinder the progress of CS1 students and can lead to feelings of confusion and frustration. A potential intervention for CS educators is the use of enhanced error messages that utilise natural-language geared towards novice learners, but the current discourse in CS Education regarding benefits of such messages is inconclusive. In this paper, we describe a planned, semi-controlled experiment running parallel to a CS1 course at the University of Toronto Scarborough. The study aims to build upon existing work in quantifying the effects of enhanced error messages by incorporating additional quantitative metrics, usability surveys, student feedback, and semi-structured interviews. The additional methods serve to measure not only the effects on error recovery, but student satisfaction, sense of frustration and overall attitude towards error messages and debugging.