Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy

Since Computer Science students are used to applying both General Purpose Programming Languages (GPPLs) and Domain-Specific Languages (DSLs), Generative Artificial Intelligence based on Language Models (LMs) can help them on automatic tasks, allowing them to focus on more creative tasks and higher skills. However, the teaching and evaluation of technical tasks in Computer Science can be inefficient and prone to errors. Thus, the main objective of this article is to explore the performance of LMs compared to that of undergraduate Computer Science students in a specific case study: designing and implementing RESTful APIs DSLs. This research aims to determine if LMs can enhance the efficiency and accuracy of these processes. Our case study involved 39 students and 5 different LMs that must use the two DSLs we also designed for their task assignment. To evaluate performance, we applied uniform criteria to student and LMs-generated solutions, enabling a comparative analysis of accuracy and effectiveness. With a case study comparing performance between students and LMs, this article contributes to checking to what extent LMs are able to carry out software development tasks involving the use of new DSLs specially designed for highly specific settings in a similar way as well-qualified Computer Science students are able to. The results underscore the importance of welldefined DSLs and effective prompting processes for optimal LM performance. Specifically, LMs demonstrated high variability in task execution, with two GPT-based LMs achieving similar grades to those scored by the best of the students for every task, obtaining 0.78 and 0.92 on a normalized scale [0, 1], with 0.23 and 0.14 Standard Deviation for ChatGPT-4 and ChatGPT-4o respectively. After the experience, we can conclude that a well-defined DSL and a proper prompting process, providing the LM with metadata, persistent prompts, and a good knowledge base, are crucial for good LM performance. When LMs receive the right prompts, both large and small LMs can achieve excellent results depending on the task.

Despite the extensive adoption of crowdsourcing for the timely, cost-effective, and high-quality completion of software development tasks, a large number of crowdsourced challenges are not able to acquire a winning solution, on time, and within the desired cost and quality thresholds. A possible reason for this is that we currently lack a systematic approach that would aid software managers during the process of designing software development tasks that will be crowdsourced. This paper attempts to extend the current knowledge on designing crowdsourced software development tasks, by empirically answering the following management questions: (a) what type of projects should be crowdsourced; (b) why should one crowdsource-in terms of acquired benefits; (c) where should one crowdsource-in terms of application domain; (d) when to crowdsource-referring to the time period of the year; (e) who will win or participate in the contest; and (f) how to crowdsource (define contest duration, prize, type of contest etc.) to acquire the maximum benefits-depending on the goal of crowdsourcing. To answer the aforementioned questions, we have performed a case study on 2,209 software development tasks crowdsourced through TopCoder platform. The results suggest that there are significant differences in the level to which crowdsourcing goals are reached, across different software development activities. Based on this observation we suggest that software managers should prioritize the goals of crowdsourcing, decide carefully upon the activity to be crowdsourced and then define the settings of the task.

Large Language Models (LLMs) have demonstrated powerful capabilities in the field of code generation, with a deep understanding of the semantics and functionality of code. Building websites is one of the most important tasks in software development, as it utilizes rich frontend displays and backend processing to achieve various service functions. It is one of the most widely used interactive software models. Although there have been some efforts in Web website generation, these efforts have been limited to the automation of generating Web pages. The advent of LLMs provides a new approach to Web site generation tasks. However, there is currently a lack of comprehensive evaluation of the generation performance of LLMs in this context, making it difficult to optimize and improve the generated results in a targeted manner. To address this issue, this paper conducts a multi-angle investigation and analysis of the performance of LLMs in Web site generation tasks. Firstly, Web generation requirements are collected, and effective prompt engineering is designed. These prompts are then input into different LLMs to initiate the self-iteration process. Next, the generated code is fed back into the LLM for security self-iteration, where the model performs vulnerability detection and repair on the code it has generated. The security-enhanced code is subsequently subjected to manual review, where it is evaluated using predefined quantitative metrics to generate indicator values. Finally, through testing, the quality, security, and code defects of the generated front-end and back-end Web code across different LLMs are analyzed, providing a comprehensive evaluation of the generation results. The experiments demonstrate that LLM systems perform well in completing and implementing the functions and layouts of pages in prompts for Web generation tasks, but there remains room for improvement in the security of the Web code.

Software development is a collaborative process in which programmers build software by integrating all the stages of the software development life cycle (SDLC). A software repository is a central file storage location where various software packages are stored, and these packages are retrieved and shared between all the software development team members at various locations. The software repositories are divided into various categories based on cooperation, coordination, and communication among the stakeholders as well as evolutionary changes to various software arti-facts such as source code repositories, software bug repositories, historical repositories, run-time repositories and requirement documents, and other documentation. The software bug repository is an essential repository among the entire repositories since the completion of the software is entirely dependent on the bug fixing mechanism associated with this repository in software development. Today's software systems are larger and more complex as they go through various stages from the requirement 98analysis phase to the maintenance phase. A variety of tasks and activities are carried out in each stage of software development, and these are expensive and vulnerable to errors. During software development, a large number of software bugs are continuously generated, and that has become the main reason for the delay in software completion. Hence, there is a vast demand for computational intelligent techniques to accomplish various tasks of software development. In recent years, fuzzy logic techniques emerged and played an important role in various fields of data mining and text mining. Since most of the content related to software bug repositories is text in nature, it is possible to effectively use fuzzy logic techniques to analyze these software bugs.

Code search is an essential task in software development. Developers often search the internet and other code databases for necessary source code snippets to ease the development efforts. Code search techniques also help learn programming as novice programmers or students can quickly retrieve (hopefully good) examples already used in actual software projects. Given the recurrence of the code search activity in software development, there is an increasing interest in the research community. To improve the code search experience, the research community suggests many code search tools and techniques. These tools and techniques leverage several different ideas and claim a better code search performance. However, it is still challenging to illustrate a comprehensive view of the field considering that existing studies generally explore narrow and limited subsets of used components. This study aims to devise a grounded approach to understanding the procedure for code search and build an operational taxonomy capturing the critical facets of code search techniques. Additionally, we investigate evaluation methods, benchmarks, and datasets used in the field of code search.

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents at the time of paper submission! Agentless also achieves more than 50% solve rate when using Claude 3.5 Sonnet on the new SWE-bench Verified benchmark. In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new o1 models; more recently, Agentless has also been used by DeepSeek to evaluate their newest DeepSeek V3 and R1 models. Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-𝑆 by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless

Role stereotypes indicate generic roles that classes play in the design of software systems (e.g. controller, information holder, or interfacer). Knowledge about the role-stereotypes can help in various tasks in software development and maintenance, such as program understanding, program summarization, and quality assurance. This paper presents an automated machine learning-based approach for classifying the role-stereotype of classes in Java. We analyse the performance of this approach against a manually labelled ground truth for a sizable open source project (of 770+ Java classes) for the Android platform. Moreover, we compare our approach to an existing rule-based classification approach. The contributions of this paper include an analysis of which machine learning algorithms and which features provide the best classification performance. This analysis shows that the Random Forest algorithm yields the best classification performance. We find however, that the performance of the ML-classifier varies a lot for classifying different role-stereotypes. In particular its performs degrades for rare role-types. Our ML-classifier improves over the existing rule-based classification method in that the ML-approach classifies all classes, while rule-based approaches leave a significant number of classes unclassified.

Bug localization, i.e., locating bugs in code snippets, is a frequent task in software development. Although static bug-finding tools are available to reduce manual effort in bug localization, these tools typically detect bugs with known project-independent bug patterns. However, many bugs in real-world code snippets are project-specific. To address this issue, in this paper, we propose a novel approach for LOcating Bugs By Searching the most similar sample snippet (LOBBYS). LOBBYS detects bugs with the help of crowd-contributed correct code, which implement the function that buggy code is expected to implement. Given a buggy code snippet, LOBBYS takes two steps to locate the bug: (1) normalize the bug-gy snippet, and then search for the most similar sample snippet from the code base; (2) align the buggy code and sample code snip-pets, find the difference between the two code snippets, and generate a bug report based on the difference. To evaluate LOBBYS, we build one algorithm-oriented code base and select some buggy snippets from two real-world systems. The result shows that LOBBYS can effectively locate bugs for buggy snippets with high precision. Under the similarity of 50%, 70% and 90%, LOBBYS achieves bug-localization precision as 67%, 83%, and 92%.

Program entanglement, feature interaction and the feature language extensions

Debugging is a time-consuming task in software development and maintenance. To accelerate this task, several approaches have been proposed to automate fault localization. In particular, testing based fault localization (TBFL), which utilizes the testing information to localize the faults, seem to be very promising. However, the similarity between test cases in the test suite has been ignored in the research on TBFL. In this paper, we investigate this similarity issue and propose a novel approach named similarity-aware fault localization (SAFL), which can calculate the suspicion probability of each statement with little impact by the similarity issue. To address and deal with the similarity between test cases, SAFL applies the theory of fuzzy sets to remove the uneven distribution of the test cases. We also performed an experimental study for two real-world programs at different size levels to evaluate SAFL together with another two approaches to TBFL. Experimental results show that SAFL is more effective than the other two approaches when the test suites contain injected redundancy, and SAFL can achieve a competitive result with normal test suites. SAFL can also be more effective than applying test suite reduction to current approaches to TBFL.

Code reading is a prerequisite of program comprehension which is a fundamental task in software development. Strategies employed on code reading affect the programmer’s success rate of understanding tasks such as debugging. However, there is still limited knowledge about the code reading strategies used by students while performing bug finding task. In this paper, the author describes a summary of her research on novice programmer debugging skills using eye tracking data as a methodology. Eye tracking data were extracted and analyzed using visual effort metrics and sequential analysis of scanpaths using a clustering algorithm to determine common code reading patterns. The author’s research findings revealed differences on the code reading patterns and code reading strategies of high and low performing students. Empirical evaluation on the effectiveness of the strategies used by high performing students was also conducted which suggests that by teaching these strategies to students, improved debugging performance can be observed.

Reading and understanding source code is a major task in software development. Code comprehension depends on the quality of code, which is impacted by code structure and identifier naming. In this paper we empirically investigated whether longer but more descriptive identifier names improve code comprehension compared to short names, as they represent useful information in more detail. In a web-based study 88 Java developers were asked to locate a semantic defect in source code snippets. With descriptive identifier names, developers spent more time in the lines of code before the actual defect occurred and changed their reading direction less often, finding the semantic defect about 14% faster than with shorter but less descriptive identifier names. These effects disappeared when developers searched for a syntax error, i.e., when no in-depth understanding of the code was required. Interestingly, the style of identifier names had a clear impact on program comprehension for more experienced developers but not for less experienced developers.

One central task in software development by means of graph-based techniques is to inspect and to query the underlying graph. Important issues are, for example, to detect general graph properties like connectivity, to explore more special features like the applicability of left-hand side rules in graph transformations, or to validate snapshots of evolving systems by checking properties in an on-the-fly way. We propose a new approach combining visual and textual techniques for exploring graphs. We emphasize a particular aspect of the underlying graph by showing or hiding nodes and edges. We offer three different ways to explore (meta-)model snapshots which may be combined: (1) selection by object identity and class membership, (2) selection by OCL expression, and (3) selection by path length. One main motivation for our work is to access large or complicated graphs in a systematic way. We evaluate our approach by different middle-sized scenarios. Our evaluation shows that the approach works for large graphs with about 1000 nodes and 2000 edges and for graphs which instantiate metamodels representing software engineering artifacts.

Aspect-Oriented Requirements Engineering (AORE) extends the existing requirements engineering approaches to support the identification and handling of crosscutting concerns. Crosscutting concerns are considered as potential aspects and can lead to the phenomenal tyranny of the dominant decomposition. Requirements-level aspects are responsible for producing scattered and tangled descriptions of requirements in the requirements document. Requirements validation artifact is an essential task in software development. This task ensures that requirements are correct and valid in terms of completeness and consistency, hence, reducing the development cost, maintenance and establish an approximately correct estimate of effort and completion time of the project. In this paper, we present a validation framework for aspectual requirements that can be used with AORE approaches to facilitate the validation of the resulting crosscutting relationships and aspects. The proposed framework comprises a high-level and low-level validation. The high-level validation is to validate the concerns with stakeholders, whereas the low-level validation validates the aspectual requirement by developers using a checklist. The approach has been evaluated using a case study. The results demonstrate that the proposed framework is feasible and acceptable.

As software projects continue to grow in scale, being able to maximize the work that developers can carry out in parallel as a set of concurrent development tasks, without incurring excessive coordination overhead, becomes increasingly important. Prevailing design models, however, are not explicitly conceived to suggest how development tasks on the software modules they describe can be effectively parallelized. In this paper, we present a design rule hierarchy based on the assumption relations among design decisions. Software modules located within the same layer of the hierarchy suggest independent, hence parallelizable, tasks. Dependencies between layers or within a module suggest the need for coordination during concurrent work. We evaluate our approach by investigating the source code and mailing list of Apache Ant. We observe that technical communication between developers working on different modules within the same hierarchy layer, as predicted, is significantly less than communication between developers working across layers.