Automating software size measurement from python code using language models

The paper studies the effectiveness of using large language models to detect common types of vulnerabilities in Python program code. In particular, using the technique of low-rank adaptation of (LoRA) models, fine-tuning of the CodeBERT-python model is performed. To train the models, we use the author's dataset, which consists of marked-up program code in Python. The trained models are used to detect and classify potential vulnerabilities. To evaluate the effectiveness of models, the number of false positives, false negatives, true positives and true negatives is determined. Also, accuracy, recall and F1-measures are calculated on a test data set for various configurations of model training macro parameters

The use of large language models (LLMs) for generating code in software development is on the rise. While LLMs demonstrate impressive capabilities, there remains a need to evaluate the quality of generated code beyond functional correctness. This paper addresses a gap in current research by evaluating Python code generated by nine state-of-the-art LLMs according to the four main code quality categories (maintainability, reliability, performance efficiency, and security) defined by the ISO/IEC 5055:2021 standard. The evaluation spans three popular application domains (high-performance computing, machine learning, and data processing). It employs a stratified prompting approach with varying levels (short, medium, and long) of detail. Using widely adopted static code analysis tools, the study identifies which LLMs perform best across these categories. The analysis involved selecting nine algorithms across three application domains, and the generated code was compared against a human developer using four static code analysis tools. Metrics were organized into the four ISO 5055 categories, and composite scores were calculated for each category following preprocessing to ensure accurate evaluation. The results showed that GPT-4-Turbo produced the most reliable, performance-efficient, and secure code, while Gemini excelled in generating maintainable Python code among the evaluated models. The study concludes that properly prompted and configured LLMs can produce code that meets or even exceeds human-developed code in terms of the four ISO categories. Future work will continue to refine and expand upon these methodologies on other programming languages.

Digital transformation in government requires the use of cutting-edge technology to improve efficiency and accuracy in decision-making. Ministries are strategic institutions that manage large-scale data and information, particularly related to regional government administration and population affairs. This article examines the application of Large Language Models (LLM) as an artificial intelligence (AI)-based solution to assist document processing, information extraction, and policy analysis. The study presents a relevant legal framework, LLM model recommendations, a schematic diagram of LLM implementation from end-to-end, and Python code examples for technical simulations. This approach is expected to strengthen the Ministry's digital system in accordance with the guidelines of the Electronic-Based Government System (SPBE) and One Data Indonesia.

The growing scale of large language models (LLMs) not only demands extensive computational resources but also raises environmental concerns due to their increasing carbon footprint. Model quantization emerges as an effective approach that can reduce the resource demands of LLMs by decreasing parameter precision without substantially affecting performance (e.g., 16 bit <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\rightarrow 4$</tex> bit). While recent studies have established quantization as a promising approach for optimizing large code models (LCMs)-a specialized subset of LLMs tailored for automated software engineering-their findings offer only limited insights into its practical implications. Specifically, current investigations only focus on the functional correctness of the code generated by quantized models, neglecting the gap that remains in understanding how quantization impacts critical aspects of quality attributes of the code, such as reliability, maintainability, and security. To bridge this gap, our study investigates the effects of quantization on the qualitative aspects of automatically generated code. We apply Activation-aware Weight Quantization (AWQ) to two widely used code models-CodeLlama and DeepSeekCoder-to generate Java and Python code. Leveraging state-of-the-art static analysis tools, we evaluate software quality metrics and static features, including cyclomatic complexity, cognitive complexity, and lines of code (LoC). Our findings reveal that quantization not only establishes itself as a robust technique capable of withstanding functional challenges-producing code that passes test cases at rates comparable to non-quantized models-but also preserves key qualitative attributes and static features often sought after by developers, such as maintainability and structural complexity.

As one of the most popular dynamic languages, Python experiences a decrease in readability and maintainability when code smells are present. Recent advancements in Large Language Models have sparked growing interest in AI-enabled tools for both code generation and refactoring. GitHub Copilot is one such tool that has gained widespread usage. Copilot Chat, released in September 2023, functions as an interactive tool aimed at facilitating natural language-powered coding. However, limited attention has been given to understanding code smells in Copilot-generated Python code and Copilot Chat's ability to fix the code smells. To this end, we built a dataset comprising 102 code smells in Copilot-generated Python code. Our aim is to first explore the occurrence of code smells in Copilot-generated Python code and then evaluate the effectiveness of Copilot Chat in fixing these code smells employing different prompts. The results show that 8 out of 10 types of code smells can be detected in Copilot-generated Python code, among which Multiply-Nested Container is the most common one. For these code smells, Copilot Chat achieves a highest fixing rate of 87.1%, showing promise in fixing Python code smells generated by Copilot itself. In addition, the effectiveness of Copilot Chat in fixing these smells can be improved by providing more detailed prompts.

Readability of Orthopaedic Patient Educational Material: An artificial intelligence application.

Pre-trained language models have achieved many prominent results in natural language processing. Since software engineering widely includes many natural language documents, the application of pre-trained language models have received much attention in software engineering tasks. However, pretraining a large volume of source code requires a huge amount of computational resources and time. In this study, we propose an additional pre-training approach to a well-trained language model. Our initial results on mT5, multilingual T5 with an additional pretraining of Python code shows improved performance on multiple software engineering tasks including code generation, code summarisation, code repair, and error diagnosis.

With the rise in using Large Language Models (LLMs), research proved the feasibility of using LLMs in various software engineering tasks including code generation. However, it remains a major challenge to ensure the correctness and the efficiency of the LLM-generated code. This paper applies the "Chain-of-thought" prompting approach to improve GPT performance in code generation and its ability to convert coding requirements from HackerRank and LeetCode online coding platforms into correct Python, C++, and JavaScript code. We evaluated the performance of GPT-3.5 and GPT-4 across 558 programming challenges on HackerRank and LeetCode platforms. Our evaluation framework employed a comprehensive test case validation system, where each generated solution was tested against predefined input-output pairs, with success determined by exact matches between expected and actual outputs. The evaluation revealed significant variations in success rates across platforms (HackerRank: 46.83% vs. LeetCode: 7.35%) and programming languages (Python leading with 28.6% overall success rate). These results demonstrate that while the chain of thought approach enhances code generation accuracy, particularly for structured programming tasks, significant challenges remain in handling complex algorithmic problems.

Employee Application (EA) is a system information for realizing the latest personal data and integrated, provide accurate employee information for planning, development, welfare and control of employees. EA in STT-PLN began to be published since 2013. The software size of the EA STT-PLN will be measured with use case point method. Measurement of the software size of EA STT-PLN will be measured with Use Case Point upon use case diagram for EA STT-PLN as shown in the project has small software size where score Use Case Point (UCP) = 70.34. UCP is another alternative implementation method which can be applied to measure application software size whenever needed to deal with time, money and people.

Software size is an important attribute in software project planning. Several methods for software size estimation are available; most of them are based on function points. Albrecht introduced function points as a technologically independent method with its own software abstraction layer. However, it is difficult to apply original abstraction elements to current technologies. Therefore researchers introduced additional rules and mappings for object-based solutions. In this paper several mapping strategies are discussed and compared. Based on the similarities in compared mappings, a common mapping strategy is then defined. This mapping is then tested on the reference application portfolio containing five applications. The aim of the test scenario is to evaluate the impact of the diverse detail levels in the class diagrams on software size measurement. Although the question of how to perform quality size measurements in object-oriented projects remains unanswered, the paper gives valuable information on the topic, supported by mathematics.

Effort and cost estimation is crucial in software management. Estimation of software size plays a key role in the estimation. Line of Code (LOC) is still a commonly used software size measure. Despite the fact that software sizing is well recognized as an important problem for more than two decades, there is still much problem in existing methods. Conceptual data model is widely used in the requirements analysis for information systems. It is also not difficult to construct conceptual data models in the early stage of developing information systems. Much characteristic of an information system is actually reflected from its conceptual data model. We explore into the use of conceptual data model for estimating LOC. This paper proposes a novel method for estimating LOC for an information system from its conceptual data model through the use of multiple linear regression model. We have validated the method through collecting samples from both the industry and open-source systems.

Effective management of the software development process requires that management be able to estimate total development effort and cost. One of the fundamental problems associated with effort and cost estimation is the a priori estimation of software size. Function point analysis has emerged over the last decade as a popular tool for this task. Criticisms of the method that relate to the way in which function counts are calculated and the impact of the processing complexity adjustment on the function point count have arisen. SPQR/20 function points among others are claimed to overcome some of these criticisms. The SPQR/20 function point method is compared to traditional function point analysis as a measure of software size in an empirical study of MIS environments. In a study of 64 projects in one organization it was found that both methods would appear equally satisfactory. However consistent use of one method should occur since the individual counts differ considerably. >

Nowadays, software effort estimation plays an important role in software project management due to its extensive use in industry to monitor progress, and performance, determine overall productivity and assist in project planning. After the success of methods such as IFPUG Function Point Analysis, MarkII Function Point Analysis, and COSMIC Full Function Points, several other extension methods have been introduced to be adopted in software projects. Despite the efficiency in measuring the software cost, software effort estimation, unfortunately, is facing several issues; it requires some knowledge, effort, and a significant amount of time to conduct the measurement, thus slightly ruining the advantages of this approach. This paper demonstrates a functional size measurement tool, named UML Point tool, that utilizes the concept of IFPUG Function Point Analysis directly to Unified Modeling Language (UML) model. The tool allows the UML eXchange Format (UXF) file to decode the UML model of mobile game requirement and extract the diagrams into component complexity, object interface complexity, and sequence diagram complexity, according to the defined measurement rules. UML Point tool then automatically compute the functional size, effort, time, human resources, and total development cost of mobile game. Besides, this paper also provides a simple case study to validate the tool. The initial results proved that the tool could be useful to improve estimation accuracy for mobile game application development and found to be reliable to be applied in the mobile game industry.

This paper reports on a study carried out within a software development organization to evaluate the use of function points as a measure of early lifecycle software size. There were three major aims to the research: firstly to determine the extent to which the component elements of function points were independent of each other and thus appropriate for an additive model of size; secondly to investigate the relationship between effort and (1) the function point components, (2) unadjusted function points, and (3) adjusted function points, to determine whether the complexity weightings and technology adjustments were adding to the effort explanation power of the metric; and thirdly to investigate the suitability of function points for sizing in client server developments. The results show that the component parts are not independent of each other which supports an earlier study in this area. In addition the complexity weights and technology factors do not improve the effort/size model, suggesting that a simplified sizing metric may be appropriate. With respect to the third aim it was found that the function point metric revealed a much lower productivity in the client server environment. This likely is a reflection of cost of the introduction of newer technologies but is in need of further research.

In order to develop cost-efficient software it is crucial to measure the accurate software size. However, measuring the software size has up to now been almost entirely a manual process and, as such, is both time-consuming and prone to human error which can end up with time and money loss. Automation of this process is a must for the software developing companies to improve the quality of project and budget planning. This paper introduces a mapping between COSMIC concept elements and UML conceptual elements and an automation tool in order to capture the information needed for functional software size measurement from UML diagrams in a component based software product line environment (CBPL) that are structured in accordance with the interface based design (IbD) method. The mapping and the tool combined is called PL FSM. The results obtained by manual measurement and automated measurement are compared and the results are observed to be close. As a result of this study, PL FSM approach is validated in CBPL environment at the design stage of software development process. The case studies have been carried out in embedded systems domain.