Security Notifications in Static Analysis Tools: Developers’ Attitudes, Comprehension, and Ability to Act on Them

Static analysis of source code security: Assessment of tools against SAMATE tests

Static code analysis is widely used to support the development of high-quality software. It helps developers detect potential bugs and security vulnerabilities in a program's source code without executing it. While the potential benefits of static analysis tools are beyond question, their usability is often criticised and prevents software developers from using static analysis to its full potential. In the past decade, researchers have studied developer needs and contrasted them to available static analysis tool functionalities. In this paper, we summarize the main design challenges for building usable static analysis tools, and show that they revolve around the notion of explainability, which is a subarea of usability. We present existing analysis tools and current research in static analysis usability, and detail how they approach those challenges. This leads us to proposing potential lines of future work in explainability for static analysis, namely turning static analysis tools into assistants and teachers.

Quality and cost have close relationship in software development process. That's why quality has to be better and cost must to be lower for a software developer organization. In our paper, we are presenting an approach that will give details analysis how cost can be reduced by detect defection in code using static analysis tool and formal inspection together with. Our proposed method will help software industry to think acutely about cost reduction by defect detection in code. Keywords- Cost reduction, Code inspection Quality assurance, Static analysis tool, inspection.

Achieving comprehensive code quality and ensuring software reliability are critical goals in modern software engineering. This paper delves into advanced strategies that encompass both technical and organizational practices, aiming to enhance code quality and boost software reliability. Key strategies include the implementation of rigorous code review processes, adoption of automated testing frameworks, and the utilization of static and dynamic code analysis tools. Firstly, rigorous code review processes are fundamental for maintaining high standards of code quality. By fostering a collaborative environment where peer reviews are routine, teams can catch potential issues early, ensure adherence to coding standards, and promote knowledge sharing. This practice not only improves code quality but also enhances team cohesion and expertise. Secondly, automated testing frameworks are indispensable in achieving comprehensive test coverage and reducing the incidence of bugs. Unit tests, integration tests, and end-to-end tests, when automated, provide continuous feedback and ensure that code changes do not introduce new defects. This continuous testing approach is crucial for maintaining software reliability, especially in agile and DevOps environments where rapid iterations are common. Furthermore, the integration of static and dynamic code analysis tools into the development pipeline offers additional layers of quality assurance. Static analysis tools inspect the codebase for potential vulnerabilities, code smells, and compliance with coding standards without executing the program. Dynamic analysis tools, on the other hand, evaluate the software during runtime to detect memory leaks, concurrency issues, and performance bottlenecks. Together, these tools provide a comprehensive assessment of the code’s health and reliability. Moreover, fostering a culture of continuous improvement and learning within the development team is essential. Regular training sessions, workshops, and knowledge-sharing activities help in keeping the team updated with the latest advancements in software engineering practices and tools. Encouraging a mindset of quality-first and reliability among developers ensures long-term benefits and sustainability. In conclusion, achieving comprehensive code quality and ensuring software reliability require a multifaceted approach that integrates rigorous code reviews, automated testing frameworks, and both static and dynamic analysis tools. Coupled with a strong culture of continuous learning and improvement, these strategies collectively contribute to the development of robust, reliable, and high-quality software systems. Keywords: Software Reliability, Code Quality, Advanced, Strategies, Comprehensive.

Web applications have been widely adopted to access a myriad of services, regardless of their criticality and context. Applications developers have accelerated their efforts to meet the demands of a competitive and dynamic market for innovative products. Despite considerable efforts to detect and mitigate vulnerabilities in applications, their prevalence continues to increase, primarily due to the rapid pace of software development, which often prioritizes deployment speed, compromising security. This paper presents KAVe, a static analysis tool that leverages a multi-layer knowledge graph and a multi-agent system to detect web application vulnerabilities with high precision. This paper showcases KAVe's implementation and ability to identify SQL injection (SQLi) and cross-site scripting (XSS) vulnerabilities in real-world PHP applications.

Static analysis tools (e.g., FindBugs) are widely used to detect potential defects in software development. A recent study suggests that there is a moderate correlation between the alerts reported by static analysis tools and software defects [1]. However, despite the actionable alerts reported by static analysis tools, they may report too many meaningless unactionable alerts. Actionable alert refers to the alert which is meaningful and fixable. Unactionable alert (i.e., false positive alert) refers to the alert which is regarded as unimportant to developers, inessential to source code, or will not be fixed by developers. Are all alerts (including both actionable and unactionable alerts) suitable for indicating software defects? To address this question, we classify all the alerts into two categories, namely actionable alerts and unactionable alerts. By the following, we conduct an empirical study to evaluate the degree of correlation between defects and alerts on the evolution of three open source projects with totally 40 releases. The objective of the study is to explore two kinds of correlation analysis: one is the correlation between all the alerts reported by FindBugs and defects among the release history of a project, the other is the correlation between the actionable alerts and defects. As a result, we find that not all the alerts but the actionable alerts are suitable to be an early predictor of defects.

The prevalence and significance of web services in our daily lives make it imperative to ensure that they are – as much as possible – free from vulnerabilities. However, developing a complex piece of software free from any security vulnerabilities is hard, if not impossible. One way to progress towards achieving this holy grail is by using static code analysis tools to root out any common or known vulnerabilities that may accidentally be introduced during the development process. Static code analysis tools have significantly contributed to addressing the problem above, but are imperfect. It is conceivable that static code analysis can be improved by using AI-powered tools, which have recently increased in popularity. However, there is still very little work in analysing both types of tools’ effectiveness, and this is a research gap that our paper aims to fill. We carried out a study involving 11 static code analysers, and one AI-powered chatbot named ChatGPT, to assess their effectiveness in detecting 92 vulnerabilities representing the top 10 known vulnerability categories in web applications, as classified by OWASP. We particularly focused on PHP vulnerabilities since it is one of the most widely used languages in web applications. However, it has few security mechanisms to help its software developers. We found that the success rate of ChatGPT in terms of finding security vulnerabilities in PHP is around 62-68%. At the same time, the best traditional static code analyser tested has a success rate of 32%. Even combining several traditional static code analysers (with the best features on certain aspects of detection) would only achieve a rate of 53%, which is still significantly lower than ChatGPT’s success rate. Nonetheless, ChatGPT has a very high false positive rate of 91%. In comparison, the worst false positive rate of any traditional static code analyser is 82%. These findings highlight the promising potential of ChatGPT for improving the static code analysis process but reveal certain caveats (especially regarding accuracy) in its current state. Our findings suggest that one interesting possibility to explore in future works would be to pick the best of both worlds by combining traditional static code analysers with ChatGPT to find security vulnerabilities more effectively.

Leveraging and acting on the results of static analysis is a well-known best practice. Static analysis also is an integral part of software quality assurance which is highlighted by the inclusion of static analysis results in software quality models like ColumbusQM and Quamoco. Although there are studies that explore if tools for static analysis are used and how they are configured, few publications explore the longitudinal effects of acting on static analysis results on software and on the evolution of software. Especially effects on quality criteria, e.g., software quality metrics, defects, or readability are missing. With our research, we will bridge this gap and measure the effects of static analysis on software quality evolution. We will measure the effect the removal of code that generates static analysis warnings has on software quality metrics. Furthermore, we will measure long term effects on external quality attributes, e.g., reported issues and defects. Finally, we want to predict false positives of static analysis warnings by training predictive models on our collected data.

Web applications are becoming an essential part of our everyday lives. Many\nof our activities are dependent on the functionality and security of these\napplications. As the scale of these applications grows, injection\nvulnerabilities such as SQL injection are major security challenges for\ndevelopers today. This paper presents the technique of automatic query\nsanitization to automatically remove SQL injection vulnerabilities in code. In\nour technique, a combination of static analysis and program transformation are\nused to automatically instrument web applications with sanitization code. We\nhave implemented this technique in a tool named ASSIST (Automatic and Static\nSQL Injection Sanitization Tool) for protecting Java-based web applications.\nOur experimental evaluation showed that our technique is effective against SQL\ninjection vulnerabilities and has a low overhead.\n

Programming environments (PEs) support a single programmer developing small- to medium-scale programs, whereas software development support systems and software engineering environments (SE 2 s) support whole project teams, developing Large-scale software. There is no reason to believe that one and only one support system may exist. The usefulness of one or the other depends on the particular situation of software development. Debugging is distinguished from testing and defined not only for removing bugs from programs (dynamic debugging) but also from documents describing the programs (static debugging). The key problem of debugging is understanding the software. Debugging may be supported by static analysis tools and by interactive debugging systems which help both to understand the software better and to estimate the impacts of an intended change. Graphical representations are also very useful for better understanding system structures and for recognising faults and clashes faster. Tools may furthermore be used to report errors and corrections, and to maintain these reports. In the context of PEs and SE 2 s the tools supporting debugging are connected. Tools can be based on a uniform internal representation, allowing a uniform user interface. Tasks and corresponding tools can be tailored to each other, avoiding duplication of work. One task knows what the others have already done. One knows if certain types of errors have been prevented or removed, i.e. if static analysis tools prevent data flow errors during runtime. Tools and results of other tasks may be used, i.e. cross reference lists and test path reports. Similarities and differences of debugging in PEs and in SE 2 s are discussed by some example systems. The discussion is concluded by demonstrating possible influences on future software development by personal computers, knowledge engineering, and predicative programming.

Supporting secure programming in web applications through interactive static analysis

Security has been always treated as an add-on feature in the software development lifecycle, and addressed by security professionals using firewalls, proxies, intrusion prevention systems, antivirus and platform security. Software is at the root of all common computer security problems, and hence hackers don't create security holes, but rather exploit them. Security holes in software applications are the result of bad design and implementation of software systems and applications. To address this problem, several initiatives for integrating security in the software development lifecycle have been proposed, along with tools to support a security-centric software development lifecycle. This paper introduces a framework for evaluating security static analysis tools such as source code analyzers, and offers evaluation of non-commercial static analysis tools such as Yasca, CAT.NET, and FindBugs. In order to evaluate the effectiveness of such tools, common software weaknesses are defined based on CWE/SANS Top 25, OWASP Top Ten and NIST source code weaknesses. The evaluation methodology is based on the NIST Software Assurance Metrics And Tool Evaluation (SAMATE). Results show that security static analysis tools are, to some extent, effective in detecting security holes in source code; source code analyzers are able to detect more weaknesses than bytecode and binary code scanners; and while tools can assist the development team in security code review activities, they are not enough to uncover all common weaknesses in software. The new test cases developed for this research have been contributed to the NIST Software Assurance Reference Dataset (samate.nist.gov/SARD).

Machine Learning (ML) projects incur novel challenges in their development and productionisation over traditional software applications, though established principles and best practices in ensuring the project's software quality still apply. While using static analysis to catch code smells has been shown to improve software quality attributes, it is only a small piece of the software quality puzzle, especially in the case of ML projects given their additional challenges and lower degree of Software Engineering (SE) experience in the data scientists that develop them. We introduce the novel concept of project smells which consider deficits in project management as a more holistic perspective on software quality in ML projects. An open-source static analysis tool mllint was also implemented to help detect and mitigate these. Our research evaluates this novel concept of project smells in the industrial context of ING, a global bank and large software- and data-intensive organisation. We also investigate the perceived importance of these project smells for proof-of-concept versus production-ready ML projects, as well as the perceived obstructions and benefits to using static analysis tools such as mllint. Our findings indicate a need for context-aware static analysis tools, that fit the needs of the project at its current stage of development, while requiring minimal configuration effort from the user.

The use of large language models (LLMs) for generating code in software development is on the rise. While LLMs demonstrate impressive capabilities, there remains a need to evaluate the quality of generated code beyond functional correctness. This paper addresses a gap in current research by evaluating Python code generated by nine state-of-the-art LLMs according to the four main code quality categories (maintainability, reliability, performance efficiency, and security) defined by the ISO/IEC 5055:2021 standard. The evaluation spans three popular application domains (high-performance computing, machine learning, and data processing). It employs a stratified prompting approach with varying levels (short, medium, and long) of detail. Using widely adopted static code analysis tools, the study identifies which LLMs perform best across these categories. The analysis involved selecting nine algorithms across three application domains, and the generated code was compared against a human developer using four static code analysis tools. Metrics were organized into the four ISO 5055 categories, and composite scores were calculated for each category following preprocessing to ensure accurate evaluation. The results showed that GPT-4-Turbo produced the most reliable, performance-efficient, and secure code, while Gemini excelled in generating maintainable Python code among the evaluated models. The study concludes that properly prompted and configured LLMs can produce code that meets or even exceeds human-developed code in terms of the four ISO categories. Future work will continue to refine and expand upon these methodologies on other programming languages.

Developing secure software in a world where companies like Anthem Blue Cross, Twitter, Facebook, and target have had massive amounts of data stolen by hackers is as challenging as it is important. Insecure coding practices are major contributors to software security vulnerabilities. Even though several static analysis tools are available that can search for and identify security holes in software applications, this process usually runs too late and any remediation will be more costly after large portions of the software have been built. The early detection tools that do exist are closed source and utilise proprietary software vulnerability rule sets. What is missing is an open-source secure coding enforcement tool utilising well-documented rules that software developers can use to predict potential pitfalls, learn from their mistakes and aid in the construction of secure programs as they build them. To address the need, we have designed a new tool called secure coding assistant for the Eclipse development environment that semi-automates several secure coding rules set forth by the CERT division at Carnegie Mellon University. The tool detects violations of the CERT rules for the Java programming language but it is easily extensible to other languages supported by Eclipse. It is an open-source tool with an emphasis on educating software developers in secure coding practices. The tool and a tool demo is disseminated via github at http://benw408701.github.io/SecureCodingAssistant/.