The Impact of AI Tool on Engineering at ANZ Bank an Empirical Study on GitHub Copilot within Corporate Environment

The paper studies the problem of software code migration during the transition from the old version of the SAP ERP 6.0 system to modern ERP platforms, in particular SAP S/4HANA. The necessity of automation of the migration process using artificial intelligence methods and tools is substantiated. A comprehensive analysis of existing academic and industrial case studies confirmed the feasibility of using large language models for code migration. A number of advantages of the approach based on large language models are revealed, in particular, the ability to handle various migration scenarios. The key risks and limitations common in the enterprise environment are highlighted, including data security, privacy, high cost of API calls and limited knowledge of models regarding the subject area. A method is developed for selecting a large open source language model that can be safely used in the enterprise landscape, thereby mitigating the risks associated with cloud solutions. The method is based on the developed system of criteria, including: model size, context window size, license type, support for fine-tuning training, focus on code quality (measured using a reference dataset), support for model quantization and community support. The hybrid AHP-TOPSIS method is proposed for systematic ranking of candidate models. The AHP method was used to check the weights of the criteria, while TOPSIS was used to rank the models based on their proximity to the ideal solution. Based on the developed method, three large language models were selected: Qwen 2.5 Coder 14B, DeepSeek-Coder-V2 16B and Llama 3.1 8B. A comprehensive testing plan for the selected models was developed for further experimental studies. The scientific novelty of the work lies in the development of a method for selecting large language models for custom code migration tasks, which differs from the existing system of model selection criteria and the use of a hybrid AHP-TOPSIS approach for ranking candidate models. The practical value of the work is that the developed method allows you to select a model taking into account the constraints of the corporate environment and information security requirements, which will increase the level of automation in the migration of AVAP when switching to new versions of ERP systems from SAP SE. The developed method can be applied to the selection of large language models when upgrading any large computer systems developed in both open source and proprietary programming languages.

Achieving comprehensive code quality and ensuring software reliability are critical goals in modern software engineering. This paper delves into advanced strategies that encompass both technical and organizational practices, aiming to enhance code quality and boost software reliability. Key strategies include the implementation of rigorous code review processes, adoption of automated testing frameworks, and the utilization of static and dynamic code analysis tools. Firstly, rigorous code review processes are fundamental for maintaining high standards of code quality. By fostering a collaborative environment where peer reviews are routine, teams can catch potential issues early, ensure adherence to coding standards, and promote knowledge sharing. This practice not only improves code quality but also enhances team cohesion and expertise. Secondly, automated testing frameworks are indispensable in achieving comprehensive test coverage and reducing the incidence of bugs. Unit tests, integration tests, and end-to-end tests, when automated, provide continuous feedback and ensure that code changes do not introduce new defects. This continuous testing approach is crucial for maintaining software reliability, especially in agile and DevOps environments where rapid iterations are common. Furthermore, the integration of static and dynamic code analysis tools into the development pipeline offers additional layers of quality assurance. Static analysis tools inspect the codebase for potential vulnerabilities, code smells, and compliance with coding standards without executing the program. Dynamic analysis tools, on the other hand, evaluate the software during runtime to detect memory leaks, concurrency issues, and performance bottlenecks. Together, these tools provide a comprehensive assessment of the code’s health and reliability. Moreover, fostering a culture of continuous improvement and learning within the development team is essential. Regular training sessions, workshops, and knowledge-sharing activities help in keeping the team updated with the latest advancements in software engineering practices and tools. Encouraging a mindset of quality-first and reliability among developers ensures long-term benefits and sustainability. In conclusion, achieving comprehensive code quality and ensuring software reliability require a multifaceted approach that integrates rigorous code reviews, automated testing frameworks, and both static and dynamic analysis tools. Coupled with a strong culture of continuous learning and improvement, these strategies collectively contribute to the development of robust, reliable, and high-quality software systems. Keywords: Software Reliability, Code Quality, Advanced, Strategies, Comprehensive.

Context: In the last years, the Grey Literature (GL) had a notable increase in interest in diverse areas of knowledge. In Software Engineering (SE), with the advance of social media, the SE practitioners are heavy consumers of GL, opposite to traditional research papers, such as the Systematic Literature Review (SLR). Problem: Despite the increase of numbers of SLRs, some researchers claim the lack of connection to the practice of SE, due to the lack of evidence derived from SE practitioners. Goal: Introduce methods and techniques to support a Grey Literature Review (GLR) in SE, aiming to approximate the literature review to the practice of SE. Method: First, we will investigate how secondary studies use GL; Second, we will employ a survey with researchers of SLRs and SE practitioners; Third, we will analyze the use of GL according to the context; Fourth, we will review the GL source in SE; Finally, we will propose a guideline to help SE researchers and practitioners to conduct the GLR. Preliminary findings: We have initiated the tertiary study conducting the manual (4,875 studies) and automatic (6,279 studies) searches. We eliminated those that were not an article, and also those were not written in English, remaining 8,019 potential studies. Currently, we are applying the inclusion and exclusion criteria.

The rapid evolution of Artificial Intelligence (AI) has brought about significant advancements in multiple domains, including software development. One of the most promising innovations is AI-powered code generation through Large Language Models (LLMs), such as OpenAI’s GPT-3 and GPT-4. These models, having been trained on large amounts of programming data, have the ability to produce human-readable code from natural language inputs, which is a big potential for simplifying and optimizing software development processes. The aim of this paper is to analyze the performance of LLMs in automated software development by testing their performance on a variety of tasks such as code generation, debugging, and optimization of software. The research explores both the strengths and weaknesses that these models have to offer, in terms of some of the most important indicators like code quality, generation time, and maintainability of the code. According to our observation, although LLMs hold immense potential to automate mundane programming tasks and enhance developer productivity, they still struggle to cope with more intricate, domain-specific programming tasks involving a higher level of understanding, for example, designing architectures and top-level decision-making. In spite of such shortcomings, LLMs can tremendously enhance software development processes, particularly for small-scale projects or act as helpers for more senior developers. The paper summarizes by reflecting on the potential for LLMs to transform software development processes in the future, while also the importance of the model's reliability, coding quality, and security to be improved if it is to be made applicable to larger, more crucial uses.

As Artificial Intelligence (AI) software becomes increasingly prevalent across various industries, concerns about its trustworthiness and reliability have come to the forefront. Although the trustworthiness of traditional software is regulated by Software Engineering (SE) practices, these practices have not been well integrated into AI model development due to the significant differences between traditional software development and AI model development. Inspired by this, we aim to systematically address trustworthiness by regulating the AI development process through the lens of SE practices. Specifically, we are inspired by the regulation of traditional software, focusing on the key phases in software regulation: software development, execution, and testing. We identify corresponding phases in AI model development: training, inference, and testing. These phases are crucial for ensuring the trustworthiness and reliability of AI models. My study aims to improve these phases to enhance the trustworthiness of AI models. Our primary approach to regulating AI model development mirrors traditional software practices. It involves first debugging these phases and then implementing repairs. Moreover, large language models (LLMs) are revolutionizing the software industry. Thus, in this thesis, I explore the debugging and repairing of AI software from three phases (i.e., training, inference, and testing), focusing on both small Deep Neural Networks (DNNs) and LLMs.

The world is experiencing an AI revolution, with large language models (LLMs) transforming various industries, including education. Academics are striving to harness the potential of LLMs while also contending with their risks. This paper presents the first bibliometric analysis focused on LLM research in programming education, identifying leading countries, authors, and institutions while analyzing key terms and popular keywords in this field. Additionally, it highlights influential studies on topics such as introductory programming, computer science, computing, programming education, and prompt engineering, discussing key insights from these works. Findings indicate that LLMs could play a significant role in programming education and may be integrated into computer science curricula. However, careful consideration is needed to ensure their benefits outweigh their risks across various use cases. This study specifically examines ChatGPT as a representative LLM, exploring its benefits and limitations as both a learning aid for students and a support tool for professionals. It also evaluates the quality of ChatGPT-generated code and its effectiveness in simplifying programming concepts for beginners. Furthermore, the ethical implications of increasing reliance on LLMs for programming tasks, including concerns about dependency, plagiarism, and potential effects on critical thinking, are addressed. By contributing to the ongoing discourse on integrating AI tools like ChatGPT in programming education, this research emphasizes the importance of responsible and ethical usage to maximize benefits for students, educators, and the broader educational community.

Clean room software engineering is a formal software development method that can strictly engineer software development and eliminate defects before they can cause serious harm. The clean room software engineering model has its own advantages and disadvantages in use. To use it in traditional software engineering, it is necessary to clarify the relationship between it and traditional software engineering methods and practices. Based on the technology and principles of the clean room software process, this paper gets the advantages and disadvantages of the clean room. Corresponding the traditional method-based software process to the key technology of the clean room, making the traditional management-based software engineering method and the tailored clean room compatible, and the software engineering practice of the clean room software process is compared with the traditional software engineering practice. To discuss the clean room and traditional software engineering, in order to expand the use of clean room methods, improve the clean room software process, and improve the quality of software.

Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities

This study investigates the integration of large language models (LLMs) alongside computer algebra systems (CAS) in a mathematics laboratory for civil engineering students, examining their combined impact on problem‐solving and inquiry‐driven learning. The intervention was designed using the integrate LLMs alongside CAS (ILAC) approach, which structures the inquiry process into key phases, guiding students through exploration, hypothesis testing, and solution validation. Six structured activities were implemented and assessed using quantitative and qualitative methods. Findings reveal that LLMs enhanced conceptual understanding, clarified methodologies, and assisted with command syntax, while CAS ensured computational accuracy and result validation. Many students critically cross‐verified LLM‐generated results with CAS, though some relied solely on LLMs, highlighting the need for better guidance on tool usage. While LLMs fostered engagement, skepticism remained regarding their ability to address deeper mathematical deficiencies. The intervention led to moderate improvements in students' familiarity with AI tools, though its short duration and the use of general‐purpose LLMs limited perceived usefulness. To maximize educational benefits, future implementations should consider longer interventions, advanced training in prompt engineering, and tailored AI solutions.

Self-bootstrapping automated program repair: using LLMs to generate and evaluate synthetic training data for bug repair

Building and delivering high-quality software is critical in software engineering and requires verification and validation processes for end-to-end testing that are reliable, robust, and deliver correct results fast. Manual testing of LLM models, while feasible, is very time-consuming and inefficient and has scalability issues depending on how big the model under test (MUT) is. Recent research and cutting-edge technology innovations in LLM models have deeply influenced software engineering. We need to integrate its impact robustly in areas of model analysis, test automation, model execution, debugging, and report generation.This paper focuses on a framework approach for automated software testing of the LLM models to reduce human interactions and has improved results in a fast, cost-efficient, and time-efficient manner for automated testing methods for industries. The proposed Automation Framework (LLMAutoE2E) leverages and integrates LLMs for testing of different LLM models (like BERT, BART, Hugging Face, and multiple models available to test) to automate the end-to-end execution lifecycle of the LLM models. By leveraging LLMs, companies and industries can generate automated test cases, automated unit test codes, automated integration and end-to-end tests, and automated reporting of the LLM model's execution results. This research emphasizes the potential of the Automation Framework (LLMAutoE2E) for LLM to automate and streamline the overall execution and result generation of the LLM models and the overall testing workflows while addressing challenges in current LLM models testing, its accuracy and scalability for deployments, and reporting. The proposed Automation Framework (LLMAutoE2E) can also automate defect analysis, which improves the software reliability by a manifold and reduces the development cycles for companies. This Research paper details the role of Automation Frameworks for LLMs and how it is transforming QA processes, key methodologies, improving reliability and efficiency, addressing current challenges like model safety, bias detection, and continuous monitoring, and future trends in AI-driven software testing.

This research investigates the impact of prompt engineering on the efficiency, code quality, and security of CRUD (Create, Read, Update, Delete) operations in software development using large language models (LLMs) like ChatGPT. Prompt engineering, which involves crafting specific inputs to guide AI outputs, has become crucial for code generation, debugging, and vulnerability detection tasks. The study addresses three key research questions: identifying the most influential aspects of prompt engineering on CRUD efficiency compared to traditional coding, recognising common error patterns in AI-generated code and preemptive mitigation strategies, and leveraging human-AI collaboration to optimise CRUD application development. This study evaluates 52 relevant papers from 2018 to 2023 through a systematic literature review from Google Scholar, ResearchGate, arXiv, and Sci-Hub. The findings indicate that effective, prompt engineering significantly enhances productivity, reduces development time, and improves code adaptability to complex data structures. Techniques such as role-prompting and chain-of-thought prompting are highlighted for their ability to produce high-quality code, reducing errors and enhancing overall code quality. The review revealed significant results, including identifying prompt engineering techniques that substantially improve the performance of LLMs in CRUD operations. ChatGPT, in particular, excels in generating and debugging code but struggles with more complex tasks and security vulnerabilities. The study emphasises the need for continuous human oversight to ensure security and correctness, addressing challenges such as managing programming variability and the stochastic nature of LLMs.The research concludes that integrating ChatGPT into CRUD operations significantly advances software development, improving efficiency, code quality, and security. However, it is crucial to address its limitations through better prompt engineering and continuous human involvement. Future work will involve developing a fully functional CRUD application using generative AI, further evaluating its practical applications and limitations in software development.

In the context of the rapid development of Large Language Models (LLMs), the field of software development has undergone significant transformations presenting both opportunities and challenges for software development professional cultivation. This study systematically analyzes the applications of LLMs in software development and their impact on this cultivation, exploring the opportunities and challenges in enhancing programming efficiency, promoting personalized learning, improving interdisciplinary skills, and addressing over-reliance on LLMs and related tools. Through literature analysis, this study reviews the impact of LLMs on programming efficiency, code quality, and project management and evaluates the requirements and directions for professional cultivation in response to these changes. The research results indicate that while LLMs bring numerous opportunities, they also pose challenges such as rapid technological updates and a tendency toward over-reliance on tools. Therefore, this study proposes a series of optimized professional cultivation strategies to adapt to the technological developments and industry demands of the new era, thereby enhancing the capability of higher education institutions to cultivate software professionals who meet future needs.

For over a decade, forensic scientists/meta-scientists, publishers, and authors have developed and used software tools to find issues in scientific pre-prints and publications. The recent rise of Large Language Models (LLMs) provides these groups with a whole new range of tools. With this symposium, we will bring together people from four ongoing projects to discuss how the latest LLMs and other AI tools are being developed, evaluated, and used to detect errors in scientific publications: The Black Spatula Project, RegCheck, SocEnRep, and Psy-RAG. These projects involve people from different countries, and academic disciplines as well as people outside academia. The structures, pipelines, and goals of these projects differ. While the Black Spatula Project is an international grassroots initiative driven mostly by volunteer efforts, SocEnRep is a research and infrastructure project funded by the German Research Foundation (DFG), and RegCheck and Psy-RAG are also academic research projects based in Switzerland. The Black Spatula Project wants to make use of LLMs for detecting errors in scientific publications, SocEnRep develops a pipeline for automated reproducibility checks in economics and the social sciences, RegCheck compares preregistration documents to papers, and Psy-RAG combines LLMs and RAG to identify problematic patterns, inconsistencies, and error propagation in psychological research. What all of these projects have in common is that they use AI/LLMs for scalable and (semi-)automated checks of academic publications. We will first present preliminary results, the challenges ahead and potential solutions, with a moderated discussion at the end. Introduction Shakhlo Nematova introduces today's subject matter and speakers

Generative AI systems based on Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering tasks, from code generation to natural language processing. However, a significant gap persists between curated demonstrations and production-grade deployment. This technical report presents a practitioner-driven analysis of LLM limitations encountered during sustained, real-world use across software development workflows. Drawing from multiple case studies—including natural-language-driven development (“vibe coding”), API integration, multi-file refactoring, and general-purpose question answering—we identify and taxonomize four critical failure modes: (1) the Complexity Cliff, where LLM performance degrades non-linearly as task interdependency grows; (2) Context Window Blindness, where finite attention spans cause silent contract violations across distributed codebases; (3) the Memory Illusion, where session discontinuity erases accumulated architectural knowledge; and (4) Confident Hallucination, where models generate plausible but fabricated outputs indistinguishable in tone from correct ones. We formalize the Verification Paradox—an inverse relationship between a user’s need for AI assistance and their capacity to validate its outputs—and propose a practical five-strategy framework for effective human–AI collaboration in software engineering contexts. We further introduce the concept of Contextual Reasoning Failure, evidenced by cases where LLMs optimize for literal query patterns while ignoring situational logic obvious to any human observer. Our findings suggest that current LLMs, while powerful pattern-matching engines, lack the contextual reasoning, persistent memory, and epistemic self-awareness necessary for reliable autonomous operation, and that practitioner expertise remains the critical safeguard against AI-induced defects in production systems.