A Dataset for Evaluating Large Language Models in Web Accessibility Auditing and Remediation

EDSA (expert data flow and static analysis tool) which uses static analysis of source code to help gain an understanding of existing code, is described. This may be for the purpose of tracking down a bug or to determine in advance whether an intended change will have any undesirable side effects. In either case, the phase of the development life cycle that is most likely to benefit from a tool like EDSA is the maintenance phase. EDSA provides three kinds of facilities. It helps to browse through code, following either the control flow or data flow, rather than the order in which the code happens to be written. It displays code with unimportant source lines eluded, so that the user can get a more global view of the program. Finally, it provides search management to make it easier to examine all possibilities when browsing. The mechanism used to create and display various views of programs is described and the approach to the user interface is discussed. The state of the current implementation is described. >

Whether the developed software system completed all functions in the software design specification and consistent with the design algorithm is an important work in software testing. Analysis of the source code artificial and verify whether the software system is to meet the requirements of the design specification is laborious job. Testers need to have rich experience in programming and strong ability of algorithm analysis. This paper proposes a method to verify consistency of software implementation and design based on function call path. The method can be summarized as analyze the calling relationship of functions from design specifications and source code, extraction function calling path, generation function cluster model. In documents aspect, the function calling relationship can be got by manual analysis of the design specification, then the designed function cluster model can be generated automatically. We can get function calling relationship, function features by static analysis of source code and the specific implementations of function with those features. Finally, compare the two function cluster model to verify that the consistency of software implementation and design. The experimental results showed that this method can obtain the system function structure and algorithm features accurately, make effective judgment of the consistency of the design specification and the software implementation, provides a new way to verify the consistency of software implementation and design.

In today’s rapidly evolving business landscape, Artificial Intelligence (AI), and specifically Large Language Models (LLMs), are redefining how organizations operate, make decisions, and engage with customers. AI-driven technologies have become indispensable, providing businesses with powerful tools to streamline operations, derive actionable insights from vast data, and foster more meaningful customer interactions. For business leaders, scholars, and practitioners alike, understanding the transformative potential of AI isn’t just advantageous—it’s essential to staying competitive in an increasingly data-driven world.This editorial delves into recent scholarly advancements in LLM applications within business contexts, analyzing studies that explore AI’s potential across various domains, from decision support to creative industries. By introducing a structured framework, this editorial highlights key insights and contributions from recent studies, assessing their value to academia and industry. The following comparative analysis sheds light on how these innovations shape our understanding of AI’s role in business while pointing to future research directions.Puyt and Madsen's (2024) study stands out as a foundational exploration of LLM accuracy, assessing ChatGPT-4's ability to recount the history of the SWOT analysis-a vital business strategy tool. Their findings reveal that, while ChatGPT-4 effectively conveys general concepts, it struggles with detailed historical information, often producing inaccuracies or "hallucinations." This gap underscores the need for LLMs to be trained with verified academic data, particularly for strategic business applications that demand precision. This study not only contributes to the literature by proposing methods to evaluate AI accuracy in historical contexts but also highlights the importance of rigorous information vetting in industry settings where reliability is crucial.In contrast, Raikov et al. (2024) explore a hybrid intelligence model that combines LLM capabilities with explainable AI (XAI) principles to enhance human-machine collaboration. Their approach emphasizes cognitive semantics, improving transparency and decision-making efficiency. The hybrid model's real-time adaptability addresses the needs of complex, regulated industries such as finance and healthcare, where trust in AI decisions is paramount. Academically, this study provides a valuable addition to XAI literature by demonstrating how LLMs can bridge the gap between AI autonomy and human oversight, making it a model for future human-AI interactions in complex business environments.Another significant study by Mariotti and colleagues (2024) examines the integration of LLMs with enterprise knowledge graphs to enhance data-driven decision-making. By enabling organizations to leverage knowledge graphs for more accurate and scalable data retrieval, this research provides a robust framework for businesses seeking efficient knowledge management systems. The academic contribution here lies in advancing the dialogue between LLMs and knowledge graphs, emphasizing ethical data handling and quality standards essential for industry applications. For enterprises, the study offers practical solutions to achieve streamlined data management, balancing automation with privacy and security. 2024) take a different approach, investigating LLMs' role in creative industries, specifically within fashion design. They introduce a hybrid intelligence model that supports creative processes, allowing AI to complement rather than replace human ingenuity. While LLMs in this field demonstrate potential in automating repetitive design tasks and enhancing customer personalization, the study reveals limitations in AI's ability to handle spatial and stylistic nuances. This study's academic contribution lies in promoting human-AI co-creation, inspiring further research into AI applications across diverse creative sectors, including media and marketing.Collectively, these studies not only illuminate LLMs' transformative potential in business but also highlight critical ethical and operational considerations. Ensuring accuracy, transparency, and data privacy are vital to responsibly integrating AI into business workflows. Future research should focus on enhancing LLM accuracy, refining hybrid intelligence models, and exploring creative AI applications, all while maintaining ethical standards. As LLMs evolve, interdisciplinary collaborations will be essential to harness their full potential, making AI an ethical, effective, and innovative force in the business world.

Unit testing is an established practice in professional software development. However, in high-performance computing (HPC) with its scientific applications, it is not widely applied. Besides general problems regarding testing of scientific software, for many HPC applications the effort of creating small test cases with a consistent set of test data is high.We have created a tool called FortranTestGenerator, that significantly reduces the effort of creating unit tests for subroutines of an existing Fortran application. It is based on Capture & Replay (C&R), that is, it extracts data while running the original application and uses the extracted data as test input data. The tool automatically generates code for capturing the input data and a basic test driver which can be extended by the developer to an appropriate unit test. A static source code analysis is conducted, to reduce the number of captured variables. Code is generated based on flexibly customizable templates. Thus, both the capturing process and the unit tests can easily be integrated into an existing software ecosystem.Since most HPC applications use message passing for parallel processing, we also present an approach to extend our C&R model to MPI communication. This allows extraction of unit tests from massively parallel applications that can be run with a single process.

Fuzzing remains to be the most effective method for identifying security vulnerabilities in software. In the context of fuzz testing, the fuzzer supplies varied inputs to fuzz targets, which are designed to comprehensively exercise critical sections of the client code. Various studies have focused on optimizing and developing advanced fuzzers, such as AFL++, libFuzzer, Hongfuzz, syzkaller, ISP-Fuzzer, which have substantially enhanced vulnerability detection in widely used software and libraries. Nevertheless, achieving greater coverage necessitates improvements in both the quality and quantity of fuzz targets. In large-scale software projects and libraries — characterized by numerous user defined functions and data types — manual creation of fuzz targets is both labor-intensive and time-consuming. This challenge underscores the need for automated techniques not only to generate fuzz targets but also to streamline the execution and analysis of their results. In this paper, we introduce an approach to improving fuzz target generation through static analysis of library source code. The proposed method encompasses several key aspects: it analyzes source code structures to accurately construct function calls and generate fuzz targets; it maps fuzzer input data to the corresponding function parameters; it synthesizes compilation information for the fuzz targets; and it automatically collects and analyzes execution results. Experimental results on C/C++ libraries demonstrate that this approach outperforms several existing methods in the field.

Static code analysis is a very important stage in the development and implementation of software, and it needs to be used to obtain a better code. The most complicated in static analysis is the analysis of source code, and further analysis. Analysis of the source code can be done multiple times and expand the set of necessary grammar. The main goal was and remains to provide a solution to reduce the time needed for a global reassessment of static analysis at code level after a change.The main problem when using static code analysis is to build an abstract semantic graph, because each software solution is provided with a separate data warehouse. The proposed solution proposes to use as a database repository using graphs. Thus, the storage mechanisms of the created abstract semantic graph have been simplified, which in turn, in addition to increasing the clarity of the information we store, provides convenient ways to further work with the stored information.The developed solution can transform a rather large source repository into a graphic representation and maintain it later. It is found that the approach is suitable for carrying out code compliance checks and for performing static analysis tests on a graphic representation. This approach also uses advanced file-level detailing, accelerating static analysis. Based on my measurements, the frameworks are fast enough to help their users quickly change the repository of codes.This article confirms the thesis about the possibility of storing an abstract semantic graph in a graph database, and after refinement, if it contains sufficient transformations and requests for language processing, can become a complete transport for communication between various static analysis tools that usually perform one of two the functions of verification either for quality or for vulnerability, thereby making a unified creation of an abstract semantic graph.As an improvement it is necessary to consider the possibility of incremental analysis – the analysis of changes in the code, in order to minimize resource costs for a rather resource intensive operation of the structure of the abstract semantic graph.Manuscript received 09.06.2020

At present, Web application vulnerability safety problems widely exist because of the lack of effective validation and filtering mechanism for untrusted user data. This paper shows the design of the prototype system against SQL injection and cross-site scripting vulnerability. The main steps of the detection are as follows: construct the taint dependency graph for the program by the static analysis of source code; use finite state automata to represent the value of tainted string; verify whether the program has effective safe handling for the user input by matching with the attack pattern and then implement the prototype system for automatic detection on the Java Web program vulnerability based on taint dependency analysis. The experimental results show that the system is comprehensive and accurate for the detection of related vulnerabilities.

Context:Static analysis of source code is a promising opportunity to detect faults and badly designed areas, which will have a negative effect on the overall product quality. In addition, it is inexpensive to integrate it into the development process, because it is easy to automate. Objective:Despite these benefits, static analysis is not as commonly accepted as other quality assurance techniques. To be able to realise its full potential, we need to better understand the problems which prevent software developers from successfully using static analysis tools. Method: To gain new insights we propose to combine the observation methodologies eye tracking, think aloud and questionnaires for a comprehensive investigation on the usage barriers. Results: Our first experimental results with students show new issues: Automated static analysis tools could benefit from a more direct feedback to changes and a more clear guidance of the users. Conclusion:Therefore, we consider the approach as promising and we will conduct further experiments to understand the barriers in static analysis usage.

Many intrusion detection systems monitor process behavior by tracing system calls. Frequent patterns or inherent rules are extracted as features from system call traces of normal process to model the behavior, and any significant deviation from the model is diagnosed as intrusive. Current approaches suffer from heavy modeling complexity in extracting essential features to reduce false alarms. In this paper, we propose a novel approach, which analyzes property of individual system call and its context at semantic level to discover function structures from system call traces efficiently without any static analysis of source code or runtime information. We monitor process behaviors by perceiving such structures as preconditions, which is effective and consistent with mechanism of process execution. Experiments are conducted on two sets of intrusion detection data and the results show that our approach is feasible and effective.

State transition machines are high-level behavior descriptions often used as modeling tools for the design and implementation of a large class of software systems. Some of the state transition machine implementation approaches are such that the static structure of the resulting code closely matches that of the original state transition machines. Therefore, having a representation of the original state transition machines is likely to improve the corresponding code understandability. We present an approach supported by a prototype tool, to extract state transition machines by static analysis of source code. An objective of this work is to enhance program comprehension with visual representations of the behavior of the programs being analyzed.

In modern software engineering, API (Application Programming Interface) is widely used to develop applications rapidly by reusing data structure, frameworks, class libs, and etc. However, due to the considerable number of interfaces, lack of documents and timely maintenance and updates, APIs are often used in a wrong way. Therefore, it has become an important problem to detect API misuse in an automatic way. Many existing automatic API detecting methods do not make full use of APIs’ potential semantic information and independent integrity of each API. In this paper, we employ Stacked LSTM to learn the API usage specification to detect the API misuse defects. Specifically, first, we obtain ACSG (API Call Syntax Graph) through the static analysis of source code. And then, based on ACSG, we generate API sequences, and transform the sequences into for training. Third, in order to represent the APIs in a semantic way, we apply word2vec as a pre-training model to embed features of each API. Though the stacked LSTM model, we regard embedding precious API sequence as the input to model the API use specifications and discover the potential API misuse defects by judging whether the next API is in the output (API probability list) or not. We design experiments to evaluate the effectiveness our method with Java Cryptography APIs and their used code, and the results show the advancement of our proposed method.

Static analysis of source code is considered to be a powerful tool for detecting potential security vulnerabilities. However, only limited information regarding the current quality of static analysis tools exist. A public assessment of the capabilities of the competing approaches and products is not available. Also, neither a common benchmark nor a standard evaluation procedure has yet been defined. In this paper, we propose a general methodology for systematically evaluating static analysis tools. We document the design of an automatic execution and evaluation framework to support iterative test case design and reliable result analysis. Furthermore, we propose a methodology for creating test cases which can assess the specific capabilities of static analysis tools on a very fine level of detail. We conclude the paper with a brief discussion of our experiences which we collected through a practical evaluation study of six commercial static analysis products.

Multitasking on FPGA is a method allowing multiple users to share a reconfigurable fabric, thus improving the flexibility of hardware task management. However, current multitasking schemes bring with it considerable performance degradation and several issues, that can be solved. In this paper, we first present a multitasking scheme based on checkpointing in the hardware description language (HDL) level. The scheme can eliminate the need for reading the bitstream back, thus reducing the task switch latency. We then propose a new HDL-based checkpointing architecture for FPGA computing. Third, we propose a static analysis of the original HDL source code in order to reduce the hardware overhead caused by the checkpointing insertion. Our evaluations show that the proposed architecture with the static analysis can reduce up to 50% of the LUT overhead, compared with the tree-based checkpointing architecture. The checkpointing architecture causes small degradation in maximum clock frequency (1.65% on average), while it consumes low memory footprints. Comparisons with previous multitasking schemes highlight the advantages of our scheme.

Parallelization of an existing sequential application to achieve a good speed-up on a data-parallel infrastructure is quite difficult and time consuming effort. One of the important steps towards this is to assess whether the existing application in its current form can be parallelized to get the desired speedup. In this paper, we propose a method of analyzing an existing sequential source code that contains data-parallel loops, and give a reasonably accurate prediction of the extent of speedup possible from this algorithm. The proposed method performs static and dynamic analysis of the sequential source code to determine the time required by various portions of the code, including the data-parallel portions. Subsequently, it uses a set of novel invariants to calculate various bottlenecks that exists if the program is to be transferred to a GPGPU platform and predicts the extent of parallelization necessary by the GPU in order to achieve the desired end-to-end speedup. Our approach does not require creation of GPU code skeletons of the data parallel portions in the sequential code, thereby reducing the performance prediction effort. We observed a reasonably accurate speedup prediction when we tested our approach on multiple well-known Rodinia benchmark applications, a popular matrix multiplication program and a fast Walsh transform program.

BackgroundThe exponential growth of medical knowledge presents a paradox for modern medical education. While access to information is immediate, applying it in a clinically meaningful way remains a challenge. Large language models (LLMs), such as ChatGPT, are widely used for information retrieval, yet their role in dynamic, high-pressure clinical learning remains poorly understood.ObjectiveThis study aims to evaluate whether unstructured access to an LLM improves decision-making, teamwork, and confidence in trauma education for medical students.MethodsThis randomized controlled pilot study involved 41 final-year medical students participating in a trauma simulation session. Students self-selected into teams of 4 to 6 and were randomized to either an LLM-assisted group (ChatGPT-4o mini) or a control group without LLM access. All teams completed 18 video-based trauma scenarios requiring time-sensitive clinical decisions. Prompting was unrestricted. Confidence and trauma exposure were assessed using pre- or postquestionnaires. Facilitators rated teamwork (1-5), decision accuracy, and response times. Knowledge retention was measured 4 weeks later via an online quiz.ResultsConfidence in trauma management improved in both groups (P<.001), with larger gains in the non-LLM group (P=.02). LLM support did not enhance the decision accuracy or speed and was associated with longer response times in some complex cases. Teams without LLMs demonstrated more active discussion and scored higher in teamwork ratings (median 5.0 [IQR 5.0-5.0] vs median 3.5 [IQR 3.0-4.5]; P=.08). Students primarily used the LLM for fact-checking but reported vague or overly general responses. Knowledge retention was high across both groups and did not differ significantly (P=.33).ConclusionsWhile students appreciated the inclusion of artificial intelligence (AI), unstructured LLM use did not improve performance and may have disrupted the group reasoning. The use of non-English prompting likely contributed to lower AI performance, underscoring the importance of language alignment in LLM applications. This pilot study highlights the need for structured AI integration and targeted instruction in AI literacy. Simulation-based trauma education proved effective and well received, but optimizing the educational value of LLMs will require thoughtful curricular design. Further studies with more students are needed to define best practices for LLM use in clinical education.