Quality Assessment for Large-Scale Industrial Software Systems: Experience Report at Alibaba

In the context of the rapid development of information technologies, software quality is becoming critical for the successful operation of organizations in various industries. The growing complexity of modern software solutions requires the involvement of highly qualified specialists in software testing and quality assessment, capable of effectively identifying shortcomings and ensuring that the product meets established standards. At the same time, assessing the level of competence of such experts remains a difficult task, which is often based on subjective criteria and methods. The relevance of the study is due to the acute need of the modern IT market for objective tools for assessing the professional level of specialists, especially in the field of software quality assurance. Traditional approaches to qualification assessment, such as interviews, test tasks or resume analysis, often do not provide a complete and objective picture of the expert's competence. This problem becomes especially acute in the conditions of the global labor market, when companies are forced to evaluate specialists remotely, relying only on a limited set of data on their experience and skills. Today, software has become an integral part of many areas of our everyday life - from automation and optimization of production processes to creating comfort for an individual. The object of the study is the process of determining the level of competence of experts in software quality assessment. The subject of the study is a mathematical model for calculating the level of competence of an expert. The practical value of the results of the work is determined by the possibility of using the developed system by HR managers for effective selection of specialists, by heads of QA departments for the formation of balanced testing teams, by certification centers for objective assessment of competence, as well as by the experts themselves for planning their own professional development. Conclusion the developed mathematical model for calculating the level of competence of an expert allows you to reduce the time for assessing the competence of specialists, minimize the influence of subjective factors when making personnel decisions, and optimize the distribution of human resources in software development and testing projects.

In the past few years, research has been actively carried out in an attempt to improve the quality and reliability of large-scale software systems. Although progress has been made on the formal proof of program correctness, proving large-scale software systems correct by formal proof is still many years away. Automated software tools have been found to be valuable in improving software reliability and attacking the high cost of software systems. This paper attempts to describe some main features of automated software tools and some software evaluation systems that are currently available.

In large-scale software systems that integrate many components originating from different vendors, the understanding of the functional importance of the components is critical for the dependability of the system. However, in general, gaining such understanding is difficult. Here we describe the application of the combination of dynamic analysis and network analysis to large-scale software systems with the aim to determine methods of classes that are functionally important with respect to a given functionality of the software. We use as a test case the Google Chrome and predict functionally important methods in a weak sense in the context of usage scenarios. We validate the predictions using mutation testing and evaluate the behavior of the software following the mutation change. Our results indicate that network analysis metrics based on measurement of structural integrity can be used to predict methods of classes that are functionally important with respect to a given functionality of the software system.

Years of planning went into the preparation of ICPE 2020 in Edmonton. There was great excitement in bringing ICPE back to North America after a while and great anticipation for productive interactions between the participants of ICPE and members of the various SPEC working groups. But, alas, the only certain thing about life is its uncertainty. The arrival of the COVID-19 pandemic led to the cancelation of the face-to-face conference in Edmonton. ICPE proceeded with the publication of these proceedings and with the availability of video presentations and slides linked to the conference program in the website. The great program for ICPE 2020 was made possible thanks to the many authors that submitted contributions to the advancement of performance modeling and engineering and many other related topics. We were anticipating great interest in the strong line of keynote speakers. The Canadian-based line up is a coincidence but it would have been a great way to celebrate the contributions from Canadian universities in this area. Gail Murphy from the University of British Columbia planned to share her experiences in increasing the productivity of software development. Sebastian Fishmeister from the University of Waterloo would talk about what they learned from mining traces from the design of embedded software systems. Ahmed Hassan from Queens University intended talk about assessing and testing large-scale software systems. There is much work that goes into putting together a complex program such as ICPE. Besides the contributing authors and presenters, many volunteering hours were dedicated by chairs, programcommittee members, local organizers, local volunteers, steering-committee members and many others. On behalf of the ICPE community we thank you all for your effort and dedication. We thank for the generous sponsorship from the Standard Performance Evaluation Corporation (SPEC). Even though ICPE could not go ahead as planned because of the 2020 pandemic, we hope that the availability of video presentations and the publication of these proceedings will lead to productive online interactions between researchers.

Putting Requirements and Quality at the Core of Global Service Delivery: Current Efforts and Future Plans at Pace University

Dealing with clones is a common problem in large-scale software projects. While most of the research in this area is focused on detecting, and investigating the reasons of clones, little research has been done on the use of clone technology in large-scale industrial software. In this paper we present an in-depth case study of cloning in Command and Control Software Framework (CCSF) developed by HAVELSAN, a large software company in the Turkish defense industry. We proposed a practical classification schema for clones based on their removal strategies. A detailed analysis on categorization of clones, and lessons learned in clone management of large-scale industrial software systems are presented.

Software quality issues play a dominant role in the development of large scale software systems. High quality software enhances reuse potential and facilitates software maintenance activities. However, at present, software quality assessment is a labor intensive, error prone, time consuming process. Automated quality assessment of software systems is a cost effective alternative to ensure compliance with system quality objectives. This paper discusses how reverse engineering technology can be leveraged to solve the needs of software quality assurance teams.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Developing a good software system is a very complex task. In order to produce a good software product, several measures for software quality attributes need to be taken into account. System complexity measurement plays a vital role in controlling and managing software quality because it generally affects the software quality attributes like software reliability, software testability and software maintainability. Thus, software quality assurance (SQA) needs to be addressed keeping in view the new strategies, tool, methodologies and techniques applicable to software development life cycle.This paper is primarily aimed at revisiting and examining peculiar aspects of software development process that affect software quality management process. These aspects of software development process include software reliability measurement, ISO approach applicable to software quality and some aspects related to software testing improvement. Software testing and evaluation methods/tools/techniques do not guarantee effective testing and ensure high software quality. The way to improve the effectiveness of testing is to improve the attitude of software developers towards testing.In this paper, all these factors affecting software quality management have been discussed as well as all the possible improvements have been suggested. The results of this paper may be quite helpful to the researchers in quantifying the specific measuring tools for these software quality attributes.

As the evolution of applications, software systems become more and more complex. However, it is a challenge to deliver easy-to-use and reliable large-scale distributed software systems. This paper first reviews current requirements on software systems and the progress made and then reports our effort in delivering a more usable and reliable large-scale distributed software development framework (iFrame) to support e-research. Bearing human collaboration in the first place, iFrame synthesizes peer-to-peer (P2P), service-oriented architecture (SOA), Web 2.0 and semantic Web into grid: P2P is deployed to construct an interconnected environment to enhance reliability and scalability, SOA and semantic Web are exploited to deliver concrete business functionality and prompt interoperability, whereas Web 2.0 is utilized to devise more usable interface.

An effective system regression testing for consecutive releases of very large software systems, such as modern telecommunications systems, depends considerably on the selection of test cases for execution. Classification models can classify, early in the test planning phase, those test cases that are likely to detect faults in the upcoming regression test. Due to the high uncertainties in regression test, classification models based on fuzzy logic are very useful. Recently, methods have been proposed for automatically generating fuzzy if-then rules by applying complicated rule generation procedures to numerical data. In this research, we introduce and demonstrate a new rule-based fuzzy classification (RBFC) modeling approach as a method for identifying high effective test cases. The modeling approach, based on test case metrics and the proposed rule generation technique, is applied to extracting fuzzy rules from numerical data. In addition, it also provides a convenient way to modify rules according to the costs of different misclassification errors. We illustrate our modeling technique with a case study of large-scale industrial software systems and the results showed that test effectiveness and efficiency was significantly improved.

Feature models are frequently used to capture the knowledge about configurable software systems and product lines. However, feature modeling of large-scale systems is challenging as many models are needed for diverse purposes. For instance, feature models can be used to reflect the perspectives of product management, technical solution architecture, or product configuration. Furthermore, models are required at different levels of granularity. Although numerous approaches and tools are available, it remains hard to define the purpose, scope, and granularity of feature models. In this paper we thus present experiences of developing feature models for two large-scale industrial automation software systems. Specifically, we extended an existing feature modeling tool to support models for different purposes and at multiple levels. We report results on the characteristics and modularity of the feature models, including metrics about model dependencies. We further discuss lessons learned during the modeling process.

The research topic focuses on the characterization of software quality considering the main software elements such as people, process and product. Many attributes (size, language, testing techniques etc.) probably could have an effect on the quality of software. In this thesis we aim to understand the impact of attributes of three P's (people, product, process) on the quality of software by empirical means. Software quality can be interpreted in many ways, such as customer satisfaction, stability and defects etc. In this thesis we adopt ‘defect density' as a quality measure. Therefore the research focus on the empirical evidences of the impact of attributes of the three P's on the software defect density. For this reason empirical research methods (systematic literature reviews, case studies, and interviews) are utilized to collect empirical evidence. Each of this research method helps to extract the empirical evidences of the object under study and for data analysis statistical methods are used. Considering the product attributes, we have studied the size, language, development mode, age, complexity, module structure, module dependency, and module quality and their impact on project quality. Considering the process attributes, we have studied the process maturity and structure, and their impact on the project quality. Considering the people attributes, we have studied the experience and capability, and their impact on the project quality. Moreover, in the process category, we have studied the impact of one testing approach called ‘exploratory testing' and its impact on the quality of software. Exploratory testing is a widely used software-testing practice and means simultaneous learning, test design, and test execution. We have analyzed the exploratory testing weaknesses, and proposed a hybrid testing approach in an attempt to improve the quality. Concerning the product attributes, we found that there exist a significant difference of quality between open and close source projects, java and C projects, and large and small projects. Very small and defect free modules have impact on the software quality. Different complexity metrics have different impact on the software quality considering the size. Product complexity as defined in Table 53 has partial impact on the software quality. However software age and module dependencies are not factor to characterize the software quality. Concerning the people attributes, we found that platform experience, application experience and language and tool experience have significant impact on the software quality. Regarding the capability we found that programmer capability has partial impact on the software quality where as analyst capability has no impact on the software quality. Concerning process attributes we found that there is no difference of quality between the project developed under CMMI and those that are not developed under CMMI. Regarding the CMMI levels there is difference of software quality particularly between CMMI level 1 and CMMI level 3. Comparing different process types we found that hybrid projects are of better quality than waterfall projects. Process maturity defined by (SEI-CMM) has partial impact on the software quality. Concerning exploratory testing, we found that exploratory testing weaknesses induce the testing technical debt therefore a process is defined in conjunction with the scripted testing in an attempt to reduce the associated technical debt of exploratory testing. The findings are useful for both researchers and practitioners to evaluate their projects

Software systems play a key role in most businesses nowadays. Building robust, reliable and scalable software systems require going through a software production cycle (or process). However, it has been noticed that software systems are subjected to changes, whether those amendments are important or not. Those changes to software systems are viewed as a considerable issue in software engineering; they are considered as a burden and cost a lot, especially in cases such as enterprises and large-scale software systems. This study aims to identify the reasons that cause software changes and suggest remedies for those reasons. We survey the opinions of experts such as technical managers, team leaders, and senior developers. We collected 81 responses to our questionnaire, which aimed to establish common software development practices in the local industry. We also conducted 16 semi-structured interviews targeting the most senior experts, in which we directly discussed the reasons and remedies for software changes. Our results highlight the most influential reasons that cause changes to software systems, such as changes to user requirements, requests for new features, software development methodology, solving bugs, refactoring, and weak user experience design. Most importantly, the study solicited solutions that can reduce the need for software changes

The purpose of this study was to identify fault-prone functions that are likely to contain faults in a given software system. Five metrics were used: Di, an internal design metric which incorporates factors related to a function's internal structure; De, an external design metric which focuses on a function's external relationships to the rest of the software system; D(G), a composite design metric which is a linear combination of Di and De; and the union and intersection of Di, De, and D(G). Since the system being considered was already developed, a very important aspect of our study was to extract the design information directly from the source code rather than from the corresponding design documentation which may not exist or, if it does exist, it may be incomplete, difficult to understand, or not updated. To make the analysis more accurate and efficient, a metric analysis tool (χMetrics) was implemented. We conducted experiments using χMetrics on part of a distributed software system, written in C, with a client–server architecture, and identified a small percentage of its functions as good candidates for fault-proneness. Files containing these functions were then validated by the real defect data collected between a recent major release and its subsequent release for their fault-proneness. The results indicate that our metrics are good indicators of fault-prone functions. Two extra experiments were also conducted to show that function size cannot replace any of our metrics; and where the function size was factored out our metrics performed better than the normalized metrics. The important benefit of our metrics is that they help project managers determine where additional testing effort should be spent and possibly which fault-prone functions should be assigned to more experienced programmers if modifications are required. Copyright © 2000 John Wiley & Sons, Ltd.

: With regard to software testing in DoD, we can summarize our conclusions in two fundamental ideas. First, DoD knows how to produce quality software at low cost. This is because organizations such as DoD STEP, Army STEP, and Software Engineering Institute have already researched and documented policies for DoD. A few commercial software developers practice many of the DoD policies and directives now, and produce quality software (for example, IBM FSC Houston). Second, quality cannot be tested into software. Only a well-defined, well-disciplined process with a continuous improvement cycle can ensure software quality. However, testing cannot be underestimated. Systematic testing activities that detect error earliest in the life cycle are necessary to drive process improvement and optimize the development of quality software. Such testing methods as formal inspection find defects early. This enables cost- effective error resolution, identification and removal of defect causes, and thus, prevention of future defect insertion. If practiced with discipline, such methods can evolve a self-correcting software development process that is stable, modeled, measured, and therefore, predictable. This development process engineers quality software faster at reduced cost.... Software testing, Software quality, Formal inspection, Quality control.