Some user-defined control functions for more readable APL programs

This article presents Sharing User-Defined Aggregate Function (SUDAF), a declarative framework that allows users to write User-defined Aggregate Functions (UDAFs) as mathematical expressions and use them in Structured Query Language statements. SUDAF rewrites partial aggregates of UDAFs using built-in aggregate functions and supports efficient dynamic caching and reusing of partial aggregates. Our experiments show that rewriting UDAFs using built-in functions can significantly speed up queries with UDAFs, and the proposed sharing approach can yield up to two orders of magnitude improvement in query execution time. The article studies also an extension of SUDAF to support sharing partial results between arbitrary queries with UDAFs. We show a connection with the problem of query rewriting using views and introduce a new class of rewritings, called SUDAF rewritings, which enables to use views that have aggregate functions different from the ones used in the input query. We investigate the underlying rewriting-checking and rewriting-existing problem. Our main technical result is a reduction of these problems to, respectively, rewriting-checking and rewriting-existing of the so-called aggregate candidates , a class of rewritings that has been deeply investigated in the literature.

We present SUDAF (Sharing User-Defined Aggregate Functions), a declarative framework that allows users to formulate UDAFs as mathematical expressions and use them in SQL statements. SUDAF rewrites partial aggregates of UDAFs using built-in aggregate functions and supports efficient dynamic caching and reusing of partial aggregates. Our evaluation shows that using SUDAF on top of Spark SQL can lead from one to two orders of magnitude improvement in query execution times compared to the original Spark SQL.

Speeding up the communications on a cluster using MPI by means of Software Defined Networks

SCOPE is adopted by thousands of developers from tens of different product teams in Microsoft Bing for daily web-scale data processing, including index building, search ranking and advertisement display. A SCOPE job is composed of declarative SQL-like queries and imperative C# user-defined functions (UDFs), which are executed in pipeline by thousands of machines. There are tens of thousands of SCOPE jobs executed on Microsoft clusters per day, while some of them fail after a long execution time and thus waste tremendous resources. Reducing SCOPE failures would save significant resources. This paper presents a comprehensive characteristic study on 200 SCOPE failures/fixes and 50 SCOPE failures with debugging statistics from Microsoft Bing, investigating not only major failure types, failure sources, and fixes, but also current debugging practice. Our major findings include (1) most of the failures (84.5%) are caused by defects in data processing rather than defects in code logic; (2) table-level failures (22.5%) are mainly caused by programmers mistakes and frequent data schema changes while row-level failures (62%) are mainly caused by exceptional data; (3) 93.0% fixes do not change data processing logic; (4) there are 8.0% failures with root cause not at the failure-exposing stage, making current debugging practice insufficient in this case. Our study results provide valuable guidelines for future development of data-parallel programs. We believe that these guidelines are not limited to SCOPE, but can also be generalized to other similar data-parallel platforms.

Many distributed in-memory data-processing systems such as Flink and Spark suffer from serious memory issues, including limited memory resources shared by many users or groups, which aggravates the competition for memory resources. If a user application is allocated with insufficient memory, significant garbage collection overheads will occur at runtime. In contrast, if the user application is provided with a memory space larger than it actually requires, memory resources will be wasted. Therefore, it is important to ensure that a user application is allocated with an appropriate memory size. In a general case, multiple applications process one specific set of data repeatedly. Often, the process logic of applications working on the same dataset is similar; for example, they can perform machine learning or graph computing tasks. This study further reveals that the process logic reflected in application programming interface and user-defined functions also affects the memory usage at runtime. Based on this observation, this paper presents a method for estimating the optimal memory size for newly submitted applications. The proposed approach was implemented on Spark. It utilizes the information of program analysis and historical applications produced when processing data to estimate a proper memory threshold for a newly submitted application based on the similarity of the data path between the new application and a historical one. The results of the experiments conducted to evaluate the method's accuracy of estimating the memory threshold and performance profit demonstrated that the proposed method is able to (1) estimate the required memory threshold with an error of 4% compared to the actual proper memory threshold, (2) guarantee the overall time overhead of estimating to be negligible compared to the execution time of a submitted job, and (3) reduce the execution time of submitted applications by up to 56% compared to when the proposed method is not applied.

The quality of code generated by two commercially available FORTRAN preprocessors is examined. These preprocessors augment FORTRAN with ‘structured’ control structures such as the IF‐THEN‐ELSE and WHILE statements. Two versions of benchmark programs were written and tested, one version produced by the preprocessor, the other, performing the same computation, produced by careful hand‐coding directly in FORTRAN. The code generated by the preprocessor and the code written directly in FORTRAN were then compiled. The resulting object modules were then compared for relative execution time and size. This experiment was repeated on three computers, and five compilers with various optimization levels. The results indicate that a substantial overhead in storage space may be paid by using a preprocessor rather than direct coding in FORTRAN, and that in some cases execution time may be increased somewhat by using a FORTRAN preprocessor.

In unsupervised domain adaptation (UDA), existing methods primarily focus on reducing the distributional discrepancy between domains, often neglecting domain shifts and intra-domain variances. A lightweight intra-class and inter-class domain adaptive network (LIIDAN) is proposed to overcome these challenges and facilitate fault diagnosis across varying working conditions. The model consists of two main components: the lightweight and optimized DarkNet19 (LODN19) module for fault feature extraction and the intra-class and inter-class domain adaptation module (IIDAM). The DarkNet19 model’s optimization is achieved by using lightweight processing to remove selected convolutional and max-pooling layers, which simplify the network structure. In addition, a gated recurrent unit layer is added to capture time-related features, while a multi-head self-attention mechanism is applied to boost the network’s capability in efficiently capturing global dependencies. Subsequently, the IIDAM incorporates an improved adversarial domain classifier, where the domain loss function is defined using the Jensen–Shannon divergence to differentiate whether the extracted features originate from the source or target domain. Next, intra-class and inter-class loss functions are defined using weighted conditional maximum mean discrepancy to minimize intra-class distances and enhance inter-class separation. To balance the model’s migration and recognition capabilities, an adaptive factor is introduced, which refines the overall loss function. Two bearing datasets have shown the efficiency of the algorithm, with an average recognition rate of 99.4% across 12 transfer tasks in the Case Western Reserve University (CWRU) dataset, and an improvement in execution time of 44.9%. An average recognition rate of 97.8% and a 46.7% improvement in execution time were obtained for a dataset containing composite faults from a self-constructed test bench.

MUSIC: A TOOL FOR SIMULATION AND REAL-TIME CONTROL

A simple neuronal network model of the baroreceptor reflex is analyzed. From a control perspective, the analysis suggests a dynamic scheduled control mechanisms by which the baroreflex may perform regulation of the blood pressure. The main objectives of this work are to investigate the static and dynamic response characteristics of the single neurons and the network, to analyze the neuromimetic dynamic scheduled control function of the model, and to apply the algorithm to nonlinear process control problems. The dynamic scheduling activity of the network is exploited in two control architectures. Control structure I is drawn directly from the present model of the baroreceptor reflex. An application of this structure for level control in a conical tank is described. Control structure II employs an explicit set point to determine the feedback error. The performance of this control structure is illustrated on a nonlinear continuous stirred tank reactor with van de Vusse kinetics. The two case studies validate the dynamic scheduled control approach for nonlinear process control applications.

Recently there has been a surge of interest in developing performance debugging tools to help programmers tune their applications for better memory performance [2, 4, 10]. These tools vary both in the detail of feedback provided to the user, and in the run-time overbead of using them. MemSpy [10] is a simulation-based tool which gives programmers detailed statistics on the memory system behavior of applications. It provides information on the frequency and causes of cache misses, and presents it in terms of source-level data and code objects with which the programmer is familiar. However, using MemSpy increases a program's execution time by roughly 10 to 40 fold. This overhead is generally acceptable for applications with execution times of several minutes or less, but it can be inconvenient when tuning applications with very long execution times.This paper examines the use of trace sampling techniques to reduce the execution time overhead of tools like MemSpy. When simulating one tenth of the references, we find that MemSpy's execution time overhead is improved by a factor of 4 to 6. That is, the execution time when using MemSpy is generally within a factor of 3 to 8 times the normal exwution time. With this improved performance, we observe only small errors in the performance statistics reported by MemSpy. On moderate sized caches of 16KB to 128KB, simulating as few as one tenth of the references (in samples of 0.5M references each) allows us to estimate the program's actual cache miss rate with an absolute error no greater than 0.3% on our five benchmarks. These errors are quite tolerable within the context of performance bugging. With larger caches we can also obtain good accuracy by using longer sample lengths. We conclude that, used with care, trace sampling is a powerful technique that makes possible performance debugging tools which provide both detailed memory statistics and low execution time overheads.

Many embedded reactive programs perform computations at different rates, while still requiring the overall application to satisfy very tight temporal constraints. We propose a method to automatically distribute programs such that the obtained parts can be run at different rates, which we call rate desynchronization . We consider general programs whose control structure is a finite state automaton and with a DAG of actions in each state. The motivation is to take into account long-duration tasks inside the programs: these are tasks whose execution time is long compared to the other computations in the application, and whose maximal execution rate is known and bounded. Merely scheduling such a long duration task at a slow rate would not work since the whole program would be slowed down if compiled into sequential code. It would thus be impossible to meet the temporal constraints, unless such long duration tasks could be desynchronized from the remaining computations. This is precisely what our method achieves: it distributes the initial program into several parts, so that the parts performing the slow computations can be run at an appropriate rate, therefore not impairing the global reaction time of the program. We present in detail our method, all the involved algorithms, and a small running example. We also compare our method with the related work.

This paper presents a new control structure for a non-holonomic robot. It consists of two controllers in series: a nonlinear trajectory tracking controller based on robot kinematics and a multivariable quasi-linear controller based on robot dynamics. The Lyapunov theory is used to obtain the nonlinear control law and a frequency domain design approach is employed to find the transfer functions of the quasi-linear controller. The stability and sensitivity analysis criteria are considered to increase the performance of the robot. The stabilisation time and overshoot are considerably reduced. The proposed controller is simple and insensitive to external disturbances. Simulation and real-time test results are used to evaluate the effectiveness of this new control structure.

Energy-based high frequency oscillation control of linear drives

This paper establishes an instantaneous voltage model to illustrate the dynamic characteristics of grid voltage in electromagnetic time scale affected by power-electronic interfaced voltage regulatory devices with two typical categories of control systems, i.e. power-current cascading control and power synchronization control, exemplified by the STATCOM and virtual synchronous condenser (VSCON) respectively. In this paper, a linearized instantaneous voltage model is firstly established to analyze the impact paths of control loops in the STATCOM and VSCON. Based on the model, the dynamic process of grid voltage is divided into three stages. Various factors on the dynamic process of grid voltage are analyzed, especially cascading control loops prior to the functioning of AC voltage control. Moreover, stiffness characteristic is proposed to evaluate the effects of power electronic interfaced voltage regulatory devices with different control structures. In addition, the effectiveness of the stiffness compensation based on the VSCON is validated by the study cases.

Scheduling can require analyzing not only the total computation time of a task, but also the remaining execution time, R(t)/sub /spl Delta/t/, after accumulated time /spl Delta/t. Often a software program's execution time is characterized by a single value (mean). When scheduling is based on partial execution (a common scenario in multimedia systems) a more accurate estimation of remaining time (R(t)/sub /spl Delta/t/) is desired than can be obtained from just the initial mean value, in order to have effective scheduling decisions. The remaining time approach can provide more accurate estimation, and therefore more effective scheduling, in time-sensitive situations. We developed an analytical model for computing expected remaining execution time, (R(t)/sub /spl Delta/t/)~ , of software programs from their execution time and probability distributions. To implement the equations, we further designed an algorithm that computes (R(t)/sub /spl Delta/t/)~ for operating system scheduling applications. We proved that the real time execution complexity of the algorithm is O(1) and is, therefore, independent of the size of the distribution. Our method of more accurate estimate of (R(t)/sub /spl Delta/t/)~ implies expect better scheduling performance in applications where remaining execution time is used, especially in CPU scheduling.