A design framework for distributed data acquisition and triggering systems in high energy physics experiments

ATLAS is a Physics experiment that explores high-energy particle collisions at the Large Hadron Collider at CERN. It uses tens of millions of electronics channels to capture the outcome of the particle bunches crossing each other every 25 ns. Since reading out and storing the complete information is not feasible (∼100 TB/s), ATLAS makes use of a complex and highly distributed Trigger and Data Acquisition (TDAQ) system, in charge of selecting only interesting data and transporting those to permanent mass storage (∼1 GB/s) for later analysis. The data reduction is carried out in two stages: first, custom electronics performs an initial level of data rejection for each bunch crossing based on partial and localized information. Only data corresponding to collisions passing this stage of selection will be actually read-out from the on-detector electronics. Then, a large computer farm (∼17 k cores) analyses these data in real-time and decides which ones are worth being stored for Physics analysis. A large network allows moving the data from ∼2000 front-end buffers to the location where they are processed and from there to mass storage. The overall TDAQ system is embedded in a common software framework that allows controlling, configuring and monitoring the data taking process. The experience gained during the first period of data taking of the ATLAS experiment (Run I, 2010-2012) has inspired a number of ideas for improvement of the TDAQ system that are being put in place during the so-called Long Shutdown 1 of the Large Hadron Collider (LHC), in 2013/14. This paper summarizes the main changes that have been applied to the ATLAS TDAQ system and highlights the expected performance and functional improvements that will be available for the LHC Run II. Particular emphasis will be put on the evolution of the software-based data selection and of the flow of data in the system. The reasons for the modified architectural and technical choices will be explained, and details will be provided on the simulation and testing approach used to validate this system.

The ATLAS experiment at the Large Hadron Collider at CERN relies on a complex and highly distributed Trigger and Data Acquisition (TDAQ) system to gather and select particle collision data at unprecedented energy and rates. The TDAQ is composed of three levels which reduces the event rate from the design bunch-crossing rate of 40 MHz to an average event recording rate of about 200 Hz. The first part of this paper gives an overview of the operational performance of the DAQ system during 2011 and the first months of data taking in 2012. It describes how the flexibility inherent in the design of the system has be exploited to meet the changing needs of ATLAS data taking and in some cases push performance beyond the original design performance specification. The experience accumulated in the TDAQ system operation during these years stimulated also interest to explore possible evolutions, despite the success of the current design. One attractive direction is to merge three systems - the second trigger level (L2), the Event Builder (EB), and the Event Filter (EF) - within a single homogeneous one in which each processing node executes all the steps required by the trigger and data acquisition process. Appealing aspects of this design are: a simplification of the software architecture and of its configuration, a better exploitation of the computing resources, the caching of fragments already collected for L2 processing, the automated load balancing between L2 and EF selection steps, the sharing of code and services on HLT nodes. Furthermore, the full treatment of the HLT selection on a single node allows more flexible approaches, for example “incremental event building” in which trigger algorithms progressively enlarge the size of the analyzed region of interest, before requiring the building of the complete event. To spot possible limitations of the new approach and to demonstrate the benefits out-lined above, a prototype has been implemented. The preliminary measurements are positive and further tests are scheduled for the next months.

High-speed serial links for trigger and data acquisition (TDAQ) systems in High Energy Physics (HEP) experiments are often implemented by means of Static RAM-based Field Programmable Gate Array (SRAM-based FPGAs). However, due to the sensitivity of SRAM-based FPGA to radiation-induced upsets, their usage is limited to off-detector electronics. In order to benefit of SRAM-based FPGAs reprogrammability and high performance in the data transmission, also in a harsh radiation environment, many techniques have been developed: memory configuration correction (memory scrubbing), hardware redundancy (e.g. Triple Modular Redundancy, TMR) and information redundancy (e.g. Forward Error Correcting Code, FEC).In this paper we describe the architecture of the bidirectional serial link running at 6.25 Gbps we implemented on the Xilinx Kintex-7 SRAM-FPGA devices. With the aim of using the link also in a radiation environment, we chose a robust error correction scheme for the line coding, we designed and incorporated a configuration scrubbing in the link architecture and we explored different TMR strategies.We present a link which offers several original features. It can repair itself thanks to the custom embedded scrubber we designed. It is also capable of self-adjusting its line code scheme depending on the measured error faults and transmitted errors. We show the effective Bit Error Ratio as a function of the measured induced upsets. Besides, we present the performance of the link in terms of mean time between failures (MTBF) and mean time between loss of lock.

This chapter introduces a programming environment for SCI clusters that takes advantage of both parallel and distributed programming paradigms. It aims at helping programmers to design high performance applications based on the assembling of generic software components. This environment is based on CORBA (Common Object Request Broker Architecture), with our own extensions to support parallelism across several cluster nodes within a distributed system. Our contribution concerns extensions to support a new kind of object, which we call a parallel CORBA object (or parallel object), as well as the integration of message-passing paradigms, mainly MPI, within a parallel object. These extensions exploit as much as possible the functionality offered by CORBA and require few modifications to an available CORBA implementation. This paper reports on these extensions and the description of a runtime system, called Cobra, which provides resource allocation services for the execution of parallel objects.

The main goal of the NA62 experiment at CERN SPS is to measure the branching ratio of the ultra-rare K+→π+νν decay, collecting about 100 events in two years of data taking to test the Standard Model of Particle Physics. Readout uniformity of sub-detectors, scalability, efficient online selection and lossless high rate readout are key issues. The TEL62 boards are the common blocks of the NA62 Trigger and Data AcQuisition (TDAQ) system. TEL62s process and store hits coming from the subdetectors in a buffer according to their timestamp, extracting only those requested by the trigger system, which merges trigger primitives also produced by TEL62s. The complete dataflow and firmware organization are described.

In the trigger and data acquisition (TDAQ) system for the ATLAS project authorization of users will be an important task. The main goal of the authorization will be to reduce the chance of potentially dangerous actions being made by mistake. An access management (AM) component is being developed within the TDAQ to handle these issues. This paper presents the design and implementation of the component. It also describes the authorization model used and how authorization data is stored and administrated for the system

The high-precision measurements of detectors in a High-Energy Physics (HEP) experiment need a continuous sampling of recorded events during collisions. Therefore, significant hardware changes are required to do online data processing due to a large amount of data generated by such detectors. Because of all these changes, a new Detector Control System (DCS) design is required. This paper presents a definition of the software requirements to be considered during the design, integration, and operation of a detector’s DCS into physics data-taking, for continuous and non-continuous measurement conditions in the experiments. For this, the main operating processes, elements, characteristics, and guidelines of the DCS in the most important HEP experiments around the world were analyzed. Additionally, characteristics, functional and non-functional requirements, and use-cases of the main actors involved in the different processes of this control system software are defined. A visual modeling and design tool based on Unified Modeling Language (UML) is used to obtain a description of these requirements.

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> During 2006 and spring 2007, integration and commissioning of trigger and data acquisition (TDAQ) equipment in the ATLAS experimental area has progressed. Much of the work has focused on a final prototype setup consisting of around eighty computers representing a subset of the full TDAQ system. There have been a series of technical runs using this setup. Various tests have been run including those where around 6 k Level-1 preselected simulated proton–proton events have been processed in a loop mode through the trigger and dataflow chains. The system included the readout buffers containing the events, event building, second level and third level trigger processors. Aspects critical for the final system, such as event processing times, have been studied using different trigger algorithms as well as the different dataflow components. </para>

Distributed computing applications rely on transmissions of messages between processes. However, as workstations are not intended to manage this kind of communications, it is necessary to use some communication tools known as message passing libraries. Currently, MPI (Message Passing Interface) and PVM (Parallel Virtual Machine) are the most used. In addition, the need for interoperability among the rapidly proliferating amount of hardware and software led to the definition of CORBA (Common Object Request Broker Architecture), which standardizes the execution support of distributed object applications. This architecture offers good interoperability and encapsulation support. Moreover, its method invocation mechanism, offers several features and can be used as a good communication model. Consequently, the use of such an architecture for distributed and grid computing applications seems interesting and promising. However, CORBA is always criticized on the performance level. By taking account of all these elements, a performance evaluation of CORBA seems necessary. The aim of the paper is to present benchmark results of some CORBA implementations (ORBacus and TAO) and to compare them with those of the two message passing libraries MPI (MPICH and LAM) and PVM, by using a Network of WorkStations (NOW).

The Trigger and Data Acquisition (TDAQ) system of the ATLAS experiment is a very complex distributed computing system, composed of more than 20000 applications running on more than 2000 computers. The TDAQ Controls system has to guarantee the smooth and synchronous operations of all the TDAQ components and has to provide the means to minimize the downtime of the system caused by runtime failures. During data taking runs, streams of information messages sent or published by running applications are the main sources of knowledge about correctness of running operations. The huge flow of operational monitoring data produced is constantly monitored by experts in order to detect problems or misbehaviours. Given the scale of the system and the rates of data to be analyzed, the automation of the system functionality in the areas of operational monitoring, system verification, error detection and recovery is a strong requirement. To accomplish its objective, the Controls system includes some high-level components which are based on advanced software technologies, namely the rule-based Expert System and the Complex Event Processing engines. The chosen techniques allow to formalize, store and reuse the knowledge of experts and thus to assist the shifters in the ATLAS control room during the data-taking activities.

A growing class of real-time systems require end-to-end support for various quality-of-service (QoS) aspects, including bandwidth, latency, jitter and dependability. Applications include command and control, manufacturing process control, videoconferencing, large-scale distributed interactive simulation, and testbeam data acquisition. These systems require support for stringent QoS requirements. To meet this challenge, developers are turning to distributed object computing middleware, such as the Common Object Request Broker Architecture, an Object Management Group (OMG) industry standard. In complex real-time systems, DOC middleware resides between applications and the underlying operating systems, protocol stacks and hardware. CORBA helps decrease the cycle time and effort required to develop high-quality systems by composing applications using reusable software component services rather than building them entirely from scratch. The Real-Time CORBA specification includes features to manage CPU, network and memory resources. The authors describe the key Real-Time CORBA features that they feel are the most relevant to researchers and developers of distributed real-time and embedded systems.

The paper describes our SoftDock system, which is a distributed model management system to support collaborative software development. SoftDock leverages the team development, tool interoperability, and model continuity across development lifecycle. It supports Unified Modeling Language (UML) to specify software models, and maintains the model information with UML eXchange Format (UXF), which is a XML-based model interchange format. UXF-encoded model descriptions are shared and managed through the Document Object Model (DOM) interface implemented on top of CORBA (Common Object Request Broker Architecture) middleware. The paper describes the SoftDock architecture, system organization, and design considerations in managing UXF descriptions in the distributed environment.

In High Energy Physics (HEP) experiments, it is useful for physics analysis and outreach if the event display software can provide sophisticated visualization effects. Unity is professional software that provides 3D modeling and animation production. GDML format files are commonly used for detector descriptions in HEP experiments. In this work, we present a method for automating the import of GDML files to the Unity engine modeling through geometric transformation. With the conversion of the file format, the complicated geometric description of the HEP detector in a GDML file can be imported directly into the Unity engine using an FBX file. This format conversion was applied to BESIII experiments to achieve an automated geometry conversion to build geometry models in the Unity engine.

Interest in many-core architectures applied to real time selections is growing in High Energy Physics (HEP) experiments. In this paper we describe performance measurements of many-core devices when applied to a typical HEP online task: the selection of events based on the trajectories of charged particles. We use as benchmark a scaled-up version of the algorithm used at CDF experiment at Tevatron for online track reconstruction - the SVT algorithm - as a realistic test-case for low-latency trigger systems using new computing architectures for LHC experiment. We examine the complexity/performance trade-off in porting existing serial algorithms to many-core devices. We measure performance of different architectures (Intel Xeon Phi and AMD GPUs, in addition to NVidia GPUs) and different software environments (OpenCL, in addition to NVidia CUDA). Measurements of both data processing and data transfer latency are shown, considering different I/O strategies to/from the many-core devices.

The ATLAS experiment at CERN has planned many upgrades, in order to cope with the instantaneous luminosity that will be delivered by the Large Hadron Collider (LHC) machine in the next decade. These upgrades will allow the ATLAS experiment to keep or improve its current performance for Run 3, expected to start in 2021, and for the High Luminosity LHC program (HL-LHC) expected to start in ∼2027. For Run 3, we expect an instantaneous luminosity of 2-3 × 1034 cm−2 s−1 and an average pileup of ~80 collisions/bunch crossing, while for the HL-LHC phase, an instantaneous luminosity of 7.5 × 1034 cm−2 s−1 and an average pileup of ~200 collisions/bunch crossing are foreseen. Although the upgrades are essential to achieve the physics goals, they represent a huge challenge for the detector, trigger and data acquisition (TDAQ) systems. The status of the upgrade activities is presented.