for pulic for researcher English
Feature Story
space >Top >Feature Story >this page
last update: 10/06/16   
space space
  New network-based DAQ framework may simplify
life for scientists

The data acquisition system (DAQ) team at KEK’s Detector Technology
Project is now offering the particle experiment community a standardized
data acquisition system (DAQ) framework of both hardware and software.
Learn here about the innovative, network-based readout electronics, and
the all-in-one DAQ-Middleware.

space space
Related Link:
DAQ-Middleware (Japanese)
RT-Middleware Official Page
KEK Detector Technology Project
J-PARC

Related Issue:
NOVA: neutron spectroscopy for a better environment

Paper:
Cosmic-ray muon imaging of magma in a conduit: Degassing process of Satsuma-Iwojima Volcano, Japan

Hyper Suprime-Cam: CCD readout electronics

Hardware-Based TCP Processor for Gigabit Ethernet

Space Saving and Power Efficient Readout System for Cosmic-Ray Muon Radiography

Development of a data acquisition sub-system using DAQ-Middleware

Functionality of DAQ-Middleware

pdf
PDF Version

(6.6MB)
 
group
space
The data acquisition system (DAQ) project team members at KEK’s detector technology project (DTP). From left: the project leader Dr. Tomohisa Uchida, Hiroshi Sendai, Kazuo Nakayoshi, Eiji Inoue, and the former project leader Dr. Yoshiji Yasu.
 
All radiation experiments in Physics use an array of detectors and analysis tools. The detectors sense photons or other particles, and produce the information needed for analysis. Everything that lies in between the detectors and the data analysis tools is part of the data acquisition system (DAQ). Although DAQ systems are not often discussed, they are a key part of many experiments, and often demand significant time and expertise to build.

A dream of both the former and current leaders of the DAQ project in the KEK Detector Technology Project (DTP), Dr. Yoshiji Yasu and Dr. Tomohisa Uchida, has been to standardize all necessary components of DAQ systems in order to provide nonprofessional users easy access to high performance DAQ systems. The next-generation DAQ project was the realization of their dream. The innovative feature of their DAQ package is the marriage of the hardware and software components. This is a challenge for DAQ scientists as some of the necessary software tools depend on the type of detectors. The DAQ team developed a complete, general-purpose package of DAQ to be used by scientists of many disciplines who are not necessarily DAQ experts.

The scheme has seen many successes since its inception in 2006. Their standardized, fast, and easy-to-use DAQ package has now been installed at many beamlines at the Material and Life Facility (MLF) at the Japan Proton Accelerator Complex (J-PARC) in Tokai, and has many potential applications in other fields of experimental particle physics.
 
daq
space
The DAQ framework developed by the DTP DAQ team includes a hardware component and a software component. The hardware consists of silicon transmission control protocol circuit boards (SiTCP) which connect detector readout electronics to network via standard Ethernet and TCP/IP, while the software consists of the DAQ-Middleware which standardizes all software components, including data acquisition, analysis, and user interface.
space
The hardware component of the new DAQ framework is a set of network-based electronics called silicon transmission control protocol (SiTCP), while the software component is called DAQ-Middleware. Signals from detectors are picked up by SiTCP, which transmits the data to PCs via a standard Ethernet and TCP/IP network, instead of detector-specific cables. The data is distributed, stored, and analyzed online by the DAQ-Middleware. DAQ-Middleware is seamlessly accessible from PCs on network and is controlled by a user interface on a standard web browser.

Small, fast,
and low-power

Tens of thousands of channels of detectors may send signals to a single DAQ. Each channel requires frontend readout electronics, which would transmit a high rate signal to multiple computers. Most experiment utilizes specialized cables for the transmission. At the back end of the DAQ, where PCs analyze the detector data, one would find a TCP/IP network. However, the frontend electronics, where the interfaces to the detectors were located, had no such standardized networking. This, Uchida thought, made flexible data distribution unnecessarily difficult.

Uchida’s innovative idea was to employ a network for the frontend electronics. In 2001, he set off to test the principle with a nonstandard network protocol. The use of a nonstandard protocol turned out to be a rather unpopular idea, but the test was a good learning experience. At that time, networks were speeding up rapidly, and the standardized network protocol used for the Internet boosted the network growth worldwide. Uchida traded his nonstandard protocol for Ethernet and TCP/IP, which were and are widely used and therefore much more cost-effective.

space subaru
space
The SiTCP board that allows real-time data transfer via gigabit Ethernet. The square chip at the center is a field programmable gate array (FPGA) on which the network capability is encoded.
Powerful hardware was necessary. One challenge was the extremely high data transmission rate required by particle experiments. A run-of-the-mill particle experiment required a data transmission rate of one gigabit per second. Additionally, most experiments generally have very limited space for the frontend electronics, so any device would need to be extremely small. In short, he had to achieve the network performance level of one ordinary PC in a single chip.

Uchida’s solution was to build a highly specialized device, one specifically designed for TCP. To do this, he utilized an application specific integrated circuit (ASIC) implemented on a field programmable gate array (FPGA). “Creating a specialized device meant we needed to only include the necessary functions. This decreased the power consumption, and made the device smaller and faster,” says Uchida.
 
The key to designing smaller and faster hardware is to make full use of all resources on the device. This means distributing computational tasks as evenly as possible among the process modules on the device so that no modules are ever at rest. To do this, most people would build a controller module onto the chip. However, such a controller requires both additional space and additional power. Thus, Uchida designed his chip without a controller. On his chip, jobs are distributed among process modules by the equivalent of a bucket brigade. This required simple yet sophisticated design, and developing the logic was a challenge. Uchida applied all his skills and experience to this task, and after many test-and-modify cycles, he eventually found the right balance between simplicity and speed.

muon2
space
Muon counter (center) and readout module (lower right). The low-power SiTCP hardware enabled solar-powered readout electronics for cosmic-ray muon radiography to probe volcano internal structure in real time.
space
The resulting SiTCP is a silicon board with a 2-centimeter by 2-centimeter FPGA whose network capability equals that of a standard desktop PC. Each device is responsible for one detector channel, and transmits data at a rate of one-gigabit per second. “The beauty of this scheme is that to access to your detectors, you just need a PC with an Internet connection,” says Uchida. With the Ethernet and TCP/IP capability, researchers can access detectors from their home institutions or anywhere else in the world via a standard Internet connection.

Solar-powered SiTCP for volcano observatory
The first experiment to take advantage of the potential of SiTCP is the neutrino observatory Super-Kamiokande. During the readout electronics upgrade in 2008, 500 SiTCP chips were delivered to Kamioka, and installed successfully. Since then, Uchida has worked with many other experiments and detectors.

One particularly interesting example is the development of power efficient readout electronics for cosmic-ray muon radiography of volcanoes. Cosmic-ray muon radiography measures the number of cosmic-ray muons that penetrate through a volcano over a period of one month or more. The high-density area of the volcano blocks muons, while low-density area does not interfere with the muon passage. The difference in the number of muons, therefore, provides information about the internal structure of the volcano. One major difficulty encountered when building an observatory near a volcano is that access to commercial power sources is generally limited.
 
space iwojima
space
An image of Mt. Iwodake obtained by cosmic-ray muon radiography. Dark area represents high-density region, and light area represents low-density region. (Image courtesy of the Earthquake Research Institute at the University of Tokyo)
To cope with such limited power availability, scientists developed nuclear emulsion film. Like the more common types of camera film, this film requires no power. This development made it possible to study the time-independent internal structure of a volcano without requiring a power source. However, to study the changing structure of an active volcano, scientists needed real time information, and this required a more active technology. However, this in turn required readout electronics, and those require electrical power.

Uchida, in collaboration with colleagues at the Earthquake Research Institute at the University of Tokyo, developed low-power readout electronics with a wireless LAN capability which could be powered by solar energy. The system has since been used to observe several volcanoes in Japan.

Other successful applications of SiTCP technology include electronics for the Hyper Suprime-Cam for the Subaru Telescope at the summit of Mauna Kea, Hawaii. Uchida has also worked with other advanced detectors such as the gas electron multiplier (GEM) and the silicon-on-insulator (SOI) board that other projects in the KEK DTP develop.

User-friendly, versatile DAQ middleware
Alongside the small, fast, power efficient readout electronics, the DAQ team has also focused on the backend software. Tens of thousands of channels of data, each having a transfer rate of one-gigabit per second, need to be distributed efficiently throughout the system. The DAQ team’s new middleware, called DAQ-Middleware, unites all the required software components in one place. These components gather data, distribute data, build events, and analyze the data acquired by the detectors, with a user interface in a standard web browser.

middleware
space
DAQ-Middleware is based on the robot technology middleware (RT-Middleware) developed at the National Institute of Advanced Industrial Science and Technology (AIST) in Tsukuba.
space
Leader of the DAQ project in the KEK Detector Technology Project (DTP) for three years, Dr. Yoshiji Yasu has a wealth of experience with DAQ in general. Yasu had long pondered what could be done to make DAQ systems more user-friendly. DAQ software systems in experiments are generally highly specialized, enormously complicated, and impossible to reuse. For example, at the ATLAS experiment at the Large Hadron Collider (LHC), the DAQ system is gigantic, having three layers of trigger systems, and employing more than 1,000 PCs. It is difficult to adapt components from the ATLAS DAQ for use in the type of small, single-beamline experiments which are much more common.

Yasu finally saw an opportunity when he heard about the robot technology middleware (RT-Middleware) software, which is specified by international standard. A software package for the RT-Middleware is developed at the National Institute of Advanced Industrial Science and Technology (AIST) in Tsukuba. “The RT-Middleware software was developed not only for robot system but also for an embedded system. An embedded system has devices with sensors, readout electronics, and software to process, analyze, and store data,” says Yasu. “The framework was just what I was looking for.”

Yasu proposed developing the DAQ-Middleware for the 2006-2008 next-generation DAQ project, and started developing DAQ specific tools to be implemented in the RT-Middleware framework, in collaboration with colleagues at KEK and AIST.

space daq middleware
space
The key technology of DAQ-Middleware is the web technology. Upon reception of a command from the control panel on a web browser, the DAQ-Operator searches an XML database for the list of required DAQ-components, and generates the components. The gatherer and monitor components in the readout module are the only software components that are detector-specific.
One of the key technologies for the DAQ-Middleware is the web technology. When operators send a command via the control panel in their web browser, the DAQ-Operator software module calls an appropriate database with a list of DAQ-Components to generate. For example, when ‘configure’ command is issued on operator’s client, the DAQ-Operator obtains the list of necessary components to display the requested information and generate them. “The advantage of this scheme is that when users need to change functions, such as functions of load distribution in the system, they can do so by just making modifications to the database. They do not need to modify the components or operations. On the other hand, in ordinary programming, such a change would require rewriting the program and recompiling all over again,” says Yasu. “This makes the DAQ-Middleware a highly flexible system.”

The first two years of the project were spent on basic research, and on developing the pre-prototype. In 2008, the prototype was installed in a neutron beamline at J-PARC, and was tested at the first beam commissioning. After a year-long debugging period, the DAQ-Middleware was installed at another neutron beamline in 2009. The installation was without trouble, and the system was immediately successful, surprising everyone at the beamline, who were all familiar with the pains of DAQ systems in general. Now, at the Material and Life Facility (MLF) at J-PARC, over half of the twenty-three neutron beamlines and one muon beamline either have installed, or plan to install, the DAQ-Middleware.

Reusable repository and easy installation
For Yasu and his team members, challenges still remain.

First, one objective is to make the DAQ-Component reusable from experiments to experiments. For example, the DAQ-Middleware depends on the software framework specification developed at the MLF according to their requirements. The DAQ-Middleware repository for such experiment-specific frameworks still needs improvements before it can be used in projects beyond MLF.

advantage
space
The advantage of the DAQ-Middleware is the flexibility of component configuration, and reusability of already developed components.
space
Second, because the DAQ-Middleware is designed to be flexible, the installation of the software can still be a challenge for a non-expert. “We aim for a seamless installation, one which is as easy as an installation of commercial software in an operating system such as Linux or Windows,” says Yasu.


A complete DAQ package
The team is united in their desire to create a versatile, user-friendly DAQ-Middleware. “The DAQ-Middleware is a new foundation for DAQ,” says team member Eiji Inoue. For the new DAQ-Middleware project leader, Kazuo Nakayoshi, the system is in the process of developing into a broader and more complete DAQ system, one which can easily be used in many different types of experiments. “The inter-university network for collaborative development of the complete DAQ-Middleware is growing right now,” says he. “We hope to make good use of this opportunity for the continued improvement of the DAQ system.”

space neutron
space
Of the twenty-three neutron beamlines at J-PARC, twelve beamlines have either installed (light blue) or plan to install the DAQ-Middleware.
The combination of SiTCP hardware and DAQ-Middleware software offers a complete, network-based DAQ system package, which is flexible, easily installed, and easy-to-use. For a complete DAQ implementation, there are just two detector-specific components necessary: the gatherer software module, and the monitor software module. The gatherer module is essentially a device driver to connect SiTCP’s to detectors, and the monitor module is software to monitor the detectors. Specific detector software components have been developed for several detectors, including position sensing detectors (PSD), scintillation counters, and GEMs. “Development of the detector-specific components requires some programming skills,” says Yasu. “However, the framework is there, meaning programmers just need to worry about their readout components, but not the basic communications among DAQ parts.”

Meanwhile, Uchida is continuing to improve the SiTCP hardware. This year, the SiTCP FPGA chips will become capable of processing data at a rate of ten gigabit per second. His other current project is clock synchronization using networks. For proper job distribution among networked computing resources, the system clocks of every computer in the network need to be synchronized. Generally, the time is defined by a nest of clock systems, which is itself extremely complicated. “Now that the readout electronics are on a network, we can talk about synchronizing clocks using a network clock,” says Uchida. This is not a straightforward job. Networks have jitter, and the arrival time of a packet can vary. Uchida is investigating possibilities to work around this difficulty.

The complete package of DAQ for particle experiments and observatories, with installation tools and full documentation, is now available, and will be a great benefit to scientists of many disciplines.
 
space
copyright (c) 2010, HIGH ENERGY ACCELERATOR RESEARCH ORGANIZATION, KEK
1-1 Oho, Tsukuba, Ibaraki 305-0801 Japan
www-admin@kek.jp | Copyright | Send Question |