Speaker
Mark Rivers
(University of Chicago)
Description
Data compression is highly desirable to increase the speed and decrease the network bandwidth and disk storage required for modern high-performance detectors.
This talk will describe new support for data compression in areaDetector. These include:
- New NDPluginCodec plugin. This plugin can compress and decompress NDArrays using any of 4 codecs: JPEG, Blosc, LZ4, and Bitshuffle/LZ4. JPEG is lossy while the other codecs are all lossless. Blosc itself supports 6 different codecs, plus bit shuffle and byte shuffle. NDPluginCodec can pass compressed NDArrays to downstream plugins, and can decompress NDArrays from upstream drivers and plugins. NDPluginCodec can run multiple threads to process multiple NDArrays in parallel. The Blosc codec can run multiple internal threads to process a single NDArray faster.
- NDPluginPva can now convert compressed NDArrays to compressed NTNDArrays that are served with pvAccess.
- The ImageJ pvAccess viewer (NTNDArrayViewer) can now receive compressed NTNDArrays, decompress them, and display them. This allows greatly reducing network traffic between the areaDetector IOC and the machine running ImageJ. The decompression in ImageJ is done using the same C/C++ libraries that are used by NDPluginCodec with a thin Java Native Access (JNA) wrapper.
- ADSupport now builds all of the codecs (JPEG, Blosc, LZ4, Bitshuffle/LZ4) on most EPICS architectures. It also builds shareable libraries that can be called dynamically from Java and HDF5.
- ADEiger, the areaDetector driver for the Eiger, can now receive both LZ4 and Bitshuffle/LZ4 compressed data over the ZeroMQ socket. This compressed data can now be sent directly to the HDF5 plugin and to ImageJ without ever decompressing it. It can also be sent to NDPluginCodec to decompress it for other plugins.
| Track | EPICS for data acquisition |
|---|
Primary authors
Mark Rivers
(University of Chicago)
Bruno Martins
(Facility for Rare Isotope Beams)
Marty Kraimer
(Advanced Photon Source)
