Speaker
Description
This talk will describe new support for data compression in the areaDetector NDPluginHDF5 plugin that writes HDF5 files. These include:
-
Support for the JPEG, Blosc, LZ4, and Bitshuffle/LZ4 plugins within NDPluginHDF5. This support receives uncompressed NDArrays and writes them using compression within the HDF5 data pipeline. This has limited performance because it is not multi-threaded, and incurs significant overhead from the pipeline.
-
Support for HDF5 Direct Chunk Write. This support receives NDArrays and writes them directly, bypassing much of the HDF5 data pipeline. This works both for writing uncompressed NDArrays, and for writing pre-compressed NDArrays, The latter can come from NDPluginCodec, where they can be compressed in multiple threads, or from the ADEiger driver, which directly receives compressed data from the Eiger detector.
-
ADSupport now builds JPEG, Blosc, LZ4, and Bitshuffle/LZ4 codecs as shareable libraries that can be dynamically loaded by the HDF5 library on Linux, Windows, and Mac. This allows existing HDF5 applications (Matlab, IDL, Python, etc.) that are built without support for these codecs to read compressed HDF5 files written by areaDetector. The only requirement is that the application be built with HDF5 1.8.11 or later.
-
Performance measurements will be presented. For example, simDetector generating 1350 frames/s of 1024x1024 32-bit images (5.4 GBytes/s) can be compressed by NDPluginCodec using Blosc LZ4+Byteshuffle and streamed to HDF5 files using the direct chunk write with no dropped frames. This is more than 3.5 times the throughput that can be obtained doing the Blosc compression in the HDF5 plugin itself.
| Track | EPICS for data acquisition |
|---|
