HDF5 (scintillometry.io.hdf5)

hdf5 contains interfaces for reading and writing an internal HDF5 format that is suitable for storing intermediate steps of pipelines. A writer can conveniently be used as the output argument for any reader, automatically writing in the relevant block size. For instance:

.. doctest-requires:: h5py, pyyaml
>>> from baseband import data, vdif
>>> from scintillometry.functions import Square
>>> from scintillometry.io import hdf5
>>> fh = vdif.open(data.SAMPLE_VDIF, 'rs')
>>> square = Square(fh)
>>> squared = square.read()
>>> h5w = hdf5.open('squared.hdf5', 'w', template=square)
>>> square.seek(0)
0
>>> square.read(out=h5w)
<HDF5StreamWriter name=squared.hdf5 offset=40000
     sample_rate=32.0 MHz, samples_per_frame=40000,
     sample_shape=(8,), dtype=float32,
     start_time=2014-06-16T05:56:07.000000000>
>>> h5w.close()
>>> fh.close()
>>> h5r = hdf5.open('squared.hdf5', 'r')
>>> recovered = h5r.read()
>>> (squared == recovered).all()
True

It is also possible to stored data encoding using the standard vdif schemes by passing in bps and complex_data. Alternatively, half-precision floats can be used by passing in encoded_dtype='f2' or 'c4'.

Reference/API

scintillometry.io.hdf5 Package

Functions

open(filename[, mode])

Open an HDF5 file as a stream.

Classes

HDF5Frame(header, payload[, valid, verify])

Representation of a HDF5 frame, consisting of a header and payload.

HDF5Header(*[, verify, mutable])

HDF5 format header.

HDF5Payload(words[, header])

Container for decoding and encoding HDF5 payloads.

Class Inheritance Diagram

Inheritance diagram of scintillometry.io.hdf5.frame.HDF5Frame, scintillometry.io.hdf5.header.HDF5Header, scintillometry.io.hdf5.payload.HDF5Payload

scintillometry.io.hdf5.header Module

Definitions for HDF5 general storage headers.

Implements a HDF5Header class used to store header definitions, and provides methods to initialize from a stream template, and to write to and read from an HDF5 Dataset, encoded as yaml file.

Classes

HDF5Header(*[, verify, mutable])

HDF5 format header.

HDF5RawHeader(*[, verify, mutable])

HDF5CodedHeader(*[, verify, mutable])

Class Inheritance Diagram

Inheritance diagram of scintillometry.io.hdf5.header.HDF5Header, scintillometry.io.hdf5.header.HDF5RawHeader, scintillometry.io.hdf5.header.HDF5CodedHeader

scintillometry.io.hdf5.payload Module

Payload for HDF5 format.

Classes

HDF5Payload(words[, header])

Container for decoding and encoding HDF5 payloads.

HDF5RawPayload(words[, header])

HDF5CodedPayload(words[, header])

HDF5DatasetWrapper(words)

Make a HDF5 Dataset look a bit more like ndarray.

Variables

DTYPE_C4

Numpy dtype used to encode half-precision complex numbers.

Class Inheritance Diagram

Inheritance diagram of scintillometry.io.hdf5.payload.HDF5Payload, scintillometry.io.hdf5.payload.HDF5RawPayload, scintillometry.io.hdf5.payload.HDF5CodedPayload, scintillometry.io.hdf5.payload.HDF5DatasetWrapper

scintillometry.io.hdf5.base Module

Interfaces for reading and writing from an internal HDF5 format.

In this format, each HDF5 File has ‘header’ and ‘payload’ Dataset instances, with the header consisting of yaml-encoded keywords describing the start time, sample rate, etc., and the payload consisting of either plain numpy data, or data encoded following the VDIF standard.

Functions

open(filename[, mode])

Open an HDF5 file as a stream.

Classes

HDF5StreamBase(fh_raw, header0, **kwargs)

HDF5StreamReader(fh_raw[, squeeze, subset, …])

HDF5StreamWriter(fh_raw[, header0, squeeze, …])

Class Inheritance Diagram

Inheritance diagram of scintillometry.io.hdf5.base.HDF5StreamBase, scintillometry.io.hdf5.base.HDF5StreamReader, scintillometry.io.hdf5.base.HDF5StreamWriter