h5col

h5col stores column-oriented tables natively in HDF5 files. A table is an ordinary HDF5 group. Each scalar column in it is an ordinary rank-1 HDF5 dataset. A variable-length list column is a small group of such HDF5 datasets. On top of that deliberately simple layout, the H5Col convention specifies the things a real tabular workload needs: a committed row count, per-column chunking and compression, fixed-length strings that never truncate silently, boolean and categorical columns, precise missing-value semantics, and persistent search indexes that accelerate row selection.

The convention is an HDF5 Enhancement Proposal: HEP001 — H5Col: Column-Oriented Tabular Data in HDF5. This package, h5col, built on h5py, is its first experimental implementation. It supports writing and reading the convention’s tables, and adds a small pyarrow-style query API on top:

import h5py
from h5col import ColumnSpec, FixedString, Table, field

with h5py.File("observations.h5", "w") as f:
    table = Table.create(
        f.create_group("obs"),
        [
            ColumnSpec(name="station", dtype=FixedString(nbytes=8)),
            ColumnSpec(name="t_air", dtype="float32", units="degC"),
        ],
        title="Surface air temperature",
    )
    table.append({"station": ["KBOS", "KJFK", "KLGA"], "t_air": [21.5, 24.0, 23.1]})
    print(table.select(field("t_air") > 22.0).read(["station"]))

A file written this way needs nothing beyond HDF5 to be read. Every column is a plain dataset that h5py, h5dump, HDFView, or any other HDF5 tool can open, whether or not this package is installed. The convention’s extra meaning is carried in attributes that convention-aware readers understand and other readers simply ignore.

Where to start

• If you already use Parquet or Arrow and are wondering what this adds, start with Why H5Col?, then work through the quickstart.

• The user guide explains one concept per chapter, from the on-disk data model to search indexes.

• Queries documents row selection: the predicate API, the complete syntax, and how indexes accelerate evaluation.

• The examples are rendered Jupyter notebooks, including a real-data walkthrough with New York City taxi trips.

• The API reference documents every public name in the package.

• Adherence of this package to the H5Col (HEP001) specification is documented in the conformance page.

• The package’s license is here.

Funding

The software initial development was funded by the U.S. Department of Energy, Office of Science, Office of Fusion Energy Sciences, under Award Number DE-SC0024442.