h5col quickstart — your first columnar table

h5col is a reference implementation of H5Col, a convention for storing column-oriented tabular data in HDF5. Each table is an HDF5 group; each column is a rank-1 HDF5 dataset that carries its own dtype, chunking, and compression. The design targets write-once, read-often workflows.

This notebook builds a small environmental-sensor table, writes rows, reads them back, validates conformance, and reopens the file from disk.

import tempfile
from pathlib import Path

import h5py

import h5col
from h5col import ColumnSpec, FixedString, Table, bool_dtype

print("h5col", h5col.__version__, "| h5py", h5py.__version__)

h5col 0.1.0 | h5py 3.16.0

Create a table

We declare the columns with ColumnSpec. A column can be a NumPy dtype, a fixed-length string (FixedString(n_bytes)), or the H5Col boolean type (bool_dtype()). We pick station as the row index.

path = Path(tempfile.gettempdir()) / "h5col_quickstart.h5"
f = h5py.File(path, "w")
table = Table.create(
    f.create_group("weather"),
    [
        ColumnSpec(name="station", dtype=FixedString(6),
                   description="station identifier"),
        ColumnSpec(name="temp_c", dtype="f4", units="degC",
                   valid_min=-90.0, valid_max=60.0),
        ColumnSpec(name="rh_pct", dtype="f4", units="percent",
                   valid_min=0.0, valid_max=100.0),
        ColumnSpec(name="qc_pass", dtype=bool_dtype(),
                   description="quality-control flag"),
    ],
    title="Surface weather observations",
    index_columns=["station"],
)
table

<h5col.Table '/weather' nrows=0>

Append rows

append() follows the H5Col write protocol: it extends every column, writes the new rows, and commits the row count (NROWS) last. You can append in batches.

table.append({
    "station": ["KJFK", "KLAX", "KORD"],
    "temp_c": [21.4, 27.9, 18.2],
    "rh_pct": [63.0, 41.5, 72.0],
    "qc_pass": [True, True, False],
})
table.append({
    "station": ["KDEN", "KSEA"],
    "temp_c": [15.0, 13.7],
    "rh_pct": [55.0, 88.0],
    "qc_pass": [True, True],
})
print("nrows:", table.nrows)
print("columns:", table.column_names)

nrows: 5
columns: ['station', 'temp_c', 'rh_pct', 'qc_pass']

Read the data back

Read one column, or the whole table as a dict of arrays.

print("stations:", list(table["station"].read()))
print("temp_c:   ", table["temp_c"].read())
print("qc_pass:  ", table["qc_pass"].read())

stations: ['KJFK', 'KLAX', 'KORD', 'KDEN', 'KSEA']
temp_c:    [21.4 27.9 18.2 15.  13.7]
qc_pass:   [ True  True False  True  True]

data = table.read()
{k: v.tolist() for k, v in data.items()}

{'station': ['KJFK', 'KLAX', 'KORD', 'KDEN', 'KSEA'],
 'temp_c': [21.399999618530273,
  27.899999618530273,
  18.200000762939453,
  15.0,
  13.699999809265137],
 'rh_pct': [63.0, 41.5, 72.0, 55.0, 88.0],
 'qc_pass': [True, True, False, True, True]}

Validate conformance

validate() checks the H5Col consistency rules (row-count attribute, equal column extents, index-column agreement, and more). It raises on any violation.

table.validate()
print("valid H5Col table ✔")

valid H5Col table ✔

Reopen from disk

Close the file, then reopen it read-only and re-wrap the group with Table.open.

f.close()

with h5py.File(path, "r") as f2:
    t2 = Table.open(f2["weather"])
    print("reopened:", t2.nrows, "rows,", t2.title)
    print("temp_c:", t2["temp_c"].read())

reopened: 5 rows, Surface weather observations
temp_c: [21.4 27.9 18.2 15.  13.7]

Recap

• A table is an HDF5 group (CLASS="COLUMN_TABLE"); columns are rank-1 datasets.

• Table.create → append → read, with validate() for conformance.

• Storage settings (drivers, cloud options) are plain h5py — h5col only owns the table layout, so you open files however you like.