Quickstart#

This page builds a small table end to end: define columns, append rows, read them back, select rows with a predicate, add a search index, and reopen the file later. It assumes h5col is installed and takes about ten minutes.

One design decision is worth knowing up front: h5col never opens files. You open an HDF5 file with h5py with whatever driver, page-buffering, or cloud-optimized settings you need, and hand any h5py.Group to h5col. The package defines what happens inside that group, and h5py keeps defining everything outside it.

Create a table#

A table is created from column specifications. Each ColumnSpec names a column and describes its storage: the datatype, an optional fill value that marks missing rows, optional valid bounds, units, and a description.

import h5py
import numpy as np

from h5col import ColumnSpec, FixedString, Table, field

columns = [
    ColumnSpec(
        name="station",
        dtype=FixedString(nbytes=8),
        description="Reporting station identifier",
    ),
    ColumnSpec(name="kind", categories=["manned", "automatic"]),
    ColumnSpec(name="t_air", dtype="float64", units="degC", fill_value=np.nan),
    ColumnSpec(name="samples", dtype="int32", fill_value=-1, valid_min=0),
]

f = h5py.File("quickstart.h5", "w")
table = Table.create(f.create_group("obs"), columns, title="Surface observations")
table.nrows

0

Each spec shows a different capability. station is a fixed-length string column with an 8-byte budget; writing a longer value will raise OversizedStringError rather than truncate. kind is a categorical column: the labels are declared once, and the column stores small integer codes. t_air uses NaN as its fill value, so missing temperatures behave the way NumPy and pandas users expect. samples marks missing rows with the sentinel -1 instead, and declares valid_min=0 so the sentinel is provably outside the valid range.

Append rows#

Rows are appended column-wise, as a mapping from column name to a sequence of values. Every provided column must have the same length. Categorical columns take labels, not codes. In any column, None marks a missing row — it is stored as that column’s fill value — and you may equally write the fill value yourself, as the NaN in t_air’s last entry below does.

table.append(
    {
        "station": ["KBOS", "KJFK", "KLGA", "KDCA"],
        "kind": ["manned", "automatic", "automatic", None],
        "t_air": [21.5, 24.0, 23.1, np.nan],
        "samples": [12, 60, 58, 60],
    }
)
table.nrows

4

The append follows the convention’s write protocol: column data is written and flushed first, and the committed row count (the table’s NROWS attribute) is updated last, so a reader never sees a row count that points at unwritten data. An unknown label in a categorical column, an oversized string, or unequal column lengths all raise before anything is committed.

Read columns back#

Reading decodes each column to friendly values: fixed-length strings come back as Python strings, categorical columns as their labels, booleans as NumPy booleans.

table.read(["station", "kind"])

{'station': array(['KBOS', 'KJFK', 'KLGA', 'KDCA'], dtype=object),
 'kind': array(['manned', 'automatic', 'automatic', None], dtype=object)}

Individual columns are reached by name. A categorical column also exposes its raw integer codes and its label array:

table["kind"].codes

array([ 0,  1,  1, -1], dtype=int8)

table["t_air"].is_missing()

array([False, False, False,  True])

The last row’s temperature is missing (it stored the NaN fill), and the same row’s kind was appended as None, so it reads back as None.

Select rows#

field() builds predicates that combine with &, |, and ~. Because Python’s bitwise operators bind more tightly than comparisons, always parenthesize each comparison:

sel = table.select((field("kind") == "automatic") & (field("t_air") > 22.0))
sel.count

2

sel.read(["station", "t_air"])

{'station': array(['KJFK', 'KLGA'], dtype=object), 't_air': array([24. , 23.1])}

A Selection is lazy — it evaluates once, on first use, and read() materializes only the requested columns for the matching rows.

Missing values follow three-valued logic, exactly as in SQL and pyarrow: a comparison against a missing value is unknown, and only rows where the whole predicate is true are selected. Station KDCA, whose temperature is missing, is not matched by field("t_air") > 22.0 — and it would not be matched by the negation either. To ask for missing rows explicitly:

table.count(field("t_air").is_null())

1

If you prefer pyarrow’s tuple form, the same selection can be written as table.select([("kind", "==", "automatic"), ("t_air", ">", 22.0)]).

Accelerate with a search index#

Queries work with no preparation — every predicate can be answered by scanning the column. A search index makes selective queries cheaper, and it is stored inside the file, next to the data:

table.build_index("t_air", "SORTED_ROWS")
print(table.select(field("t_air") > 22.0).explain())

QueryPlan: 2 / 4 rows matched
   AND-term 0:
      t_air > via sorted_rows

explain() reports how each part of the predicate was evaluated — here the sorted-rows index answered the range predicate exactly, without scanning. With kind left unindexed, a query on it would report via scan. The user guide covers the three index families and how to choose one.

Appends invalidate, refreshing restores#

By default, appending does not rewrite indexes — the hot write path stays fast, and every index is left detectably stale. Queries notice and quietly fall back to scanning, so results are always correct. When the writing is done, one call rebuilds them:

table.append(
    {
        "station": ["KIAD"],
        "kind": ["automatic"],
        "t_air": [25.9],
        "samples": [55],
    }
)
table.refresh_indexes()

1

Appends that must keep indexes valid throughout (at a cost on the write path) can pass append(..., maintain_indexes=True).

Reopen and validate#

Everything written so far is just HDF5 in a group. Reopening is the same two steps in reverse — open the file with h5py, then hand the group to Table.open:

f.close()

f = h5py.File("quickstart.h5", "r")
table = Table.open(f["obs"])
table.column_names

['station', 'kind', 't_air', 'samples']

table.validate()
f.close()

validate() checks the convention’s consistency rules and raises on the first violation; with deep=True it also re-derives every valid search index and compares contents.

Where to go next#