Column datatypes

A scalar column holds one value per row, and H5Col defines four families for it: numeric columns, fixed-length strings, booleans, and categoricals. Each family maps to plain HDF5 datatypes, so files remain readable everywhere. What the convention adds is precise semantics on top. Variable-length values per row are the job of list columns, which have their own chapter.

Every family is declared through the same object, a ColumnSpec, whose fields cover identity (name, description), the datatype, storage (chunks, filters — see filters), and value semantics (fill_value, valid_min, valid_max, units, units_vocabulary).

Numeric columns

The convention’s core numeric set covers: int8 through int64, their unsigned counterparts, float32, and float64. They can be declared in several flavors: np.float64, "float64", or np.dtype("<f8"):

ColumnSpec(name="t_air", dtype="float64", units="degC", valid_min=-90, valid_max=60)

Unless you pass fill_value=, the column receives the convention’s recommended per-dtype sentinel as its fill (the exact values are tabulated in missing values). A numeric dtype outside that set, e.g., float16, has no recommended sentinel, so it is accepted only with an explicit fill_value=. Creation raises a FillValueError without one. valid_min and valid_max are stored as column attributes for consumers, and they guard the fill: a fill value inside the declared valid range is rejected at creation with FillValueError.

Fixed-length strings

A string column declares its byte size with FixedString:

ColumnSpec(name="station", dtype=FixedString(nbytes=8))

Values are UTF-8 encoded into exactly nbytes bytes of storage per row, padded with NULs. The budget counts bytes, not characters — a four-character string of non-ASCII text can need more than four bytes. The guarantee that matters is that a value whose encoding exceeds the budget raises OversizedStringError and names the row, instead of being silently truncated. If the data can outgrow the column, you find out at write time, not at analysis time.

Reading decodes back to Python strings (a NumPy object array). The default fill value for a string column is the empty string, so an empty value reads as missing by default; if empty strings are meaningful data in your model, account for that when choosing the column’s fill.

Why fixed-length rather than HDF5’s variable-length strings? Variable-length data is stored on the HDF5 file’s global heap, outside the dataset’s chunks. This means chunk compression never touches the actual characters, and reads scatter across the file. Fixed-length strings keep every byte inside the chunk pipeline, where filters and contiguous reads work as intended. When string lengths genuinely vary too much for a fixed length, a list column with string values provides variable-length text that still lives in filterable chunks.

Boolean columns

HDF5 has no native boolean type, so the convention fixes one: an enumeration over signed 8-bit integers with exactly two members, FALSE = 0 and TRUE = 1. Declare it with bool_dtype():

ColumnSpec(name="qc_passed", dtype=bool_dtype())

Booleans read back as a NumPy boolean array. The implementation is strict in both directions: appending accepts Python/NumPy booleans or exact 0/1 integers (SchemaError otherwise), and reading a stored code other than 0 or 1 raises ConformanceError rather than guessing (NumPy would happily call every nonzero value true).

A boolean column declares no fill value because a boolean cannot be missing so it must be supplied in every append, and it may not declare valid_min or valid_max.

Categorical columns

A categorical column stores small integer codes and keeps the label values in a labels dataset under the table’s CATEGORIES group. Declare one by listing its categories; the code dtype is chosen automatically — the smallest signed integer whose positive range covers the category count, so up to 127 labels cost one byte per row:

ColumnSpec(name="payment_type", categories=["Credit card", "Cash", "Dispute"])

Labels may be strings (stored as a fixed-length string dataset sized to the longest label) or numbers. The column dataset carries a CATEGORIES object reference to its labels dataset, so the association survives renames and moves.

The API works in labels, not codes. append() takes label values. An unknown label raises SchemaError, and None marks a missing row. Column.read returns labels, with None where the code is the fill. The raw codes remain available as Column.codes, the labels as Column.categories, and an optional ordered flag (for ordinal categories) round-trips through the spec and Column.ordered.

The default fill code is -1 for signed code dtypes (and the type maximum for unsigned), which cannot collide with a valid code as long as the code type leaves room for it. A fill that would collide, an unsigned code type fully saturated by its categories, is rejected at creation, and an explicit categorical fill must likewise lie outside [0, ncategories).

Categoricals earn their keep twice: in storage, where a repeated 20-byte label costs one byte per row, and in queries, where equality predicates compare labels (field("payment_type") == "Cash") and a bitmap index answers them exactly without touching the column data.