Column types & missing values

H5Col supports numeric, fixed-length string, boolean, and categorical columns, each with an optional physical units, a description, and a valid value range (valid_min/valid_max). This notebook tours them, plus how missing values work.

import tempfile
from pathlib import Path

import h5py
import numpy as np

from h5col import (
    ColumnSpec,
    FixedString,
    OversizedStringError,
    Table,
    bool_dtype,
)

path = Path(tempfile.gettempdir()) / "h5col_types.h5"
f = h5py.File(path, "w")

Fixed-length strings — no silent truncation

HDF5 fixed-length strings are sized in bytes. FixedString enforces the budget and raises instead of silently truncating (a UTF-8 code point can take up to 4 bytes).

fs = FixedString(5)          # 5 bytes
print("encoded:", fs.encode(["hello", "world"]))

try:
    fs.encode(["café ☕"])      # far more than 5 bytes
except OversizedStringError as e:
    print("refused oversized value:", e)

encoded: [b'hello' b'world']
refused oversized value: string value at index 0 needs 9 bytes but the column allows at most 5: 'café ☕'

Categorical columns

Give categories (the label set) instead of a dtype. H5Col stores compact integer codes and a separate categories dataset under a CATEGORIES group; ordered=True records an ordinal scale.

table = Table.create(
    f.create_group("obs"),
    [
        ColumnSpec(name="count", dtype="i4", units="1",
                   description="event count"),
        # flux is non-negative, so we mark missing with a negative sentinel:
        # a fill value must lie strictly OUTSIDE [valid_min, valid_max].
        ColumnSpec(name="flux", dtype="f8", units="W m-2",
                   valid_min=0.0, fill_value=-1.0),
        ColumnSpec(name="flagged", dtype=bool_dtype()),
        ColumnSpec(name="quality",
                   categories=["good", "suspect", "bad"], ordered=True),
    ],
)
table.append({
    "count":   [10, 3, 7, 0],
    "flux":    [412.5, 0.0, 88.1, 5.0],
    "flagged": [False, True, False, True],
    "quality": ["good", "suspect", "bad", None],   # None -> missing category
})
table.read()

{'count': array([10,  3,  7,  0], dtype=int32),
 'flux': array([412.5,   0. ,  88.1,   5. ]),
 'flagged': array([False,  True, False,  True]),
 'quality': array(['good', 'suspect', 'bad', None], dtype=object)}

q = table["quality"]
print("is_categorical:", q.is_categorical)
print("categories:", list(q.categories))
print("ordered:", q.ordered)
print("integer codes:", q.codes)         # -1 marks the missing category
print("decoded:", list(q.read()))

is_categorical: True
categories: ['good', 'suspect', 'bad']
ordered: True
integer codes: [ 0  1  2 -1]
decoded: ['good', 'suspect', 'bad', None]

Missing values

Every non-boolean column has a fill value marking missing rows. It must lie outside the valid range — here flux is non-negative, so we chose -1.0. A column omitted from an append leaves its new rows missing. Boolean columns cannot be missing, so they must always be supplied.

# 'flux' omitted here -> those two rows are missing.
table.append({
    "count":   [4, 9],
    "flagged": [True, False],
    "quality": ["good", "good"],
})
print("flux values:", table["flux"].read())
print("flux fill value:", table["flux"].fill_value)
print("missing mask:", table["flux"].is_missing().tolist())

flux values: [412.5   0.   88.1   5.   -1.   -1. ]
flux fill value: -1.0
missing mask: [False, False, False, False, True, True]

The canonical missing-value test compares each row against the fill value, so is_missing() cleanly separates real data from absent rows regardless of the sentinel chosen.

present = ~table["flux"].is_missing()
print("mean flux over present rows:",
      float(np.mean(table["flux"].read()[present])))
f.close()

mean flux over present rows: 126.4