Low-level modules

Six submodules are exported as public, stable entry points below the class-based API. They operate directly on h5py objects — most take a table group or a dataset rather than a Table — and exist for tools that need the convention’s mechanics without the wrappers: validators, migration scripts, or readers in constrained environments. Most users never need them.

h5col.missing

Fill values and the canonical H5Col missing-value test.

A column marks a missing row with its HDF5 fill value. H5Col recommends a per-datatype sentinel chosen to lie outside the column’s logical value range, and defines a single canonical missing-value test that both the fill-equality and NaN cases reduce to.

h5col.missing.recommended_fill(dtype: Any) → Any

Return H5Col’s recommended fill value for dtype.

• Fixed- or variable-length string dtypes → b"".

• Enumerations with a MISSING member → the integer code of that member (the spec’s enum fill convention).

• Enumerations without a MISSING member, including the H5Col boolean datatype (which MUST NOT declare a fill value at all) → raises.

• Integer and float families → the table sentinel.

• Anything else (e.g. float16) → raises FillValueError.

h5col.missing.is_missing(values: Any, fill_value: Any) → NDArray[bool]

Apply the canonical missing-value test element-wise.

missing(v, f) = isnan(f) ? isnan(v) : v == f — i.e. when the fill value is a NaN bit pattern the test is isnan(v); otherwise it is bit/value equality.

h5col.missing.validate_fill_outside_range(fill: Any, valid_min: Any | None = None, valid_max: Any | None = None) → None

Check that fill lies strictly outside [valid_min, valid_max].

A None bound is treated as unbounded on that side. Raises FillValueError if fill falls inside the declared range.

h5col.ordering

The H5Col canonical ordering: orderability and min/max under the defined order.

H5Col defines a total order for a fixed set of datatypes (spec, “Sorted-row permutation index” / Ordering): integers arithmetically, floats by IEEE 754 over finite values and infinities (NaN unordered), booleans and enums by code, and strings byte-wise over UTF-8 with trailing storage padding stripped. CHUNK_MINMAX and SORTED_ROWS indexes may only be built over orderable datatypes; is_orderable() is that predicate.

NumPy’s S-dtype comparison is full-width memcmp over NUL-padded values, which is order-equivalent to the spec’s trailing-NUL-stripped byte-wise rule (NUL sorts below every byte), so NULLTERM/NULLPAD fixed strings compare natively. SPACEPAD strings need their trailing spaces stripped first (normalize_strings()).

h5col.ordering.is_orderable(dtype: Any) → bool

True if dtype has an H5Col-defined order.

Excluded per the spec: object/region references, compound datatypes, array datatypes, and variable-length-array datatypes. Everything else the spec enumerates — integers, floats, booleans, strings (fixed and variable length), opaque, and enumerations — is orderable.

h5col.ordering.is_spacepad(dataset: Any) → bool

True if dataset stores fixed-length strings with space padding.

h5col.ordering.normalize_strings(values: ndarray, *, spacepad: bool) → ndarray

Strip trailing storage padding so byte-wise comparison matches the spec.

NUL padding needs no work (memcmp order-equivalence above); space padding is stripped explicitly.

h5col.ordering.min_max(values: ndarray) → tuple[Any, Any]

Min and max of values under the H5Col order.

The caller passes only orderable, non-missing, non-NaN elements — at least one. Flexible dtypes (fixed strings) cannot use NumPy reductions, so they sort instead.

Raises:

SchemaError – If values is empty.

h5col.references

HDF5 object-reference backend for H5Col.

All object-reference creation and resolution in H5Col goes through this module, so the on-disk reference representation can be changed in one place.

Warning

H5Col mandates the unified H5T_STD_REF datatype (HDF5 1.12+) and forbids the deprecated H5T_STD_REF_OBJ. h5py (as of 3.16) cannot create H5T_STD_REF, so this backend currently writes H5T_STD_REF_OBJ. This is a documented deviation (see docs/DEVIATIONS.md D1). The read side accepts either representation. A conformant backend can replace this module without changing any caller.

h5col.references.ref_dtype() → dtype

Return the NumPy dtype used to store object references.

h5col.references.is_reference_dtype(dtype: Any) → bool

Return True if dtype is an HDF5 object/region reference dtype.

h5col.references.make_ref(obj: Any) → Reference

Create an object reference to an open HDF5 object.

Raises:

ObjectReferenceError – If obj exposes no ref (it is not a referable HDF5 object).

h5col.references.is_null_ref(ref: Any) → bool

Return True if ref is a null object reference.

h5col.references.write_ref_attr(target: Any, name: str, obj: Any) → None

Write a scalar object-reference attribute onto target.

h5col.references.write_ref_array_attr(target: Any, name: str, objs: Iterable[Any]) → None

Write a 1-D object-reference array attribute onto target.

h5col.references.append_ref_to_array_attr(target: Any, name: str, obj: Any) → None

Append a reference to obj to a 1-D reference-array attribute on target.

Creates the attribute when absent. HDF5 attributes cannot be resized in place, so an existing attribute is rewritten with the extended array. The new reference is created before the old attribute is touched, and the old array is restored if the rewrite fails, so a failed append cannot silently drop the existing references. (A hard crash between the delete and the create can still lose the attribute — HDF5 offers no atomic attribute rewrite.)

h5col.references.resolve(where: Any, ref: Any) → Any

Dereference ref relative to file/group where.

Raises ObjectReferenceError for a null reference or a reference that does not resolve.

h5col.reserved

H5Col reserved names, tokens, and name-validation helpers.

H5Col writes its reserved attribute and group names in fixed-length uppercase ASCII (with a few lowercase exceptions borrowed from broader community practice or from AnnData). This module centralizes those tokens so the rest of H5Col never hard-codes a spelling, and provides validators for producer-chosen names.

h5col.reserved.RESERVED_ATTRIBUTE_NAMES = frozenset({'CATEGORIES', 'CLASS', 'GENERATION', 'INDEX_COLUMNS', 'KIND', 'MASK', 'NROWS', 'SEARCH_INDEX_LIST', 'SOURCE_GENERATION', 'SOURCE_NROWS', 'TITLE', 'VALUES', 'VERSION', 'valid_max', 'valid_min'})

Attribute names H5Col treats as reserved (producers must not repurpose them).

h5col.reserved.is_valid_link_name(name: object) → bool

Return True if name is usable as an HDF5 link name (UTF-8, no //NUL).

h5col.reserved.validate_column_name(name: str) → str

Validate a producer-chosen column name and return it unchanged.

Raises:

• SchemaError – If name is not a valid HDF5 link name.

• ReservedNameError – If name collides with a H5Col reserved group, member, or attribute name.

h5col.reserved.validate_index_dataset_name(name: str) → str

Validate a search-index dataset name and return it unchanged.

Datasets in SEARCH_INDEXES may have any name — the spec assigns names no meaning — but the name must still be a single HDF5 link name, and reserved-names rule 2 forbids reusing any reserved name for a search-index dataset.

Raises:

• SchemaError – If name is not a valid single HDF5 link name.

• ReservedNameError – If name is a H5Col reserved name.

h5col.reserved.is_discouraged_column_name(name: str) → bool

Return True for names H5Col says producers SHOULD avoid (leading _).

h5col.lists

List columns: the H5Col offsets encoding for variable-length row values.

A list column is an HDF5 group (a direct child of the table group) with CLASS="LIST_COLUMN" and KIND="OFFSETS". It stores a variable-length list per row using the Apache Arrow offsets layout: all elements are flattened back-to-back into a VALUES member, and a monotonic OFFSETS dataset records each entry’s slice. VALUES is one of three things — a rank-1 leaf dataset, a nested LIST_COLUMN group (lists of lists), or a STRING_VALUES group (variable-length UTF-8 via a second OFFSETS/CHARS level). An optional MASK at any level distinguishes a null entry from an empty one.

The create/append/read/validate functions here are driven by the file structure (not the Python spec), so a table can be reopened and appended without its original ListColumnSpec. Writing follows the H5Col leaf-first order (deepest elements first, each enclosing OFFSETS last) so that committed rows stay fully described at every moment.

h5col.lists.reject_vlen(dtype: Any) → None

Raise if dtype — or any datatype nested inside it — is variable-length.

H5Col rule 11 forbids any HDF5 variable-length datatype anywhere below a list column, including one hidden inside a compound field or an array subtype. h5py’s check_string_dtype / check_vlen_dtype only inspect the top level, so this descends into compound fields and array bases first.

h5col.lists.validate_list_column_spec(spec: ListColumnSpec) → None

Validate a list column spec without touching the file.

h5col.lists.create_list_column(table_group: Any, spec: ListColumnSpec, *, default_chunk_bytes: int | None = None) → Any

Create an empty list column group under table_group from spec.

h5col.lists.append_list_column(level_group: Any, rows: list[Any], n_old: int) → None

Append rows (a list of per-row list values) to a list column group.

h5col.lists.read_list_column(level_group: Any, count: int) → list[Any]

Read entries [0, count) of a list column as a list of (list | None).

h5col.lists.validate_list_column(level_group: Any, count: int) → None

Validate a list column subtree at count entries (recursively).

h5col.indexes

Search-index engine: validity tokens, SEARCH_INDEXES, and index families.

Like h5col.lists, this engine is file-driven: every operation reads the structure it needs from the file, so indexes built in one session are maintainable and queryable after reopening.

The validity-token protocol (spec, “Index validity tokens”) is the backbone:

• GENERATION (table group) identifies the current state of the column data; it is created the first time an index is added and incremented by every mutation thereafter.

• SOURCE_GENERATION / SOURCE_NROWS (each index dataset) name the table state the index content was built against. index_is_valid() is the consumer check; a failed check means “treat the index as absent”, never an error.

• Write ordering keeps every crash state detectable. Mutations gated by the NROWS commit (append) write future-valued tokens before the index content; building or refreshing an index over an already-committed state writes content first and current-valued tokens last.

h5col.indexes.INDEX_CHUNK_BYTES = 65536

Byte target for one chunk of a search-index dataset. Index datasets scale with the source column’s chunk count, not its row count, so the column chunk policy would waste a mostly-empty multi-MiB chunk here; 64 KiB keeps index datasets compact while still amortizing appends.

h5col.indexes.MINMAX_FIELDS = ('min', 'max', 'nan_count', 'fill_count', 'n')

CHUNK_MINMAX compound fields, in required declaration order.

h5col.indexes.SUPPORTED_KINDS = frozenset({'BITMAP', 'CHUNK_MINMAX', 'SORTED_ROWS'})

Index kinds this implementation can build and maintain (grows in 4c).

h5col.indexes.append_refresh_indexes(table_group: Any, g_old: int, n_new: int) → bool

Mutation-protocol step 4: maintain every supported index for the new state.

Used by any NROWS-gated mutation — append and truncation share the same steps 4-6, with n_new the post-mutation row count. For each maintained index, the future-valued tokens (SOURCE_GENERATION = g_old + 1, SOURCE_NROWS = n_new) are written before the content — the index fails the validity check throughout its own rebuild, because the new generation does not yet exist on the table group. The caller commits GENERATION and NROWS afterwards.

Indexes this producer cannot rebuild are left untouched, with their tokens intact — including any index claimed by more than one column, where there is no correct column to rebuild against (the spec forbids the state, and validate reports it). Returns True when any index was rewritten, so the caller knows to flush.

h5col.indexes.bitmap_bytes(nrows: int) → int

Bytes per bitmap row for a table of nrows (ceil(nrows / 8)).

h5col.indexes.bitmap_values_dataset(table_group: Any, index_ds: Any) → Any | None

Resolve a BITMAP index’s accompanying values dataset, or None.

Consumer-lenient: a missing, non-scalar, null, dangling, or unlinked VALUES reference — or a target that is not a KIND-less rank-1 dataset sitting next to the index under SEARCH_INDEXES — yields None, making the bitmap unusable rather than an error; validate reports the violation separately.

h5col.indexes.column_datasets(table_group: Any) → dict[str, Any]

Direct-child rank-1 datasets of the table group (the scalar columns).

h5col.indexes.column_index_datasets(table_group: Any, column_ds: Any) → list[Any]

Resolve the column’s SEARCH_INDEX_LIST references, in order.

Consumer-lenient: null, dangling, and unlinked references are skipped — their indexes are treated as absent, per the spec’s tolerance rules — while validate reports them as rule-4 violations. A malformed (non-1-D) attribute raises, because no reference can be read from it.

h5col.indexes.compute_bitmap(column_ds: Any, nrows: int) → tuple[ndarray, ndarray, bool]

Recompute a BITMAP enumeration for rows [0, nrows).

Returns (values, bits, exhaustive): the distinct non-missing values in H5Col order, the (K, ceil(nrows / 8)) uint8 bit matrix with bit r % 8 of byte r // 8 set where row r equals the k-th value (pad bits zero), and the exhaustive claim. NaN cannot be enumerated — IEEE 754 equality never matches it — so non-missing NaN elements are left out of the enumeration and make the claim exhaustive = False.

h5col.indexes.compute_chunk_minmax(column_ds: Any, nrows: int) → ndarray

Recompute the CHUNK_MINMAX entries for rows [0, nrows).

This is the build oracle: creation, append maintenance, refresh, and deep validation all derive the index content from this one function.

h5col.indexes.compute_sorted_rows(column_ds: Any, nrows: int) → tuple[ndarray, int, int]

Recompute the SORTED_ROWS permutation for rows [0, nrows).

Returns (permutation, fill_tail_length, nan_tail_length). The permutation is total and deterministic: the body is sorted under the H5Col order with ties broken by increasing row position (a stable argsort over rows already in increasing order), followed by the fill tail and then the NaN tail, each in increasing row order. A row goes to the NaN tail if its value is NaN, and otherwise to the fill tail if it matches a non-NaN fill; with a NaN fill every missing row is a NaN row and the fill tail is empty.

h5col.indexes.create_bitmap(table_group: Any, column_ds: Any, *, name: str | None = None, description: str | None = None) → Any

Build a BITMAP index over column_ds, with its values dataset.

The accompanying values dataset is created as <name>_values next to the bitmap (its name carries no meaning; the linkage is the bitmap’s scalar VALUES object reference). The enumeration is the distinct non-missing values in H5Col order, so ordered is true; exhaustive is true unless the column holds non-missing NaN elements, which IEEE 754 equality makes impossible to enumerate.

Raises:

• ConformanceError – If the table group has no NROWS attribute.

• SchemaError – If the column’s datatype is unsupported, or SEARCH_INDEXES already holds the bitmap name or its <name>_values name.

• ReservedNameError – If name (or <name>_values) is a H5Col reserved name.

h5col.indexes.create_chunk_minmax(table_group: Any, column_ds: Any, *, name: str | None = None, description: str | None = None) → Any

Build a CHUNK_MINMAX index over column_ds and link it.

Building over an already-committed table state writes the index content first and the current-valued tokens last, so a crash mid-build leaves a dataset that fails the validity check.

Raises:

• ConformanceError – If the table group has no NROWS attribute.

• SchemaError – If the column’s datatype is unsupported, or SEARCH_INDEXES already holds a dataset of the chosen name.

• ReservedNameError – If name is a H5Col reserved name.

h5col.indexes.create_sorted_rows(table_group: Any, column_ds: Any, *, name: str | None = None, description: str | None = None) → Any

Build a SORTED_ROWS index over column_ds and link it.

Building over an already-committed table state writes the index content first and the current-valued tokens last, so a crash mid-build leaves a dataset that fails the validity check.

Raises:

• ConformanceError – If the table group has no NROWS attribute.

• SchemaError – If the column’s datatype is unsupported, or SEARCH_INDEXES already holds a dataset of the chosen name.

• ReservedNameError – If name is a H5Col reserved name.

h5col.indexes.data_chunk_count(column_ds: Any, nrows: int) → int

Chunks of column_ds that contain logical-table rows.

ceil(nrows / chunk_len) for a chunked column, 1 for a contiguous column, 0 when nrows == 0. Tail-only chunks are not counted.

h5col.indexes.ensure_generation(table_group: Any) → int

Return the table’s GENERATION, creating or repairing it when needed.

Per the spec, a table acquires GENERATION the first time a search index is built over it (“writing GENERATION first if the table did not previously carry it”); building over an unchanged table is not a mutation, so no increment happens here.

A missing-with-indexes or malformed GENERATION (rule-12 violations some foreign tool left behind) fails the strict validity check, so every token this producer would write against it is dead on arrival. It is repaired as scalar uint64 with a value strictly above the old value and above every index’s SOURCE_GENERATION — a spurious increment is explicitly safe (it can only disable indexes, never validate stale ones), whereas any reused value could equal some index’s token and spuriously validate content nobody has verified.

h5col.indexes.find_index_column(table_group: Any, index_ds: Any) → Any | None

The column whose SEARCH_INDEX_LIST references index_ds, or None.

The column-side attribute is the only linkage the spec defines — index datasets carry no back-pointer — so this scans the table’s column datasets. A malformed (non-1-D) SEARCH_INDEX_LIST on some column is skipped, so one bad column cannot break lookups for every other column’s indexes.

An index claimed by more than one column violates the spec’s “a single search-index dataset MUST NOT cover multiple columns”; there is then no correct answer, and silently picking one would make pruning against the wrong column’s data possible — so this raises instead.

h5col.indexes.index_is_valid(index_ds: Any, table_group: Any) → bool

The consumer validity check for one search-index dataset.

SOURCE_GENERATION == GENERATION AND SOURCE_NROWS == NROWS, with any absent or wrong-datatype token failing the check. A False result means “behave as if the index were not present” — it is never an error.

h5col.indexes.index_kind(index_ds: Any) → str | None

The dataset’s KIND value, or None when absent or not a scalar string.

A malformed KIND (non-string or non-scalar value) yields None so that no kind-dispatched code path ever acts on it; validate flags the malformed attribute separately.

h5col.indexes.minmax_dtype(element_dtype: Any) → dtype

The CHUNK_MINMAX compound dtype for a column of element_dtype.

h5col.indexes.mutation_generation(table_group: Any) → int | None

The pre-mutation GENERATION (g_old) for append/truncate.

Strict read: a well-formed token is returned as-is, and an absent one is None (a table that does not carry GENERATION skips the bump steps). A malformed token must not simply be incremented — its lenient integer value bypasses the safety property, because g_old + 1 could equal some index’s residue SOURCE_GENERATION and spuriously validate unverified content once step 5 rewrites the attribute as uint64. It is repaired via ensure_generation(), which picks a value above every source token.

h5col.indexes.refresh_all_indexes(table_group: Any) → int

Rebuild every supported stale index against the committed state.

Returns the number of indexes refreshed. Indexes this producer cannot rebuild are left untouched — and stay detectably stale if they already were. Currently valid indexes are also left untouched: they already describe the committed state, and rewriting them in place would open a crash window with torn content behind passing tokens.

h5col.indexes.refresh_index(table_group: Any, index_ds: Any, column_ds: Any) → None

Rebuild index_ds against the table’s current committed state.

The current GENERATION/NROWS are already committed, so the order is content first, tokens last (writing current-valued tokens before the content would let a mid-rebuild index pass the check).

A currently valid index is left untouched: its content already describes the committed state (rule 9), and rewriting it in place would open a crash window where torn content sits behind still-passing tokens — the one state the token protocol exists to prevent.

Raises:

ConformanceError – If the table group has no NROWS attribute.

h5col.indexes.search_index_datasets(table_group: Any) → dict[str, Any]

Every search-index dataset (carries KIND) under SEARCH_INDEXES.

A SEARCH_INDEXES child that is not a group (a misuse of the reserved name) holds no index datasets; validate flags it, the consumer paths simply see no indexes.

h5col.indexes.source_chunk_len(column_ds: Any, nrows: int) → int

Rows per chunk of the source column (its full extent when contiguous).

h5col.indexes.supported_index_dtype(dtype: Any) → bool

True if this implementation can build its index families over dtype.

A producer subset of is_orderable(), shared by every supported family: the spec also orders variable-length strings and opaque values, but this implementation does not build indexes over them (building an index is always optional for a producer).

h5col.indexes.supported_minmax_dtype(dtype: Any) → bool

Backwards-compatible name from sub-phase 4a; the same predicate applies to every family this implementation builds.

h5col.indexes.table_generation(table_group: Any) → int | None

The table’s GENERATION, or None when it carries none.

h5col.indexes.validate_search_indexes(table_group: Any, nrows: int, *, deep: bool = False) → None

Enforce consistency rules 3, 4, 12, and rule 9 for every supported kind.

Rule 9 is applied to every index family this implementation understands (CHUNK_MINMAX, SORTED_ROWS, BITMAP); an index of an unsupported kind is skipped. Rule 9 applies only to indexes whose validity check passes; a stale index is exempt (consumers treat it as absent). Structural rule-9 checks always run; the semantic check — recomputing the index from its column — is O(index build) and runs only with deep=True.

Raises:

ConformanceError – On the first consistency violation found.