Conformance#

h5col is the reference implementation of HEP001 — H5Col: Column-Oriented Tabular Data in HDF5. Being a reference implementation cuts two ways: the code aims to follow the convention exactly, and where it cannot, it says so precisely. This page is that record.

Tables are written with VERSION = "1.0". On open, the major version is checked and a table from a newer major raises VersionError; validate() checks the convention’s consistency rules on demand, structurally by default. With deep=True it goes further, re-deriving every valid index of the implemented families over the element datatypes its builders handle and comparing contents; indexes of other kinds, or over datatypes the builders cannot recompute, still receive the structural checks only.

What is implemented#

  • Table groups with the full write protocol: creation from specs, appends with commit-last NROWS ordering, logical truncation, schema evolution via add_column(), and validation.

  • All scalar column families — numeric, fixed-length UTF-8 strings (with the no-silent-truncation guarantee), the boolean enumeration, and categorical columns with label datasets under CATEGORIES.

  • Missing-value semantics: recommended per-datatype fills, the canonical missing-value test, and valid_min/valid_max enforcement against the fill.

  • List columns in the offsets encoding, including string values (STRING_VALUES), nested lists, null-versus-empty distinction, and per-level masks.

  • Per-column filter pipelines over the HDF5 filter ecosystem.

  • Search indexes: BITMAP, SORTED_ROWS, and CHUNK_MINMAX, with the GENERATION-based validity protocol, index maintenance inside the write protocol (maintain_indexes=True), and refresh.

  • The query layer with pyarrow-parity predicate syntax and three-valued missing-value semantics.

Not implemented#

The convention’s fourth index family, CHUNK_BLOOM (per-chunk Bloom filters for equality pruning on high-cardinality columns), is not built by this package; requesting it raises SchemaError. Files containing one are still safe to use — unknown index kinds are left untouched and simply never consulted.

Known deviation: object references#

This is the one place the implementation knowingly departs from the convention.

HEP001 requires every reference attribute (CATEGORIES, SEARCH_INDEX_LIST, INDEX_COLUMNS, and the bitmap VALUES link) to use the unified H5T_STD_REF datatype introduced in HDF5 1.12, and forbids the older H5T_STD_REF_OBJ. h5py — the foundation this package is built on — cannot yet create H5T_STD_REF values, so h5col writes H5T_STD_REF_OBJ instead.

The practical consequences are small, and in one respect the deviation is the more portable choice: H5T_STD_REF_OBJ is readable by effectively every HDF5 installation ever shipped. The deviation is also contained by design. Every reference is created and resolved through one module, h5col.references, whose read side accepts either datatype; when h5py gains H5T_STD_REF support, swapping the write side requires no changes anywhere else, and previously written files remain readable.

The engineering record of this decision, including the exact h5py versions probed, is kept in the repository at design/DEVIATIONS.md.

Reading foreign files#

Conformance checking is consumer-lenient where the convention says to be: Table.open verifies the CLASS marker and the version major, while validate() performs the full rule check when asked. Non-conformant structures fail with ConformanceError and a message naming the violated rule rather than with a generic h5py error.