Tables and columns#

The three classes below are the reading-and-writing heart of the package. Table wraps a table group; its create() / open() classmethods are the two entry points, and its mapping-style access (table["name"], in, iteration) hands out the column wrappers.

class h5col.Table(group: Any)[source]#

A H5Col column-oriented table backed by an HDF5 group.

static is_table_group(group: Any) → bool[source]#: True if group is a H5Col table group (lenient CLASS check).

classmethod open(group: Any) → Table[source]#

Open an existing table group, checking its CLASS and VERSION major.

Raises:

ConformanceError – If group is not a H5Col table group, or its VERSION is missing or unparsable.
VersionError – If the table’s VERSION major exceeds the supported major.

Create a new, empty table (NROWS = 0) with the given columns.

default_chunk_bytes overrides the automatic (chunk-cache-scaled) target for columns that do not set an explicit chunks shape.

Raises:

SchemaError – If group is already a H5Col table group, a column spec is invalid, or an index_columns name is not among the declared columns.
ReservedNameError – If a column name is a H5Col reserved name.
FillValueError – If a column’s fill value lies inside its declared valid range.

classmethod from_arrays(group: Any, arrays: Mapping[str, Any], *, specs: Sequence[ColumnSpec] | None = None, **table_kwargs: Any) → Table[source]#

Create a table from column arrays and write them in one call.

When specs is omitted, a ColumnSpec is inferred per array (boolean, fixed-string sized to the longest value, or the array dtype). The column order follows arrays.

property group: Any#: The underlying h5py Group backing the table.

property nrows: int#

The table’s logical row count (its NROWS attribute).

Raises:: ConformanceError – If the group carries no NROWS attribute.

property version: str | None#: The table’s H5Col VERSION string, or None when absent.

property title: str | None#: The table’s title attribute, or None when unset.

property description: str | None#: The table’s description attribute, or None when unset.

property generation: int | None#

The table’s GENERATION validity token, or None when absent.

A table acquires GENERATION when its first search index is built and increments it on every subsequent mutation of committed data.

property column_names: list[str]#: Column names in logical order (column-order if present).

property index_columns: list[str]#: Names of the row-index columns, outermost first.

property columns: dict[str, Column | ListColumn]#: The table’s columns by name, in column order, as wrapper objects.

append(data: Mapping[str, Any], *, maintain_indexes: bool = False) → None[source]#

Append rows following the H5Col write protocol.

Every provided column must supply the same number of rows K. A scalar column absent from data is extended and left as its fill value (missing); a boolean column (which has no fill) must always be provided. A list column absent from data must be nullable — its new rows become null lists; a non-nullable list column must always be provided. NROWS is committed last, then the file is flushed.

None in a column’s values marks that row as missing and is stored as the column’s fill value (for a categorical column, its fill code). A column with no fill to store — a boolean, which H5Col forbids from declaring one — rejects None instead of coercing it.

By default, search indexes are not maintained: the GENERATION increment that publishes the append disables them, detectably, and refresh_indexes() restores them later — this keeps the hot append path fast. With maintain_indexes=True, every supported index is rewritten inside the append protocol (future-valued tokens before content) and remains valid after the commit; indexes this implementation cannot rebuild — unsupported kinds, element dtypes the builder does not handle, non-growable index datasets — are left entirely untouched, tokens included.

Raises:

OversizedStringError – If a fixed-length string value’s encoding exceeds the column’s byte budget (H5Col never silently truncates).
SchemaError – For unknown columns, values that are not 1-D, unequal column lengths, an omitted fill-less/boolean column, an omitted non-nullable list column, an unknown category label, or a None in a column that declares no fill value.

truncate(nrows: int, *, maintain_indexes: bool = False) → None[source]#

Shrink the logical table to nrows rows (H5Col logical truncation).

The truncation is logical: no column dataset changes extent, and the rows [nrows, old_NROWS) become reserved storage that consumers ignore. List columns need no extra writes — the smaller NROWS bounds their offsets recursively. Reclaiming physical space would require rewriting each column to its new extent, which this implementation does not do.

Index handling mirrors append() (the spec applies the same steps 4-6 with the new row count): by default every search index is left detectably stale by the GENERATION bump; with maintain_indexes=True the supported indexes are rebuilt inside the protocol and remain valid after the commit.

Truncating to the current row count is a no-op (nothing changes, so nothing is published); growing is an error — that is what append() is for.

Raises:: SchemaError – If nrows is negative or greater than the current row count.

read(columns: Sequence[str] | None = None, *, where: Any = None, explain: bool = False) → Any[source]#

Read columns (default all) as {name: array} over [0, NROWS).

With where= (a query Expression, a List[Tuple] = AND, or a List[List[Tuple]] = OR-of-ANDs), only the matching rows are returned. With explain=True the return value is a (result, QueryPlan) pair.

Raises:

KeyError – If a requested column name is not a column of the table.
SchemaError – If where= is malformed or references an unknown column.

select(where: Any = None) → Selection[source]#

Build a lazy Selection over the table.

Accepts a query Expression, a List[Tuple] (AND), or a List[List[Tuple]] (OR-of-ANDs, pyarrow DNF). None selects every row.

count(where: Any = None) → int[source]#: Number of rows matching where (no column materialization).

build_index(column: str, kind: str | None = None, *, name: str | None = None, description: str | None = None) → SearchIndex[source]#: Build a search index over column (alias of add_search_index()).

property search_indexes: dict[str, SearchIndex]#: Every search-index dataset under SEARCH_INDEXES, wrapped by kind.

add_search_index(column: str, kind: str | None = None, *, name: str | None = None, description: str | None = None) → SearchIndex[source]#

Build a search index over column and link it to the column.

With kind=None the family is picked automatically: BITMAP for boolean and categorical columns (low cardinality, exact equality answers), CHUNK_MINMAX for any other orderable column. Building an index over an unchanged table is not a mutation: GENERATION is created (0) if absent but never incremented. The default dataset name is <column>__<kind, lowercased> — a readable convention only; the linkage is the object reference in the column’s SEARCH_INDEX_LIST.

Raises:

KeyError – If column is not a column of the table.
SchemaError – If column is a list column, no index family applies to its dtype (kind=None), kind is unimplemented, or SEARCH_INDEXES already holds a dataset of the chosen name.
ReservedNameError – If name is a H5Col reserved name.
ConformanceError – If the table carries no NROWS attribute.

refresh_indexes() → int[source]#

Rebuild every supported search index against the current table state.

Restores indexes left stale by append(maintain_indexes=False) or by any other mutation. Returns the number of indexes refreshed; indexes of unsupported kinds are left untouched (and stay detectably stale).

index_is_valid(index: SearchIndex | Any) → bool[source]#: The H5Col consumer validity check for index (wrapper or dataset).

validate(*, deep: bool = False) → None[source]#

Check the H5Col consistency requirements, raising on any violation.

deep=True additionally re-derives every valid search index from its column and compares (consistency rule 9’s semantic half) — an O(index build) check; the default run is structural only. A stale index is never an error: the validity check disables it, as the spec intends.

Raises:

ConformanceError – On the first consistency violation found.
VersionError – If the table’s VERSION major exceeds the supported major.

add_column(spec: ColumnSpec | ListColumnSpec, *, default_chunk_bytes: int | None = None) → Column | ListColumn[source]#

Add a new column to an existing table (schema evolution).

A scalar column is created and grown to the table’s current extent; its existing rows read as its fill value (missing). A fill-less (boolean) column, or a list column (whose “missing” analogue is a null list that would need explicit backfilling), cannot represent pre-existing rows, so adding one to a table that already has rows is refused.

Raises:: SchemaError – If a column of that name already exists, the spec is invalid, or the column cannot backfill pre-existing rows (a boolean or list column on a non-empty table).

class h5col.Column(dataset: Any, table: Table)[source]#

One column of a H5Col table, wrapping its rank-1 HDF5 dataset.

Reads decode to friendly Python values: fixed-length strings become str, boolean columns become NumPy bool, and numeric columns pass through.

property name: str#: The column’s name (the final component of its HDF5 path).

property dataset: Any#: The underlying h5py Dataset (for advanced/low-level access).

property dtype: dtype#

The column’s NumPy dtype (its stored HDF5 datatype).

For a categorical column this is the integer code dtype, not the category labels (see categories).

property is_boolean: bool#: True if this is an H5Col boolean column.

property is_string: bool#: True if this is a fixed-length string column.

property is_categorical: bool#: True if this is a categorical column (its values are category codes).

property categories: NDArray[object_] | None#: The category labels, or None for a non-categorical column.

property ordered: bool | None#: The categories’ ordered flag, or None if not categorical/unset.

property codes: NDArray[Any]#: Raw integer category codes over [0, NROWS) (categorical columns).

property fill_value: Any#

The column’s fill value, or None when it declares none.

None is returned for boolean columns and for any column not in the H5D_FILL_VALUE_USER_DEFINED state (e.g. a full-domain column that declares no missing-row semantics). h5py’s library-default fill value is not a H5Col sentinel and must not be surfaced as one.

property units: str | None#: The column’s units attribute, or None when unset.

property description: str | None#: The column’s description attribute, or None when unset.

property valid_min: Any#: The column’s valid_min attribute, or None when unset.

property valid_max: Any#: The column’s valid_max attribute, or None when unset.

read() → NDArray[Any][source]#: Read the logical rows [0, NROWS), decoded to friendly values.

read_rows(rows: Any) → NDArray[Any][source]#

Read just rows, decoded, in the order given.

Values are fetched with coalesced, chunk-aligned block reads, so a selection confined to a few chunks costs a few chunks rather than the whole column — in both time and peak memory. Rows may be given in any order and may repeat.

Raises:

IndexError – If a row position is negative or not below NROWS.
ValueError – If rows is not one-dimensional.

property search_indexes: list[SearchIndex]#

Search indexes bound to this column, from SEARCH_INDEX_LIST.

The wrappers are bound to this column, so their queries always run against it — even on a non-conformant file where another column also claims the same index dataset.

Raises:: ConformanceError – If the column’s SEARCH_INDEX_LIST attribute is malformed (not a 1-D array of object references).

add_search_index(kind: str | None = None, *, name: str | None = None, description: str | None = None) → SearchIndex[source]#

Build a search index over this column (Table.add_search_index()).

Raises:

SchemaError – If no index family applies to the column’s dtype (kind=None), kind is unimplemented, or SEARCH_INDEXES already holds a dataset of the chosen name.
ReservedNameError – If name is a H5Col reserved name.
ConformanceError – If the table carries no NROWS attribute.

build_index(kind: str | None = None, *, name: str | None = None, description: str | None = None) → SearchIndex[source]#: Build a search index over this column (alias of add_search_index()).

is_missing() → NDArray[bool][source]#

Boolean mask of missing rows over [0, NROWS).

A column with no user-defined fill value (boolean columns, or full-domain columns per H5Col) declares no missing-row semantics, so every row reads as present.

class h5col.ListColumn(group: Any, table: Table)[source]#

One list column of a H5Col table, wrapping its CLASS=LIST_COLUMN group.

Reading returns one Python list per row (or None for a null list), with elements decoded to friendly values — nested lists become nested Python lists, string elements become str, and missing leaf elements become None.

property name: str#: The list column’s name (the final component of its HDF5 path).

property group: Any#: The underlying h5py Group (for advanced/low-level access).

property nullable: bool#: True if the top level carries a MASK (null lists are possible).

property units: str | None#: The list column’s units attribute, or None when unset.

property units_vocabulary: str | None#: The list column’s units_vocabulary attribute, or None when unset.

property description: str | None#: The list column’s description attribute, or None when unset.

read() → list[Any][source]#: Read rows [0, NROWS) as a list of per-row lists (None = null).

is_missing() → NDArray[bool][source]#

Boolean mask of null-list rows over [0, NROWS).

A list column with no top-level MASK cannot mark a row missing, so every row reads as present.