List columns#

Some columns do not hold one value per row. A list column stores a variable-length sequence of values per row, including nested sequences, while keeping every byte in ordinary, chunked, filterable HDF5 datasets.

The offsets layout#

A list column is a group (a direct child of the table group) with CLASS = "LIST_COLUMN" and KIND = "OFFSETS" attributes. Its storage follows the same offsets encoding used by Arrow’s list arrays: all elements are stored back-to-back in a values member, and a uint64 OFFSETS dataset marks the boundaries:row i’s elements are VALUES[OFFSETS[i]:OFFSETS[i+1]], with OFFSETS[0] = 0 and the offsets non-decreasing. A nullable list column adds a MASK dataset that records which rows are null.

A nullable list-of-strings column, for example, is laid out as:

GROUP "tags"                         CLASS = "LIST_COLUMN"
│                                    KIND  = "OFFSETS"
├── DATASET "OFFSETS"   uint64       row boundaries into VALUES
├── DATASET "MASK"                   which rows are null lists
└── GROUP "VALUES"                   CLASS = "STRING_VALUES"
    ├── DATASET "OFFSETS"  uint64    per-string boundaries into CHARS
    ├── DATASET "CHARS"              the UTF-8 bytes, back to back
    └── DATASET "MASK"               which string elements are null

The values member takes one of three shapes, and this is where the layout becomes recursive:

  • a leaf dataset: a rank-1 dataset of any scalar column datatype (declared with LeafValuesSpec);

  • a STRING_VALUES group: variable-length UTF-8 strings stored as an offsets-plus-bytes pair (StringValuesSpec), so even variable text lives in chunked, compressible datasets rather than on the HDF5 global heap;

  • another list level (NestedListSpec), giving lists of lists to any depth.

Variable-length HDF5 datatypes are deliberately forbidden anywhere below a list column; the offsets encoding exists precisely to avoid them.

Declaring and writing#

A list column is declared with ListColumnSpec alongside ordinary columns:

from h5col import LeafValuesSpec, ListColumnSpec, StringValuesSpec, Table

columns = [
    ListColumnSpec(name="depths", values=LeafValuesSpec(dtype="float32"), units="m"),
    ListColumnSpec(name="tags", values=StringValuesSpec(nullable=True), nullable=True),
]
table = Table.create(f.create_group("profiles"), columns)

table.append(
    {
        "depths": [[0.5, 1.0, 2.0], [], [0.7]],
        "tags": [["qc", "raw"], None, []],
    }
)

Rows are Python sequences; None writes a null row (permitted because tags is nullable). Reading returns row-wise Python lists, with None for null rows and decoded strings for string values:

table["tags"].read()

[['qc', 'raw'], None, []]

For deeper nesting, wrap another level — this declares list<list<int32>>:

ListColumnSpec(
    name="pairs",
    values=NestedListSpec(values=LeafValuesSpec(dtype="int32"), nullable=True),
)

Null, empty, and missing are three different things#

The layout distinguishes states that formats without a mask conflate:

  • A null row (None): the row has no value at all. Recorded in the top-level MASK; only nullable list columns can hold one. This is also what new rows become when a nullable list column is omitted from an append — and why a non-nullable list column must be provided in every append.

  • An empty row ([]): the row’s value is a sequence with zero elements — two equal consecutive offsets. Any list column can hold one.

  • A missing element inside a row: leaf values reuse the ordinary fill-value mechanism, and string values use the STRING_VALUES group’s own element mask.

ListColumn.is_missing() returns the null-row mask.

Storage control#

Each piece of the layout accepts its own chunks and filters: on the ListColumnSpec they apply to the top-level OFFSETS dataset, and on a LeafValuesSpec to the element dataset. On a StringValuesSpec, filters applies to the CHARS byte buffer — usually the one worth compressing — while chunks sets the chunk size of both the group’s inner OFFSETS and CHARS. See filters.

Limits worth knowing#

List columns are stored and appended like any other column, but they stand outside two features: they cannot serve as a table’s row-index column, and they cannot carry search indexes — predicates on list columns are not supported by the query layer. Where selection matters, keep the selective keys in scalar columns beside the list.