List columns — variable-length values per row#
A list column stores a variable-length list in each row — list<float>,
list<str>, or even list<list<...>> nested to any depth. Internally it is an
HDF5 group using the Apache Arrow offsets layout (an OFFSETS dataset plus a
flattened VALUES member), so access stays O(1) per row and every member keeps
its own compression.
Why a list column instead of an HDF5 variable-length type?#
HDF5 can store a variable-length value per element (vlen strings, ragged arrays), but those bytes live on the HDF5 global heap, outside the dataset’s chunks. That defeats compression (only the heap pointers pass through the filters), defeats direct chunk I/O and cloud-optimized readers, and cannot be written with parallel collective I/O. H5Col therefore prohibits variable-length datatypes anywhere inside a list column and lays the data out columnar instead, so every byte flows through the normal chunk + filter pipeline.
Three kinds of element storage#
A list column’s VALUES member is exactly one of:
Spec |
On disk |
Use for |
|---|---|---|
|
a rank-1 dataset of scalars |
|
|
a |
|
|
another |
|
import tempfile
from pathlib import Path
import h5py
from h5col import (
ColumnSpec,
LeafValuesSpec,
ListColumnSpec,
NestedListSpec,
StringValuesSpec,
Table,
)
path = Path(tempfile.gettempdir()) / "h5col_lists.h5"
f = h5py.File(path, "w")
Declaring list columns#
LeafValuesSpec(dtype=...)— a list of scalars (list<float32>).StringValuesSpec()— a list of variable-length UTF-8 strings.NestedListSpec(values=...)— a nested list (list<list<...>>).
nullable=True adds a mask so a null list is distinct from an empty list.
table = Table.create(
f.create_group("events"),
[
ColumnSpec(name="event_id", dtype="i8"),
ListColumnSpec(
name="readings",
values=LeafValuesSpec(dtype="f4"),
nullable=True,
units="V",
description="per-event sensor readings",
),
ListColumnSpec(name="tags", values=StringValuesSpec()),
ListColumnSpec(
name="matrix", values=NestedListSpec(values=LeafValuesSpec(dtype="i2"))
),
],
)
table.append(
{
"event_id": [1, 2, 3],
"readings": [[0.10, 0.12, 0.09], None, []], # values / null / empty
"tags": [["alpha", "beta"], ["gamma"], []],
"matrix": [[[1, 2], [3]], [[4]], []], # ragged nested lists
}
)
table.nrows
3
Reading — ragged Python lists#
Each row reads back as a Python list (or None for a null list).
for name in ["readings", "tags", "matrix"]:
print(f"{name:9s}:", table[name].read())
readings : [[np.float32(0.1), np.float32(0.12), np.float32(0.09)], None, []]
tags : [['alpha', 'beta'], ['gamma'], []]
matrix : [[[np.int16(1), np.int16(2)], [np.int16(3)]], [[np.int16(4)]], []]
How the offsets encoding works#
There is no per-row length stored; a single monotonic OFFSETS array marks the
boundaries. Row i’s elements are the slice VALUES[OFFSETS[i] : OFFSETS[i+1]],
so reading a row costs one small OFFSETS read plus one VALUES read — O(1),
with no scanning.
For readings above, OFFSETS = [0, 3, 3, 3]:
row |
slice |
value |
|---|---|---|
0 |
|
|
1 |
|
null (empty slice and |
2 |
|
|
Rows 1 and 2 both occupy zero elements of VALUES; only the MASK tells them
apart.
r = table["readings"]
print("nullable:", r.nullable)
print("null-list mask:", r.is_missing().tolist()) # row 1 was null
nullable: True
null-list mask: [False, True, False]
Null vs empty#
Row 1’s readings is null (unknown — masked), row 2 is an empty list
(known to have zero readings). They are different, and H5Col preserves the
distinction; a plain scalar column could not.
vals = table["readings"].read()
print("row 1 (null): ", vals[1])
print("row 2 (empty):", vals[2])
row 1 (null): None
row 2 (empty): []
Where “missing” lives in a list column#
Because a list column has structure, absence is expressed at three distinct places, and H5Col keeps them separate:
a null list (the whole row is unknown) → the level’s
MASK(needsnullable=True);a null string element inside a
list<str>→ theSTRING_VALUESMASK;a missing scalar element inside a
list<number>→ the leaf’s fill value, using the same canonical missing-value test that scalar columns use.
Under the hood — the offsets layout#
A list column is a group; here are its member datasets.
from h5col._hdf5 import read_str_attr
g = table["readings"].group
print("readings/ members:", list(g.keys()))
print("OFFSETS:", g["OFFSETS"][: table.nrows + 1].tolist())
print("VALUES: ", g["VALUES"][: int(g["OFFSETS"][table.nrows])].tolist())
print("MASK: ", g["MASK"][: table.nrows].tolist())
print()
sv = table["tags"].group["VALUES"] # a STRING_VALUES group
print("tags/VALUES CLASS =", read_str_attr(sv, "CLASS"), " members:", list(sv.keys()))
readings/ members: ['MASK', 'OFFSETS', 'VALUES']
OFFSETS: [0, 3, 3, 3]
VALUES: [0.10000000149011612, 0.11999999731779099, 0.09000000357627869]
MASK: [True, False, True]
tags/VALUES CLASS = STRING_VALUES members: ['CHARS', 'OFFSETS']
String lists use a second offsets level#
tags has no VALUES dataset — its VALUES is a STRING_VALUES group,
because UTF-8 strings are themselves variable-length. All the string bytes sit
back-to-back in a uint8 CHARS buffer, and a second OFFSETS cuts that buffer
into individual strings. So reading tags[i] resolves two offset levels: the
list OFFSETS picks which strings belong to row i, and STRING_VALUES/OFFSETS
picks each string’s bytes out of CHARS. Both levels compress like any other
dataset — still no global heap.
Every member is an ordinary dataset#
OFFSETS, VALUES, CHARS, and MASK are each independent HDF5 datasets, and
all inherit h5col’s cache-aware default chunking just like a scalar column. You
can override chunking and filters on the data-bearing members via the spec’s
chunks= / filters= — LeafValuesSpec tunes the leaf VALUES,
StringValuesSpec tunes the CHARS buffer, and ListColumnSpec /
NestedListSpec tune that level’s OFFSETS. The MASK always uses the default.
table.validate()
print("valid H5Col table with list columns ✔")
f.close()
valid H5Col table with list columns ✔