{ "cells": [ { "cell_type": "markdown", "id": "fc86f7b3", "metadata": {}, "source": [ "# List columns — variable-length values per row\n", "\n", "A *list column* stores a variable-length list in each row — `list`,\n", "`list`, or even `list>` nested to any depth. Internally it is an\n", "HDF5 group using the Apache Arrow **offsets** layout (an `OFFSETS` dataset plus a\n", "flattened `VALUES` member), so access stays O(1) per row and every member keeps\n", "its own compression." ] }, { "cell_type": "markdown", "id": "94e10cff", "metadata": {}, "source": [ "## Why a list column instead of an HDF5 variable-length type?\n", "\n", "HDF5 *can* store a variable-length value per element (vlen strings, ragged\n", "arrays), but those bytes live on the HDF5 **global heap**, outside the dataset's\n", "chunks. That defeats compression (only the heap *pointers* pass through the\n", "filters), defeats direct chunk I/O and cloud-optimized readers, and cannot be\n", "written with parallel collective I/O. H5Col therefore **prohibits variable-length\n", "datatypes anywhere inside a list column** and lays the data out columnar instead,\n", "so every byte flows through the normal chunk + filter pipeline.\n", "\n", "### Three kinds of element storage\n", "\n", "A list column's `VALUES` member is *exactly one* of:\n", "\n", "| Spec | On disk | Use for |\n", "|---|---|---|\n", "| `LeafValuesSpec(dtype=…)` | a rank-1 dataset of scalars | `list`, `list`, `list`, `list` |\n", "| `StringValuesSpec()` | a `STRING_VALUES` group (`OFFSETS` + `CHARS`) | `list` — variable-length UTF-8 |\n", "| `NestedListSpec(values=…)` | another `LIST_COLUMN` group | `list>` — nesting to any depth |\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "68236042", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.571854Z", "iopub.status.busy": "2026-07-11T23:31:08.571749Z", "iopub.status.idle": "2026-07-11T23:31:08.800142Z", "shell.execute_reply": "2026-07-11T23:31:08.799499Z" } }, "outputs": [], "source": [ "import tempfile\n", "from pathlib import Path\n", "\n", "import h5py\n", "\n", "from h5col import (\n", " ColumnSpec,\n", " LeafValuesSpec,\n", " ListColumnSpec,\n", " NestedListSpec,\n", " StringValuesSpec,\n", " Table,\n", ")" ] }, { "cell_type": "code", "execution_count": 2, "id": "cfbc38bf", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.801388Z", "iopub.status.busy": "2026-07-11T23:31:08.801264Z", "iopub.status.idle": "2026-07-11T23:31:08.805393Z", "shell.execute_reply": "2026-07-11T23:31:08.805068Z" } }, "outputs": [], "source": [ "path = Path(tempfile.gettempdir()) / \"h5col_lists.h5\"\n", "f = h5py.File(path, \"w\")" ] }, { "cell_type": "markdown", "id": "ea80e950", "metadata": {}, "source": [ "## Declaring list columns\n", "\n", "- `LeafValuesSpec(dtype=...)` — a list of scalars (`list`).\n", "- `StringValuesSpec()` — a list of variable-length UTF-8 strings.\n", "- `NestedListSpec(values=...)` — a nested list (`list>`).\n", "\n", "`nullable=True` adds a mask so a *null* list is distinct from an *empty* list." ] }, { "cell_type": "code", "execution_count": 3, "id": "3e6243f9", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.806244Z", "iopub.status.busy": "2026-07-11T23:31:08.806178Z", "iopub.status.idle": "2026-07-11T23:31:08.833217Z", "shell.execute_reply": "2026-07-11T23:31:08.832691Z" } }, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "table = Table.create(\n", " f.create_group(\"events\"),\n", " [\n", " ColumnSpec(name=\"event_id\", dtype=\"i8\"),\n", " ListColumnSpec(\n", " name=\"readings\",\n", " values=LeafValuesSpec(dtype=\"f4\"),\n", " nullable=True,\n", " units=\"V\",\n", " description=\"per-event sensor readings\",\n", " ),\n", " ListColumnSpec(name=\"tags\", values=StringValuesSpec()),\n", " ListColumnSpec(\n", " name=\"matrix\", values=NestedListSpec(values=LeafValuesSpec(dtype=\"i2\"))\n", " ),\n", " ],\n", ")\n", "table.append(\n", " {\n", " \"event_id\": [1, 2, 3],\n", " \"readings\": [[0.10, 0.12, 0.09], None, []], # values / null / empty\n", " \"tags\": [[\"alpha\", \"beta\"], [\"gamma\"], []],\n", " \"matrix\": [[[1, 2], [3]], [[4]], []], # ragged nested lists\n", " }\n", ")\n", "table.nrows" ] }, { "cell_type": "markdown", "id": "007ddd46", "metadata": {}, "source": [ "## Reading — ragged Python lists\n", "\n", "Each row reads back as a Python `list` (or `None` for a null list)." ] }, { "cell_type": "code", "execution_count": 4, "id": "7617edcc", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.834314Z", "iopub.status.busy": "2026-07-11T23:31:08.834246Z", "iopub.status.idle": "2026-07-11T23:31:08.842801Z", "shell.execute_reply": "2026-07-11T23:31:08.842331Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "readings : [[np.float32(0.1), np.float32(0.12), np.float32(0.09)], None, []]\n", "tags : [['alpha', 'beta'], ['gamma'], []]\n", "matrix : [[[np.int16(1), np.int16(2)], [np.int16(3)]], [[np.int16(4)]], []]\n" ] } ], "source": [ "for name in [\"readings\", \"tags\", \"matrix\"]:\n", " print(f\"{name:9s}:\", table[name].read())" ] }, { "cell_type": "markdown", "id": "739e31a7", "metadata": {}, "source": [ "## How the offsets encoding works\n", "\n", "There is no per-row *length* stored; a single monotonic `OFFSETS` array marks the\n", "boundaries. Row `i`'s elements are the slice `VALUES[OFFSETS[i] : OFFSETS[i+1]]`,\n", "so reading a row costs one small `OFFSETS` read plus one `VALUES` read — `O(1)`,\n", "with no scanning.\n", "\n", "For `readings` above, `OFFSETS = [0, 3, 3, 3]`:\n", "\n", "| row | slice | value |\n", "|---|---|---|\n", "| 0 | `VALUES[0:3]` | `[0.10, 0.12, 0.09]` |\n", "| 1 | `VALUES[3:3]` | *null* (empty slice **and** `MASK=False`) |\n", "| 2 | `VALUES[3:3]` | `[]` (empty slice, `MASK=True`) |\n", "\n", "Rows 1 and 2 both occupy zero elements of `VALUES`; only the `MASK` tells them\n", "apart.\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "e5604d2d", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.843649Z", "iopub.status.busy": "2026-07-11T23:31:08.843584Z", "iopub.status.idle": "2026-07-11T23:31:08.846483Z", "shell.execute_reply": "2026-07-11T23:31:08.846146Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "nullable: True\n", "null-list mask: [False, True, False]\n" ] } ], "source": [ "r = table[\"readings\"]\n", "print(\"nullable:\", r.nullable)\n", "print(\"null-list mask:\", r.is_missing().tolist()) # row 1 was null" ] }, { "cell_type": "markdown", "id": "a6c95d0c", "metadata": {}, "source": [ "## Null vs empty\n", "\n", "Row 1's `readings` is **null** (unknown — masked), row 2 is an **empty list**\n", "(known to have zero readings). They are different, and H5Col preserves the\n", "distinction; a plain scalar column could not." ] }, { "cell_type": "code", "execution_count": 6, "id": "4b8d39f9", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.847232Z", "iopub.status.busy": "2026-07-11T23:31:08.847178Z", "iopub.status.idle": "2026-07-11T23:31:08.850898Z", "shell.execute_reply": "2026-07-11T23:31:08.850509Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "row 1 (null): None\n", "row 2 (empty): []\n" ] } ], "source": [ "vals = table[\"readings\"].read()\n", "print(\"row 1 (null): \", vals[1])\n", "print(\"row 2 (empty):\", vals[2])" ] }, { "cell_type": "markdown", "id": "75079df8", "metadata": {}, "source": [ "## Where \"missing\" lives in a list column\n", "\n", "Because a list column has structure, *absence* is expressed at three distinct\n", "places, and H5Col keeps them separate:\n", "\n", "- a **null list** (the whole row is unknown) → the level's `MASK` (needs\n", " `nullable=True`);\n", "- a **null string element** inside a `list` → the `STRING_VALUES` `MASK`;\n", "- a **missing scalar element** inside a `list` → the leaf's *fill value*,\n", " using the same canonical missing-value test that scalar columns use.\n" ] }, { "cell_type": "markdown", "id": "17d90bd4", "metadata": {}, "source": [ "## Under the hood — the offsets layout\n", "\n", "A list column is a group; here are its member datasets." ] }, { "cell_type": "code", "execution_count": 7, "id": "b452fda0", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.851732Z", "iopub.status.busy": "2026-07-11T23:31:08.851675Z", "iopub.status.idle": "2026-07-11T23:31:08.856704Z", "shell.execute_reply": "2026-07-11T23:31:08.856335Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "readings/ members: ['MASK', 'OFFSETS', 'VALUES']\n", "OFFSETS: [0, 3, 3, 3]\n", "VALUES: [0.10000000149011612, 0.11999999731779099, 0.09000000357627869]\n", "MASK: [True, False, True]\n", "\n", "tags/VALUES CLASS = STRING_VALUES members: ['CHARS', 'OFFSETS']\n" ] } ], "source": [ "from h5col._hdf5 import read_str_attr\n", "\n", "g = table[\"readings\"].group\n", "print(\"readings/ members:\", list(g.keys()))\n", "print(\"OFFSETS:\", g[\"OFFSETS\"][: table.nrows + 1].tolist())\n", "print(\"VALUES: \", g[\"VALUES\"][: int(g[\"OFFSETS\"][table.nrows])].tolist())\n", "print(\"MASK: \", g[\"MASK\"][: table.nrows].tolist())\n", "print()\n", "sv = table[\"tags\"].group[\"VALUES\"] # a STRING_VALUES group\n", "print(\"tags/VALUES CLASS =\", read_str_attr(sv, \"CLASS\"), \" members:\", list(sv.keys()))" ] }, { "cell_type": "markdown", "id": "77db90ad", "metadata": {}, "source": [ "## String lists use a second offsets level\n", "\n", "`tags` has no `VALUES` *dataset* — its `VALUES` is a `STRING_VALUES` **group**,\n", "because UTF-8 strings are themselves variable-length. All the string bytes sit\n", "back-to-back in a `uint8` `CHARS` buffer, and a second `OFFSETS` cuts that buffer\n", "into individual strings. So reading `tags[i]` resolves *two* offset levels: the\n", "list `OFFSETS` picks which strings belong to row `i`, and `STRING_VALUES/OFFSETS`\n", "picks each string's bytes out of `CHARS`. Both levels compress like any other\n", "dataset — still no global heap.\n", "\n", "### Every member is an ordinary dataset\n", "\n", "`OFFSETS`, `VALUES`, `CHARS`, and `MASK` are each independent HDF5 datasets, and\n", "all inherit `h5col`'s cache-aware default chunking just like a scalar column. You\n", "can override chunking and filters on the *data-bearing* members via the spec's\n", "`chunks=` / `filters=` — `LeafValuesSpec` tunes the leaf `VALUES`,\n", "`StringValuesSpec` tunes the `CHARS` buffer, and `ListColumnSpec` /\n", "`NestedListSpec` tune that level's `OFFSETS`. The `MASK` always uses the default.\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "1d33867d", "metadata": { "execution": { "iopub.execute_input": "2026-07-11T23:31:08.857502Z", "iopub.status.busy": "2026-07-11T23:31:08.857447Z", "iopub.status.idle": "2026-07-11T23:31:08.873415Z", "shell.execute_reply": "2026-07-11T23:31:08.873080Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "valid H5Col table with list columns ✔\n" ] } ], "source": [ "table.validate()\n", "print(\"valid H5Col table with list columns ✔\")\n", "f.close()" ] } ], "metadata": { "kernelspec": { "display_name": "examples", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.6" } }, "nbformat": 4, "nbformat_minor": 5 }