Filters, compression & the on-disk layout#

Each column can carry its own HDF5 filter pipeline. h5col models a pipeline as an ordered list of filters (mirroring the HDF5 filter pipeline), accepting both the built-ins and hdf5plugin compressors. This notebook compresses a column, checks the storage savings, and then walks the raw HDF5 structure h5col produced.

import tempfile
from pathlib import Path

import h5py
import hdf5plugin
import numpy as np

from h5col import (
    ColumnSpec,
    Deflate,
    FilterPipeline,
    Shuffle,
    Table,
)

path = Path(tempfile.gettempdir()) / "h5col_filters.h5"

A per-column filter pipeline#

FilterPipeline([...]) applies filters in order. Here temp_k uses shuffle + deflate, and pressure uses shuffle + hdf5plugin Zstd compressor. We generate smooth, compressible data over several thousand rows so the effect is visible.

n = 20_000
temp_k = 288.0 + 5.0 * np.sin(np.linspace(0, 20, n)).astype("f4")
pressure = (1013.0 + np.cumsum(np.random.default_rng(0).normal(0, 0.1, n))).astype("f4")

f = h5py.File(path, "w")
table = Table.create(
    f.create_group("timeseries"),
    [
        ColumnSpec(name="temp_k", dtype="f4", units="K", chunks=4096,
                   filters=FilterPipeline([Shuffle(), Deflate(6)])),
        ColumnSpec(name="pressure", dtype="f4", units="hPa", chunks=4096,
                   filters=FilterPipeline([Shuffle(), hdf5plugin.Zstd(clevel=5)])),
    ],
)
table.append({"temp_k": temp_k, "pressure": pressure})
table.nrows

20000

Storage savings#

Compare each column’s on-disk allocated size to its uncompressed footprint.

_FILTER_NAMES = {1: "deflate", 2: "shuffle", 3: "fletcher32",
                 32001: "blosc", 32015: "zstd"}
for name in ["temp_k", "pressure"]:
    ds = table[name].dataset
    dcpl = ds.id.get_create_plist()
    raw = ds.dtype.itemsize * table.nrows
    stored = ds.id.get_storage_size()
    ids = [dcpl.get_filter(i)[0] for i in range(dcpl.get_nfilters())]
    filts = [_FILTER_NAMES.get(i, str(i)) for i in ids]
    print(f"{name:9s}  raw={raw:>8d}  stored={stored:>8d}  "
          f"ratio={raw/stored:4.1f}x  filters={filts}")

temp_k     raw=   80000  stored=   30647  ratio= 2.6x  filters=['shuffle', 'deflate']
pressure   raw=   80000  stored=   40781  ratio= 2.0x  filters=['shuffle', 'zstd']

Chunk shape and the chunk cache#

Columns are chunked, and h5col picks the default chunk shape for write-once / read-often (and cloud) access: bigger chunks mean fewer chunks, better compression, and fewer/larger object-store reads. Rather than hard-code a size, the default scales with the file’s raw-data chunk cache — which HDF5 ≥ 2.0 defaults to 8 MiB per dataset (it was 1 MiB before). The target is half the cache, clamped to [2 MiB, 8 MiB], so on HDF5 2.x a column defaults to 4 MiB chunks. The chunk length therefore scales inversely with the item size — a 1-byte column packs 4× as many rows per chunk as a 4-byte one.

from h5col._hdf5 import target_chunk_bytes

MiB = 1 << 20
rdcc = f.id.get_access_plist().get_cache()[2]
print(f"file chunk cache (rdcc_nbytes): {rdcc // MiB} MiB")
print(f"default chunk target:           {target_chunk_bytes(f) // MiB} MiB")

# A table with DEFAULT chunking (no explicit chunks=): every column targets the
# same byte budget, so the row count per chunk tracks the item size.
f.create_group("defaults")
td = Table.create(
    f["defaults"],
    [
        ColumnSpec(name="f8", dtype="f8"),  # 8 bytes
        ColumnSpec(name="f4", dtype="f4"),  # 4 bytes
        ColumnSpec(name="i2", dtype="i2"),  # 2 bytes
        ColumnSpec(name="u1", dtype="u1"),  # 1 byte
    ],
)
for name in ["f8", "f4", "i2", "u1"]:
    ds = td[name].dataset
    clen = ds.chunks[0]
    print(f"  {name}: {clen:>9d} rows/chunk = {clen * ds.dtype.itemsize // MiB} MiB")

file chunk cache (rdcc_nbytes): 8 MiB
default chunk target:           4 MiB
  f8:    524288 rows/chunk = 4 MiB
  f4:   1048576 rows/chunk = 4 MiB
  i2:   2097152 rows/chunk = 4 MiB
  u1:   4194304 rows/chunk = 4 MiB

Scaling and overrides#

Because the target follows the cache, a file opened with a larger cache gets larger chunks — up to the 8 MiB cap, which keeps chunks friendly to readers that open with the default cache (the chunk shape is baked into the file, but the cache is chosen per open). Two escape hatches bypass the automatic sizing:

  • Table.create(…, default_chunk_bytes=N) sets the byte target for the table;

  • ColumnSpec(chunks=rows) sets an exact per-column chunk length.

# Reopen with a 64 MiB cache -> the target clamps at the 8 MiB cap, and an
# explicit default_chunk_bytes overrides the cache-derived size entirely.
big_path = Path(tempfile.gettempdir()) / "h5col_bigcache.h5"
with h5py.File(big_path, "w", rdcc_nbytes=64 * MiB) as bf:
    print("64 MiB cache -> target", target_chunk_bytes(bf) // MiB, "MiB (capped)")
    tb = Table.create(
        bf.create_group("t"),
        [ColumnSpec(name="x", dtype="f8")],
        default_chunk_bytes=1 * MiB,   # explicit override wins over the cache
    )
    print("default_chunk_bytes=1 MiB -> f8 chunk:", tb["x"].dataset.chunks[0], "rows")

64 MiB cache -> target 8 MiB (capped)
default_chunk_bytes=1 MiB -> f8 chunk: 131072 rows

The on-disk layout#

Everything h5col writes is standard HDF5 you can inspect with h5py. Let’s build a small table that exercises categorical and list columns, then walk the tree — the table-group attributes (CLASS/VERSION/NROWS), the column datasets, the CATEGORIES group, and a LIST_COLUMN group’s members.

from h5col import LeafValuesSpec, ListColumnSpec
from h5col._hdf5 import read_str_attr, read_uint64_attr

f.create_group("catalog")
t2 = Table.create(
    f["catalog"],
    [
        ColumnSpec(name="name", dtype=h5py.string_dtype(length=8)),
        ColumnSpec(name="band", categories=["red", "green", "blue"], ordered=True),
        ListColumnSpec(name="samples", values=LeafValuesSpec(dtype="f4")),
    ],
)
t2.append({
    "name": ["alpha", "beta"],
    "band": ["red", "blue"],
    "samples": [[1.0, 2.0], [3.0]],
})

def walk(name, obj):
    indent = "  " * name.count("/")
    if isinstance(obj, h5py.Group):
        print(f"{indent}{name.split('/')[-1] or '/'}/   (Group)")
    else:
        print(f"{indent}{name.split('/')[-1]}   {obj.shape} {obj.dtype.name[:24]}")

cat = f["catalog"]
print("/catalog  CLASS =", read_str_attr(cat, "CLASS"),
      "| VERSION =", read_str_attr(cat, "VERSION"),
      "| NROWS =", read_uint64_attr(cat, "NROWS"))
print()
cat.visititems(walk)

/catalog  CLASS = COLUMN_TABLE | VERSION = 1.0 | NROWS = 2

CATEGORIES/   (Group)
  band__CATEGORIES   (3,) bytes40
band   (2,) int8
name   (2,) bytes64
samples/   (Group)
  OFFSETS   (3,) uint64
  VALUES   (3,) float32

Notice:

  • catalog/ holds the column datasets name, band (integer codes), and the samples list column group.

  • CATEGORIES/ holds the band label dataset, linked from the band column by an HDF5 object reference.

  • samples/ is a LIST_COLUMN group with OFFSETS + VALUES members.

All of it is ordinary HDF5 — any HDF5 tool can read the bytes; h5col just gives them the H5Col meaning.

t2.validate()
print("valid ✔")
f.close()

valid ✔