Missing values

Real tables have holes: the sensor did not report, the field was not filled in, the join found no partner. H5Col gives missing values a precise, file-level definition instead of leaving each application to invent one.

The model

A missing row simply stores the column’s fill value, the same fill value HDF5 already defines as a dataset creation property. What the convention adds is a single canonical test that every consumer applies:

missing(v, fill) =  isnan(v)   if fill is NaN
                    v == fill  otherwise

Both styles, a sentinel that compares equal or NaN, which never compares equal to itself, reduce to this one rule. The package applies it for you: Column.is_missing() returns the boolean mask over the logical rows, and h5col.is_missing() is the same test as a standalone function.

Every column has a fill, chosen or recommended

When a ColumnSpec does not set fill_value, the column receives the convention’s recommended sentinel for its datatype:

Column dtype	Recommended fill
int8 / int16 / int32 / int64	-127, -32767, -2147483647, -9223372036854775807
uint8 / uint16 / uint32 / uint64	the type maximum
float32 / float64	9.9692099683868690e+36
fixed-length string	"" (the empty string)
categorical codes	-1 (signed) or the type maximum (unsigned)

(h5col.recommended_fill() returns the values in the first four rows. The categorical default is chosen from the code dtype itself by h5col.categorical.default_categorical_fill.)

Two of these deserve a note. The floating-point sentinel is not NaN but the value long used as the default fill in the netCDF world, kept here so files behave consistently across that ecosystem. NaN remains available as an explicit choice. And the empty string as the string fill means an empty value reads as missing by default, so if empty strings are data in your model, choose the column’s semantics deliberately.

Boolean columns are the exception: a boolean cannot be missing, declares no fill at all, and therefore must be provided in every append.

Sentinel or NaN

For floating-point columns you will usually pick one of two styles.

The sentinel style keeps missingness distinguishable from every computational outcome. A mean over values that accidentally include a NaN is poisoned, while a sentinel that leaks into arithmetic is at least loud. It is also the only option for integer columns, which have no NaN:

ColumnSpec(name="samples", dtype="int32", fill_value=-1, valid_min=0)

The NaN style is the natural one when the data flows to and from NumPy or pandas, where NaN already means missing:

ColumnSpec(name="t_air", dtype="float64", fill_value=np.nan, units="degC")

The NYC taxi example uses both styles side by side on real data.

Valid ranges keep the fill honest

A sentinel only works if no genuine value can equal it. Declaring valid_min/valid_max makes that checkable. At creation time the fill must lie strictly outside the declared range, or FillValueError is raised. In the samples spec above, -1 is provably not a value, because values start at 0.

Writing missing rows

There are three ways a missing row comes to exist:

• Append None in place of a value. Whatever the column’s datatype, None is stored as that column’s fill value, so a None in a sentinel-filled integer column becomes the sentinel and a None in a NaN-filled float column becomes NaN. In a categorical column it becomes the fill code and reads back as None.

• Write the fill value itself — NaN into a NaN-filled float column, the sentinel into a sentinel-filled one. This is equivalent to writing None, and is often the natural form when the data already arrives as a NumPy array.

• Omit the column from an append entirely: the column is extended and its new rows keep the fill value.

The exception is a column with no fill value to store. A boolean column declares none so it must be supplied in every append, and a None in one raises SchemaError rather than being coerced to False. Non-nullable list columns must likewise always be provided.

Missing values in queries

Selections treat missing rows with three-valued logic, exactly as SQL and pyarrow do: a comparison with a missing value is neither true nor false but unknown, and only rows whose whole predicate evaluates true are selected. The consequence worth internalizing is that field("x") == 5 and its negation together do not cover the missing rows — those match only field("x").is_null(). The queries section defines the semantics fully.

List columns are different

A list column distinguishes a null list (no value for the row) from an empty list (a value with zero elements) using an explicit MASK dataset, not a fill value. The list columns chapter explains that in more detail.