{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3ca4619",
   "metadata": {},
   "source": [
    "# Column types & missing values\n",
    "\n",
    "H5Col supports numeric, fixed-length string, boolean, and categorical columns,\n",
    "each with an optional physical `units`, a `description`, and a valid value range\n",
    "(`valid_min`/`valid_max`). This notebook tours them, plus how *missing* values\n",
    "work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ba203b3c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.334450Z",
     "iopub.status.busy": "2026-07-11T12:48:27.334218Z",
     "iopub.status.idle": "2026-07-11T12:48:27.470335Z",
     "shell.execute_reply": "2026-07-11T12:48:27.470009Z"
    }
   },
   "outputs": [],
   "source": [
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "import h5py\n",
    "import numpy as np\n",
    "\n",
    "from h5col import (\n",
    "    ColumnSpec,\n",
    "    FixedString,\n",
    "    OversizedStringError,\n",
    "    Table,\n",
    "    bool_dtype,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6277cac0",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Path(tempfile.gettempdir()) / \"h5col_types.h5\"\n",
    "f = h5py.File(path, \"w\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bbf969",
   "metadata": {},
   "source": [
    "## Fixed-length strings — no silent truncation\n",
    "\n",
    "HDF5 fixed-length strings are sized in **bytes**. `FixedString` enforces the\n",
    "budget and *raises* instead of silently truncating (a UTF-8 code point can take\n",
    "up to 4 bytes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5ed27796",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.471717Z",
     "iopub.status.busy": "2026-07-11T12:48:27.471607Z",
     "iopub.status.idle": "2026-07-11T12:48:27.473863Z",
     "shell.execute_reply": "2026-07-11T12:48:27.473484Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoded: [b'hello' b'world']\n",
      "refused oversized value: string value at index 0 needs 9 bytes but the column allows at most 5: 'café ☕'\n"
     ]
    }
   ],
   "source": [
    "fs = FixedString(5)          # 5 bytes\n",
    "print(\"encoded:\", fs.encode([\"hello\", \"world\"]))\n",
    "\n",
    "try:\n",
    "    fs.encode([\"café ☕\"])      # far more than 5 bytes\n",
    "except OversizedStringError as e:\n",
    "    print(\"refused oversized value:\", e)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf82961",
   "metadata": {},
   "source": [
    "## Categorical columns\n",
    "\n",
    "Give `categories` (the label set) instead of a dtype. H5Col stores compact integer *codes* and a separate categories dataset under a `CATEGORIES` group; `ordered=True` records an ordinal scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "056b496c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.474772Z",
     "iopub.status.busy": "2026-07-11T12:48:27.474702Z",
     "iopub.status.idle": "2026-07-11T12:48:27.482889Z",
     "shell.execute_reply": "2026-07-11T12:48:27.482493Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'count': array([10,  3,  7,  0], dtype=int32),\n",
       " 'flux': array([412.5,   0. ,  88.1,   5. ]),\n",
       " 'flagged': array([False,  True, False,  True]),\n",
       " 'quality': array(['good', 'suspect', 'bad', None], dtype=object)}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table = Table.create(\n",
    "    f.create_group(\"obs\"),\n",
    "    [\n",
    "        ColumnSpec(name=\"count\", dtype=\"i4\", units=\"1\",\n",
    "                   description=\"event count\"),\n",
    "        # flux is non-negative, so we mark missing with a negative sentinel:\n",
    "        # a fill value must lie strictly OUTSIDE [valid_min, valid_max].\n",
    "        ColumnSpec(name=\"flux\", dtype=\"f8\", units=\"W m-2\",\n",
    "                   valid_min=0.0, fill_value=-1.0),\n",
    "        ColumnSpec(name=\"flagged\", dtype=bool_dtype()),\n",
    "        ColumnSpec(name=\"quality\",\n",
    "                   categories=[\"good\", \"suspect\", \"bad\"], ordered=True),\n",
    "    ],\n",
    ")\n",
    "table.append({\n",
    "    \"count\":   [10, 3, 7, 0],\n",
    "    \"flux\":    [412.5, 0.0, 88.1, 5.0],\n",
    "    \"flagged\": [False, True, False, True],\n",
    "    \"quality\": [\"good\", \"suspect\", \"bad\", None],   # None -> missing category\n",
    "})\n",
    "table.read()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "36f18d52",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.483786Z",
     "iopub.status.busy": "2026-07-11T12:48:27.483723Z",
     "iopub.status.idle": "2026-07-11T12:48:27.486719Z",
     "shell.execute_reply": "2026-07-11T12:48:27.486382Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is_categorical: True\n",
      "categories: ['good', 'suspect', 'bad']\n",
      "ordered: True\n",
      "integer codes: [ 0  1  2 -1]\n",
      "decoded: ['good', 'suspect', 'bad', None]\n"
     ]
    }
   ],
   "source": [
    "q = table[\"quality\"]\n",
    "print(\"is_categorical:\", q.is_categorical)\n",
    "print(\"categories:\", list(q.categories))\n",
    "print(\"ordered:\", q.ordered)\n",
    "print(\"integer codes:\", q.codes)         # -1 marks the missing category\n",
    "print(\"decoded:\", list(q.read()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "031cbd06",
   "metadata": {},
   "source": [
    "## Missing values\n",
    "\n",
    "Every non-boolean column has a **fill value** marking missing rows. It must lie\n",
    "*outside* the valid range — here `flux` is non-negative, so we chose `-1.0`. A\n",
    "column *omitted* from an `append` leaves its new rows missing. Boolean columns\n",
    "cannot be missing, so they must always be supplied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "19d6fb03",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.487468Z",
     "iopub.status.busy": "2026-07-11T12:48:27.487414Z",
     "iopub.status.idle": "2026-07-11T12:48:27.491991Z",
     "shell.execute_reply": "2026-07-11T12:48:27.491688Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "flux values: [412.5   0.   88.1   5.   -1.   -1. ]\n",
      "flux fill value: -1.0\n",
      "missing mask: [False, False, False, False, True, True]\n"
     ]
    }
   ],
   "source": [
    "# 'flux' omitted here -> those two rows are missing.\n",
    "table.append({\n",
    "    \"count\":   [4, 9],\n",
    "    \"flagged\": [True, False],\n",
    "    \"quality\": [\"good\", \"good\"],\n",
    "})\n",
    "print(\"flux values:\", table[\"flux\"].read())\n",
    "print(\"flux fill value:\", table[\"flux\"].fill_value)\n",
    "print(\"missing mask:\", table[\"flux\"].is_missing().tolist())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9db54611",
   "metadata": {},
   "source": [
    "The canonical missing-value test compares each row against the fill value, so `is_missing()` cleanly separates real data from absent rows regardless of the sentinel chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "80ac0477",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T12:48:27.492893Z",
     "iopub.status.busy": "2026-07-11T12:48:27.492836Z",
     "iopub.status.idle": "2026-07-11T12:48:27.496322Z",
     "shell.execute_reply": "2026-07-11T12:48:27.495980Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean flux over present rows: 126.4\n"
     ]
    }
   ],
   "source": [
    "present = ~table[\"flux\"].is_missing()\n",
    "print(\"mean flux over present rows:\",\n",
    "      float(np.mean(table[\"flux\"].read()[present])))\n",
    "f.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}