{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9030ee28",
   "metadata": {},
   "source": [
    "# h5col quickstart — your first columnar table\n",
    "\n",
    "`h5col` is a reference implementation of **H5Col**, a convention for storing\n",
    "column-oriented tabular data in HDF5. Each table is an HDF5 *group*; each column\n",
    "is a rank-1 HDF5 *dataset* that carries its own dtype, chunking, and compression.\n",
    "The design targets **write-once, read-often** workflows.\n",
    "\n",
    "This notebook builds a small environmental-sensor table, writes rows, reads them\n",
    "back, validates conformance, and reopens the file from disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9a3aa70f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.465982Z",
     "iopub.status.busy": "2026-07-11T22:03:17.465762Z",
     "iopub.status.idle": "2026-07-11T22:03:17.687861Z",
     "shell.execute_reply": "2026-07-11T22:03:17.687440Z"
    }
   },
   "outputs": [],
   "source": [
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "import h5py\n",
    "\n",
    "import h5col\n",
    "from h5col import ColumnSpec, FixedString, Table, bool_dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7a76253e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "h5col 0.1.0 | h5py 3.16.0\n"
     ]
    }
   ],
   "source": [
    "print(\"h5col\", h5col.__version__, \"| h5py\", h5py.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affdd4e6",
   "metadata": {},
   "source": [
    "## Create a table\n",
    "\n",
    "We declare the columns with `ColumnSpec`. A column can be a NumPy dtype, a\n",
    "fixed-length string (`FixedString(n_bytes)`), or the H5Col boolean type\n",
    "(`bool_dtype()`). We pick `station` as the row index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9d955ca1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.688897Z",
     "iopub.status.busy": "2026-07-11T22:03:17.688778Z",
     "iopub.status.idle": "2026-07-11T22:03:17.699018Z",
     "shell.execute_reply": "2026-07-11T22:03:17.698649Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h5col.Table '/weather' nrows=0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = Path(tempfile.gettempdir()) / \"h5col_quickstart.h5\"\n",
    "f = h5py.File(path, \"w\")\n",
    "table = Table.create(\n",
    "    f.create_group(\"weather\"),\n",
    "    [\n",
    "        ColumnSpec(name=\"station\", dtype=FixedString(6),\n",
    "                   description=\"station identifier\"),\n",
    "        ColumnSpec(name=\"temp_c\", dtype=\"f4\", units=\"degC\",\n",
    "                   valid_min=-90.0, valid_max=60.0),\n",
    "        ColumnSpec(name=\"rh_pct\", dtype=\"f4\", units=\"percent\",\n",
    "                   valid_min=0.0, valid_max=100.0),\n",
    "        ColumnSpec(name=\"qc_pass\", dtype=bool_dtype(),\n",
    "                   description=\"quality-control flag\"),\n",
    "    ],\n",
    "    title=\"Surface weather observations\",\n",
    "    index_columns=[\"station\"],\n",
    ")\n",
    "table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd0e0b3a",
   "metadata": {},
   "source": [
    "## Append rows\n",
    "\n",
    "`append()` follows the H5Col write protocol: it extends every column, writes the\n",
    "new rows, and commits the row count (`NROWS`) last. You can append in batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "71d2b76e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.699916Z",
     "iopub.status.busy": "2026-07-11T22:03:17.699852Z",
     "iopub.status.idle": "2026-07-11T22:03:17.706543Z",
     "shell.execute_reply": "2026-07-11T22:03:17.706237Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nrows: 5\n",
      "columns: ['station', 'temp_c', 'rh_pct', 'qc_pass']\n"
     ]
    }
   ],
   "source": [
    "table.append({\n",
    "    \"station\": [\"KJFK\", \"KLAX\", \"KORD\"],\n",
    "    \"temp_c\": [21.4, 27.9, 18.2],\n",
    "    \"rh_pct\": [63.0, 41.5, 72.0],\n",
    "    \"qc_pass\": [True, True, False],\n",
    "})\n",
    "table.append({\n",
    "    \"station\": [\"KDEN\", \"KSEA\"],\n",
    "    \"temp_c\": [15.0, 13.7],\n",
    "    \"rh_pct\": [55.0, 88.0],\n",
    "    \"qc_pass\": [True, True],\n",
    "})\n",
    "print(\"nrows:\", table.nrows)\n",
    "print(\"columns:\", table.column_names)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c2683e0",
   "metadata": {},
   "source": [
    "## Read the data back\n",
    "\n",
    "Read one column, or the whole table as a dict of arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4f433c9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.707477Z",
     "iopub.status.busy": "2026-07-11T22:03:17.707415Z",
     "iopub.status.idle": "2026-07-11T22:03:17.711409Z",
     "shell.execute_reply": "2026-07-11T22:03:17.710978Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stations: ['KJFK', 'KLAX', 'KORD', 'KDEN', 'KSEA']\n",
      "temp_c:    [21.4 27.9 18.2 15.  13.7]\n",
      "qc_pass:   [ True  True False  True  True]\n"
     ]
    }
   ],
   "source": [
    "print(\"stations:\", list(table[\"station\"].read()))\n",
    "print(\"temp_c:   \", table[\"temp_c\"].read())\n",
    "print(\"qc_pass:  \", table[\"qc_pass\"].read())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5ce3386f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.712338Z",
     "iopub.status.busy": "2026-07-11T22:03:17.712267Z",
     "iopub.status.idle": "2026-07-11T22:03:17.715708Z",
     "shell.execute_reply": "2026-07-11T22:03:17.715429Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'station': ['KJFK', 'KLAX', 'KORD', 'KDEN', 'KSEA'],\n",
       " 'temp_c': [21.399999618530273,\n",
       "  27.899999618530273,\n",
       "  18.200000762939453,\n",
       "  15.0,\n",
       "  13.699999809265137],\n",
       " 'rh_pct': [63.0, 41.5, 72.0, 55.0, 88.0],\n",
       " 'qc_pass': [True, True, False, True, True]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = table.read()\n",
    "{k: v.tolist() for k, v in data.items()}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "772f0ee2",
   "metadata": {},
   "source": [
    "## Validate conformance\n",
    "\n",
    "`validate()` checks the H5Col consistency rules (row-count attribute, equal\n",
    "column extents, index-column agreement, and more). It raises on any violation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "95d1bf55",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.716512Z",
     "iopub.status.busy": "2026-07-11T22:03:17.716451Z",
     "iopub.status.idle": "2026-07-11T22:03:17.718843Z",
     "shell.execute_reply": "2026-07-11T22:03:17.718568Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "valid H5Col table ✔\n"
     ]
    }
   ],
   "source": [
    "table.validate()\n",
    "print(\"valid H5Col table ✔\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d874590",
   "metadata": {},
   "source": [
    "## Reopen from disk\n",
    "\n",
    "Close the file, then reopen it read-only and re-wrap the group with `Table.open`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "42ceb7f8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-11T22:03:17.719718Z",
     "iopub.status.busy": "2026-07-11T22:03:17.719641Z",
     "iopub.status.idle": "2026-07-11T22:03:17.722751Z",
     "shell.execute_reply": "2026-07-11T22:03:17.722422Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reopened: 5 rows, Surface weather observations\n",
      "temp_c: [21.4 27.9 18.2 15.  13.7]\n"
     ]
    }
   ],
   "source": [
    "f.close()\n",
    "\n",
    "with h5py.File(path, \"r\") as f2:\n",
    "    t2 = Table.open(f2[\"weather\"])\n",
    "    print(\"reopened:\", t2.nrows, \"rows,\", t2.title)\n",
    "    print(\"temp_c:\", t2[\"temp_c\"].read())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e83d9e",
   "metadata": {},
   "source": [
    "### Recap\n",
    "\n",
    "- A table is an HDF5 group (`CLASS=\"COLUMN_TABLE\"`); columns are rank-1 datasets.\n",
    "- `Table.create` → `append` → `read`, with `validate()` for conformance.\n",
    "- Storage settings (drivers, cloud options) are plain h5py — `h5col` only owns the\n",
    "  table layout, so you open files however you like."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "examples",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}