HDF5 User Guide

The HDF5 Data Model and File Structure

• Introduction
• The Abstract Data Model
  • File
  • Group
  • Dataset
  • Dataspace
  • Datatype
  • Attribute
  • Property List
  • Link
• The HDF5 Storage Model
  • The Abstract Storage Model: the HDF5 Format Specification
  • Concrete Storage Model
• The Structure of an HDF5 File
  • Overall File Structure
  • HDF5 Path Names and Navigation
  • Examples of HDF5 File Structures

The HDF5 Library and Programming Model

• Introduction
• The HDF5 Programming Model
  • Creating an HDF5 File
  • Creating and Initializing a Dataset
  • Closing an Object
  • Writing or Reading a Dataset to or from a File
  • Reading and Writing a Portion of a Dataset
  • Getting Information about a Dataset
  • Creating and Defining Compound Datatypes
  • Creating and Writing Extendable Datasets
  • Creating and Working with Groups
  • Working with Attributes
• The Data Transfer Pipeline

The HDF5 File

• Introduction
• File Access Modes
• File Creation and File Access Properties
• Low-level File Drivers
• Programming Model for Files
  • Creating a New File
  • Opening an Existing File
  • Closing a File
• Using h5dump to View a File
• File Function Summaries
• Creating or Opening an HDF5 File
• Closing an HDF5 File
• File Property Lists
  • Creating a Property List
  • File Creation Properties
  • File Access Properties
• Alternate File Storage Layouts and Low-level File Drivers
  • Identifying the Previously‐used File Driver
  • The POSIX (aka SEC2) Driver
  • The Direct Driver
  • The Log Driver
  • The Windows Driver
  • The STDIO Driver
  • The Memory (aka Core) Driver
  • The Family Driver
  • The Multi Driver
  • The Split Driver
  • The Parallel Driver
• Code Examples for Opening and Closing Files
  • Example Using the H5F_ACC_TRUNC Flag
  • Example with the File Creation Property List
  • Example with the File Access Property List
• Working with Multiple HDF5 Files

HDF5 Groups

• Introduction
• Description of the Group Object
  • The Group Object
  • The Hierarchy of Data Objects
  • HDF5 Path Names
  • Group Implementations in HDF5
• Using h5dump
• Group Function Summaries
• Programming Model for Groups
  • Creating a Group
  • Opening a Group and Accessing an Object in that Group
  • Creating a Dataset in a Specific Group
  • Closing a Group
  • Creating Links
  • Discovering Information about Objects
  • Discovering Objects in a Group
  • Discovering All of the Objects in the File
• Examples of File Structures

HDF5 Datasets

• Introduction
• Dataset Function Summaries
• Programming Model for Datasets
  • General Model
  • Create Dataset
  • Data Transfer Operations on a Dataset
  • Retrieve the Properties of a Dataset
• Data Transfer
  • The Data Pipeline
  • Data Pipeline Filters
  • File Drivers
  • Data Transfer Properties to Manage the Pipeline
  • Storage Strategies
  • Partial I/O Sub‐setting and Hyperslabs
  • Data Pipeline Dynamically Loaded Filters
• Allocation of Space in the File
  • Storage Allocation in the File: Early, Incremental, Late
  • Deleting a Dataset from a File and Reclaiming Space
  • Releasing Memory Resources
  • External Storage Properties
• Using HDF5 Filters
  • Using the N‐bit Filter
  • Using the Scale‐offset Filter
  • Using the SZip Filter
  • Using Dynamically-Loadable Filters

HDF5 Datatypes

• Introduction and Definitions
• Datatype Model
  • Datatype Classes and Properties
  • Predefined Datatypes
• How Datatypes are Used
  • The Datatype Object and the HDF5 Datatype API
  • Dataset Creation
  • Data Transfer (Read and Write)
  • Discovery of Data Format
  • Creating and Using User‐defined Datatypes
• Datatype Function Summaries
• Programming Model for Datatypes
  • Discovery of Datatype Properties
  • Definition of Datatypes
• Other Non-numeric Datatypes
  • Strings
  • Reference
  • ENUM
  • Opaque
  • Bitfield
• Fill Values
• Complex Combinations of Datatypes
  • Creating a Complicated Compound Datatype
  • Analyzing and Navigating a Compound Datatype
• Life Cycle of the Datatype Object
• Data Transfer: Datatype Conversion and Selection
• Text Descriptions of Datatypes: Conversion to and from

HDF5 Dataspaces and Partial I/O

• Introduction
• Dataspace Function Summaries
• Definition of Dataspace Objects and the Dataspace Programming Model
  • Dataspace Objects
  • Dataspace Programming Model
• Dataspaces and Data Transfer
  • Data Selection
  • Programming Model
• Dataspace Selection Operations and Data Transfer
• References
  • Example Uses for Region References
  • Creating References to Regions
  • Reading References to Regions
• Sample Programs

HDF5 Attributes

• Introduction
• Programming Model for Attributes
  • To Open and Read or Write an Existing Attribute
• Attribute Function Summaries
• Working with Attributes
  • The Structure of an Attribute
  • Creating, Writing, and Reading Attributes
  • Accessing Attributes by Name or Index
  • Obtaining Information Regarding an Object's Attributes
  • Iterating across an Object's Attributes
  • Deleting an Attribute
  • Closing an Attribute
• Special Issues

HDF5 Error Handling

• Introduction
• Programming Model for Error Handling
• Error Handling Function Summaries
• Basic Error Handling Operations
  • Error Stack and Error Message
  • Print and Clear an Error Stack
  • Mute Error Stack
  • Customized Printing of an Error Stack
  • Walk through the Error Stack
  • Traverse an Error Stack with a Callback Function
• Advanced Error Handling Operations
  • More Error API Functions
  • Pushing an Application Error Message onto Error Stack

Properties and Property Lists in HDF5

• Introduction
• Property List Classes, Property Lists, and Properties
  • Property List Classes
  • Property Lists
  • Properties
• Programming Model for Properties and Property Lists
  • Using Default Property Lists
  • Basic Steps of the Programming Model
  • Additional Property List Operations
• Generic Properties Interface and User-defined Properties
• Property List Function Summaries
• Additional Property List Resources
• Notes

The HDF5 Virtual Object Layer (VOL)

• Introduction
• The VOL Abstraction Layer
• Quickstart
  • Read The Documentation For The New VOL Connector
  • Use A VOL-Enabled HDF5 Library
  • Determine How You Will Set The VOL Connector
  • If Needed: Update Your Code To Load And Use A VOL Connector
  • If Using A Plugin: Make Sure The VOL Connector Is In The Search
  • Optional: Set The VOL Connector Via The Environment Variable
• VOL Connectors
  • Registration
  • Connector Versioning
  • Connector-Specific Registration Calls
  • H5Pset_vol()
  • VOL Connector Search Path
  • Parameter Strings
  • Environment Variable
• Adapting HDF5 Software to Use the VOL
  • haddr_t → H5O_token_t
  • Specific API Call Substitutions
  • Protect Native-Only API Calls
• Language Wrappers
  • C++
  • Fortran
  • Java/JNI
• Using VOL Connectors With The HDF5 Command-Line Tools
• Compatibility
  • List of HDF5 Native VOL API Calls
  • List of HDF5 VOL-Independent API Calls
  • List of Native VOL Optional Operation Values By Subclass

HDF5 File Image

• Introduction to HDF5 File Image Operations
• File Image C API Call Syntax
  • Low-level C API Routines
  • High-level C API Routine
• C API Call Semantics
  • File Image Callback Semantics
  • Initial File Image Semantics
• Examples
  • Reading an In-memory HDF5 File Image
  • In-memory HDF5 File Image Construction
  • Using HDF5 to Construct and Read a Data Packet
  • Using a Template File
• Java Signatures for File Image Operations API Calls
• Fortran Signatures for File Image Operations API Calls
  • Fortran Low-Level APIs
  • Fortran High-Level APIs

The HDF5 Event Set Interface

• Introduction

HDF5 Map Object

HDF5 High Level APIs

• HDF5 High Level Optimizations
  • Direct Chunk Write Function
• HDF5 High Level Dimension Scales
  • HDF5 Standard for Dimension Scales
  • Conceptual model
  • The HDF5 Dimension Scale Specification
  • Programming Model and API
• HDF5 High Level Images
• HDF5 High Level Lite
• HDF5 High Level Table
• HDF5 High Level Packet Table

Command Line Tools for HDF5 Files

• The HDF5 h5copy Tool
• The HDF5 h5diff Tool
• The HDF5 h5dump Tool
• The HDF5 h5format_convert Tool
• The HDF5 h5import Tool
• The HDF5 h5jam/h5unjam Tool
• The HDF5 h5ls Tool
• The HDF5 h5repack Tool
• The HDF5 h5stat Tool
• The HDF5 h5clear Tool
• The HDF5 h5debug Tool
• The HDF5 h5delete Tool
• The HDF5 h5mkgrp Tool
• The HDF5 h5repart Tool
• The HDF5 h5watch Tool

Additional Resources