Exporting data: the .h5dcnvs format

The core rule #

Anything that's metadata about a measurement, including its units, its label, which variable it is, and which observation it belongs to, is an HDF5 attribute. The only two kinds of HDF5 dataset in the whole file are a series' values array and a scalar's single value. Once you know this, the tree is predictable to browse: attributes tell you what something is, datasets give you the numbers.

Layout at a glance #

/                                        (root group, file-level attributes, see below)
├─ observations_summary                  (a JSON mirror of the whole tree, for convenience)
├─ catalog/
│   ├─ quantity_kinds/
│   │   └─ temperature/                  (one group per quantity kind, named by its key)
│   └─ variables/
│       └─ temperature/                  (one group per variable, named by its key)
└─ observations/
    └─ obs_000001/
        ├─ sample/                       (present only if the observation has a sample)
        ├─ scalars/
        │   └─ contextual_furnace_setpoint/
        │       └─ value                 (Dataset)
        ├─ series/
        │   ├─ index_time/
        │   │   └─ values                (Dataset)
        │   └─ dependent_temperature/
        │       └─ values                (Dataset)
        └─ series_groups/                (present only when series share a common index)
            └─ group_000001/

Group and field names are stable, deterministic, and human-readable. The same observation exported twice produces the same tree, and a field's HDF5 path (e.g. observations/obs_000001/series/dependent_temperature) doubles as its plain-English location.

Root attributes #

A handful of attributes on the file's root group describe the export as a whole:

Catalog: variables and quantity kinds #

catalog/quantity_kinds/<key>/ and catalog/variables/<key>/ hold the shared metadata that scalar and series fields refer back to, so it's only written once even if several fields use the same variable. A quantity kind group looks like:

catalog/quantity_kinds/temperature/
├─ @key             = "temperature"
├─ @label           = "Temperature"
├─ @canonical_units = "K"
├─ @units           = "K"          (duplicate, CF-style, see below)
├─ @symbol          = "T"
└─ @common_units    = ["degC", "degF", "K"]

A variable group is similar, and additionally carries @quantity_kind_key, which is the string you look up in catalog/quantity_kinds/ to resolve its units and definition.

The observation #

The observations/obs_000001/ group is the observation you exported, with obs_type, title, technique, notes, ingestion_status, and created_at as attributes, plus a sample/ sub-group if a sample is attached.

Inside, every scalar and series field gets its own group under scalars/ or series/, named by a readable slug (e.g. contextual_furnace_setpoint, dependent_temperature, index_time). Every field group carries the same attribute vocabulary, in two overlapping styles, side by side:

The CF-style names (units, long_name, standard_name) exist because they're widely recognized by generic scientific HDF5/NetCDF tools, so you get readable units and labels without any DataCNVS-specific plugin. The DataCNVS-style names are there for unambiguous, code-friendly lookups. Both describe the same field; use whichever fits your tool.

Series fields additionally carry series_kind ("series" or "index"), length, and, if the series shares a common index with others (e.g. a time axis shared by several dependent curves), series_group_ref, pointing at a series_groups/ entry (see next section).

Series groups #

When two or more series in the observation share a common index (most often: one time or position axis with several dependent measurements against it), they're linked via observations/obs_000001/series_groups/group_NNNNNN/. This group doesn't duplicate any data; it just lists the member paths and (as a browsing convenience only) provides HDF5 soft links into the real values datasets, so a tree browser visually shows the relationship. Code should always resolve the relationship from the series_group_ref attribute, not from link traversal.

A series with no series_group_ref attribute simply has no shared index, so treat its values as row-position ordered (point 0, point 1, point 2, …).

observations_summary #

The root-level observations_summary dataset holds the same information as the whole tree, restated as one JSON document containing the observation, its fields, and the catalog entries with their HDF5 paths. It's a convenience for scripts that would rather parse one JSON blob than walk the group tree, and it's guaranteed to be a strict mirror of what the attributes already say (never a source of extra information).