HDF5 (Hierarchical Data Format V5) is a highly flexible binary file format that stores data contiguously in memory, making memory-mapping highly effective.

It’s frequently used for storing numerical data, and can be manipulated using both the h5py library, as well as numpy and vaex.

Because hdf5 (.h5) files are so flexible, the processing logic will necessarily change per file. In this tutorial, we will show you how you can understand your hdf5 file, the layout, and walk through a specific example of how to upload some data.

Nominal is the leading platform for operationalizing test data.

This guide describes an automatable pipeline for uploading HDF5 files to Nominal.

Connect to Nominal

There are two primary ways of authenticating the Nominal Client. The first is to use a profile stored on disk, and the second is to use a token directly.

$
$ nom config profile add default
>
>
# Alternatively, if `nom` is missing from the path:
>
$ python -m nominal config profile add default

1
from nominal.core import NominalClient
2
3
client = NominalClient.from_profile("default")
4
5
# Get details about the currently logged-in user to validate authentication
6
# Will display an object like: `User(display_name='your_email@your_company.com', ...)`
7
print(client.get_user())

$
nom config migrate
>
>
# Or, if `nom` is missing from your path:
>
python -m nominal config migrate

1
from nominal.core import NominalClient
2
3
# Get an instance of the client using provided credentials
4
client = NominalClient.from_token("<insert api key>")
5
6
# Get details about the currently logged-in user to validate authentication
7
# Will display an object like: `User(display_name='your_email@your_company.com', ...)`
8
print(client.get_user())

Download sample data

For convenience and educational purposes, Nominal hosts test sample data on the Nominal Hugging Face account.

Let’s download a 3D hdf5 (h5) file containing x, y, and z coordinates.

We’ll use the hub to download our hdf5 file to disk.

You will need to make sure to install the nominal hdf5 extra: pip install 'nominal[hdf5]'

1
from huggingface_hub import hf_hub_download
2
3
repo_id = "nominal-io/hdf5-sample"
4
filename = "3d.h5"
5
6
local_path = hf_hub_download(
7
    repo_id=repo_id,
8
    filename=filename,
9
    repo_type="dataset"
10
)

local_path now contains the path to our hdf5 file.

To understand our data a bit more, we will look at the layout of the file to find where the data is and its schema.

Using the local_path from the download command above, we will inspect the file.

1
import h5py
2
3
def print_structure(name, obj):
4
    if isinstance(obj, h5py.Group):
5
        print(f"Group: {name}")
6
    elif isinstance(obj, h5py.Dataset):
7
        print(f"Dataset: {name}, Shape: {obj.shape}, Type: {obj.dtype}")
8
9
with h5py.File(local_path, 'r') as f:
10
    f.visititems(print_structure)

We will see the layout of the file, with the data residing in data/1/meshes/B in “columns” x, y, and z. Each of these columns is a 3D matrix of shape (47, 47, 47). Column E also contains x, y, and z, but these columns are empty, so there is nothing to upload.

Group: data
Group: data/1
Group: data/1/meshes
Group: data/1/meshes/B
Dataset: data/1/meshes/B/x, Shape: (47, 47, 47), Type: float64
Dataset: data/1/meshes/B/y, Shape: (47, 47, 47), Type: float64
Dataset: data/1/meshes/B/z, Shape: (47, 47, 47), Type: float64
Group: data/1/meshes/E
Group: data/1/meshes/E/x
Group: data/1/meshes/E/y
Group: data/1/meshes/E/z

Upload your data to Nominal

How we chose to upload this data is up to the use case. A benefit of hdf5, beyond flexibility, is its ease of indexing and slicing. By slicing in batches, we can avoid bringing the entire dataset into memory. This can allow us to upload files larger than memory.

We will upload our dataset in batches, flattening the dataset into x1, x2, x3, y1, y2, y3, z1, z2, z3

We will:

1. index the data into a reasonable size
2. create our flattened structure
3. create a pandas df
4. upload the batch with nominal.thirdparty.pandas.upload_dataframe

1
import pandas as pd
2
import h5py
3
import io
4
5
from nominal.core import NominalClient
6
from nominal.thirdparty.pandas import upload_dataframe
7
8
9
client = NominalClient.from_profile("default")
10
11
12
dataset = None
13
num_rows = 0
14
with h5py.File(local_path, 'r') as file:
15
    data = file["data"]["1"]["meshes"]["B"]
16
17
    batch_size = 5
18
    total_size = data['x'].shape[0]  # Assuming the same shape for 'x', 'y', and 'z'
19
20
    for start_idx in range(0, total_size, batch_size):
21
        end_idx = min(start_idx + batch_size, total_size)
22
23
        batch = {}
24
        for col in ["x", "y", "z"]:
25
            for col_idx in [0, 1, 2]:
26
                # Fetch the batch using slicing (start_idx:end_idx)
27
                batch[f"{col}{col_idx+1}"] = data[col][start_idx:end_idx, :, col_idx].ravel()
28
29
        df = pd.DataFrame(batch)
30
        df["ts"] = list(range(num_rows, num_rows+len(df)))
31
        if dataset is None:
32
            # First batch, create the dataset from dataframe
33
            dataset = upload_dataframe(
34
                client,
35
                df,
36
                name="h5_flattened",
37
                timestamp_column="ts",
38
                timestamp_type="epoch_seconds",
39
                wait_until_complete=True
40
            )
41
        else:
42
            # Subsequent batches; convert batch dataframe to buffer and append to dataset
43
            buffer = io.BytesIO()
44
            df.to_csv(buffer, index=False)
45
            buffer.seek(0)
46
            dataset.add_from_io(
47
                dataset=buffer,
48
                timestamp_column="ts",
49
                timestamp_type="epoch_seconds"
50
            )
51
        num_rows += len(df)

We can take a look at the uploaded dataset by running:

dataset.to_pandas()

That’s it!

After upload, navigate to Nominal’s Datasets page (login required). You’ll see your file at the top!