Load a CSV file in Python

CSV (Comma Separated Values) is a widely used format for logging flight, sensor, and machine data. It is human-readable, editable with spreadsheets, and compatible with practically any data analysis tool.

Nominal is the leading platform for operationalizing test data.

This guide describes an automatable pipeline for uploading CSV files to Nominal.

Connect to Nominal

Concepts
  • Base URL: The URL through which the Nominal API is accessed (typically https://api.gov.nominal.io/api).
  • Workspace: A mechanism by which to isolate datasets; each user has one or more workspace, and data in one cannot be seen from another. Note that one token may access multiple workspaces.
  • Profile: A combination of base URL, API key, and workspace.

There are two primary ways of authenticating the Nominal Client. The first is to use a profile stored on disk, and the second is to use a token directly.

Run the following in a terminal and follow on-screen prompts to set up a connection profile:

$$ nom config profile add default
>
># Alternatively, if `nom` is missing from the path:
>$ python -m nominal config profile add default

Here, “default” can be any name chosen to represent this profile (reminder: a profile represents a base URL, API key, and workspace).

The profile will be stored in ~/.config/nominal/config.yml, and can then be used to create a client:

1from nominal.core import NominalClient
2
3client = NominalClient.from_profile("default")
4
5# Get details about the currently logged-in user to validate authentication
6# Will display an object like: `User(display_name='your_email@your_company.com', ...)`
7print(client.get_user())

If you have previously used nom to store credentials, prior to the availability of profiles, you will need to migrate your old configuration file (~/.nominal.yml) to the new format (~/.config/nominal/config.yml).

You can do this with the following command:

$nom config migrate
>
># Or, if `nom` is missing from your path:
>python -m nominal config migrate
1from nominal.core import NominalClient
2
3# Get an instance of the client using provided credentials
4client = NominalClient.from_token("<insert api key>")
5
6# Get details about the currently logged-in user to validate authentication
7# Will display an object like: `User(display_name='your_email@your_company.com', ...)`
8print(client.get_user())

NOTE: you should never share your Nominal API key with anyone. We therefore recommend that you not save it in your code and/or scripts.

  • If you trust the computer you are on, use nom to store the credential to disk.
  • Otherwise, use a password manager such as 1password or bitwarden to keep your token safe.
If you’re not sure whether your company has a Nominal tenant, please reach out to us.

Download sample data

For convenience and educational purposes, Nominal hosts test sample data on the Nominal Hugging Face account.

Let’s download a 1000 rows of data generated in the X-Plane flight simulator.

We’ll use the popular Polars library to download and inspect the data. Polars is installed automatically as part of the nominal Python library.

1import polars as pl
2
3df = pl.read_csv('hf://datasets/nominal-io/frosty-flight/frosty_flight_1k_rows.csv')
4df.write_csv('frosty_flight_1k_rows.csv')
5
6df.head().select(df.columns[:6])
source_timef_act_secf_sim_secframe_timecpu_timegpu_time
strf64f64f64f64f64
”2024-06-08T05:58:42.000Z”0.2764519.93.617280.00220.00078
”2024-06-08T05:58:51.000Z”0.2315419.94.318881.633620.13793
”2024-06-08T05:58:52.000Z”0.2643532.300793.782893.772450.00089
”2024-06-08T05:58:52.000Z”0.2992919.93.341233.772450.00089
”2024-06-08T05:58:52.000Z”0.3415730.360992.92773.772450.00089

Upload your data to Nominal

1from nominal.core import NominalClient
2
3client = NominalClient.from_profile("default") # replace with your profile name
4
5dataset = client.create_dataset('Frosty Flight')
6dataset.add_tabular_data(
7 'frosty_flight_1k_rows.csv',
8 timestamp_column = 'source_time',
9 timestamp_type = 'iso_8601',
10)
11
12print('Uploaded dataset:', dataset.rid)

Equivalently, since your CSV is already loaded in the Polars dataframe df, you can upload it with the upload_dataframe() method:

1from nominal.thirdparty.pandas import upload_dataframe
2
3dataset = upload_dataframe(
4 client,
5 df.to_pandas(),
6 "name",
7 timestamp_column='source_time',
8 timestamp_type='iso_8601'
9)
10
11print('Uploaded dataset:', dataset.rid)

After upload, navigate to Nominal’s Datasets page (login required). You’ll see your CSV at the top!

Acceptable values for timestamp_type include:

iso_8601, epoch_days, epoch_hours, epoch_minutes, epoch_seconds, epoch_milliseconds, epoch_microseconds, epoch_nanoseconds, relative_days, relative_hours, relative_minutes, relative_seconds, relative_milliseconds, relative_microseconds, relative_nanoseconds

Timestamps in the form 2024-06-08T05:58:42.000Z will have a timestamp_type of iso_8601.

Timestamps in the form 1581933170999989 will most likely be epoch_microseconds.

epoch_ timestamps refers to timestamps in Unix format.

For more information about Nominal timestamps in Python, see the nominal.ts docs page.