Retrieve decimated data points in Python
When exploring large datasets, it is often useful to decimate the data so it can be faster to download and visualize. The Nominal platform has a very fast decimation service that retains details in the signal. You can see this in action on the Nominal platform when viewing a timeseries chart in a workbook. This chart shows a decimated signal while retaining details, retrieving higher resolution data points as you zoom in.
This guide demonstrates how to retrieve decimated data points from the Nominal platform in a Jupyter notebook using Python and display them in an interactive time series chart. The chart will update to show higher-resolution data as you zoom in, similar to the behavior in the Nominal workbook.
Prerequisites
Make sure you have the following Python packages installed:
- nominal
- jupyterlab
- pandas
- plotly
- ipywidgets
You can install them all using:
Generate a sample dataset
For this guide, we’ll generate a sample dataset with 720,000 rows of random data.
This should display a Pandas DataFrame with a Time
column and a value
column with 720,000 rows.
Upload the dataset to Nominal
Before uploading the data, ensure you’re connected to Nominal.
Concepts
- Base URL: The URL through which the Nominal API is accessed (typically
https://api.gov.nominal.io/api
). - Workspace: A mechanism by which to isolate datasets; each user has one or more workspace, and data in one cannot be seen from another. Note that one token may access multiple workspaces.
- Profile: A combination of base URL, API key, and workspace.
There are two primary ways of authenticating the Nominal Client. The first is to use a profile stored on disk, and the second is to use a token directly.
Storing credentials to disk
Run the following in a terminal and follow on-screen prompts to set up a connection profile:
Here, “default” can be any name chosen to represent this profile (reminder: a profile represents a base URL, API key, and workspace).
The profile will be stored in ~/.config/nominal/config.yml
, and can then be used to create a client:
If you have previously used nom
to store credentials, prior to the availability of profiles, you will need to migrate your old configuration file (~/.nominal.yml
) to the new format (~/.config/nominal/config.yml
).
You can do this with the following command:
Directly using credentials in your scripts
NOTE: you should never share your Nominal API key with anyone. We therefore recommend that you not save it in your code and/or scripts.
Upload the dataset to Nominal using the upload_pandas()
method:
This should display the dataset metadata.
Retrieve decimated data points
Now that the dataset is uploaded, we can retrieve decimated data points using the get_decimated()
method of Channel.
This should display a Pandas DataFrame with the 2000 decimated data points.
Display the data in a time series chart
We can display the data in a time series chart using Plotly.
This will display a time series chart with the decimated data points. Note the outliers in the signal that are preserved.
Increase resolution when zooming in
When you zoom in on the chart, the resolution of the data points will not increase.
In this chapter we will add an event handler and request higher resolution data points when zooming in.
This will display a time series chart with the decimated data points. When you zoom in, the resolution of the data increases.