Loading sensor data ¶

Below we list all the functions available in the NURI software specifically designed to search for and extract data from a given period of time. More details about each function, with examples on how to execute them, can be found further down this page. A live example can be executed from this Google Colab tutorial .

`get_data` ([tmin, tmax, path, sensor, ...])	Extract time-series data from a range of dates.
`data_lookup` (path, tmin, tmax[, station, offset])	Lookup available data files within target time period.
`check_continuity` (path, dataset)	Check if subsequent data file is found.
`data_offset` (tmin, tmax, station[, add])	Determine UTC to local Local offset to be applied.
`data_extractor` (dataset[, offset, rate, ...])	Retrieve data time series
`biomed_converter` (dataset, station[, rate, dest])	Convert binary data into new binary files.
`biomed_timing` (filename, rate, offset)	Read timing binary file and construct full timestamp array.
`read_axis` (filename, rate)	Read the binary file of magnetic field axis.
`get_coord` (filename)	Extracted longitude and latitude from version 2 timing data files.
`bulk_decimate` (input_dir, output_dir, ...[, rate])	Decimate data in bulk.
`get_vmr_data` (path[, rate, offset, ufactor, ...])	Extract data from individual VMR data log file
`get_gnome` (station, t0, t1[, local, rep])	Glob all files withing user-defined period and extract data.

Search & Extract ¶

Retrieve data ¶

nuri. get_data ( tmin = None , tmax = None , path = None , sensor = None , station = None , rate = None , scalar = True ) [source] ¶

Extract time-series data from a range of dates. This function should also work to extract data from individual file. However, we recommend to use the corresponding function in such case. That is, data_extractor() for Biomed data or get_vmr_data() for TwinLeaf data.

Parameters :

tmin str: Start timestamp of the target period.
tmax str: Final timestamp of the target period.
path str: Path to data repository or file. If repository, this method will execute the data_lookup() function to identify the directory structure and look for suitable data files.
sensor str: Sensor manufacturer ( biomed or twinleaf ). If None , Biomed sensor is assumed.
station int: Index of the station, 1 to 4.
rate int: Resampling rate at which data should be downsampled. If None , the data are not downsampled. This parameter is used during the reading the of Biomed sensor axis data (see read_axis() method) and the reading of the Twinleaf VMR sensor (see get_vmr_data() method).
scalar bool: Type of output value. Default is the scalar magnitude among all directions.

Returns :

data gwpy.timeseries.timeseries.TimeSeries: Full extracted magnetic field time series.

Examples

              >>> berkeley = nuri.get_data('2016-4-1','2016-4-2','/content/drive/MyDrive/CityMagData/1Hz',station=2,sensor='biomed',scalar=True)
>>> print(berkeley)
TimeSeries([49.28078686, 49.3740789 , 49.39699749, ...,
            49.23254931, 49.23953604, 49.23309161]
           unit: dimensionless,
           t0: 1459468800.0 s,
           dt: 1.0001736412571627 s,
           name: None,
           channel: None)

             

Data Lookup ¶

nuri. data_lookup ( path , tmin , tmax , station = None , offset = 0 ) [source] ¶

Lookup available data files within target time period. If direct path to binary file provided, this will simply return a list with the given path as a single element. Alternatively, and if a specific time period is provided by the user, this function will look for either zip files from the Google drive directory structure (i.e. year/month/day/hour/station) or the customised architecture (i.e. station/).

Parameters :

path str: Path to data file or repository
tmin datetime.datetime: Starting time in format YYYY-MM-DD-HH-MM
tmax datetime.datetime: Ending time in format YYYY-MM-DD-HH-MM
station int: Station number

Returns :

dataset list: List of paths to identified data files

Examples

              >>> tmin = nuri.utils.str2datetime('2018-5-7')
>>> tmax = nuri.utils.str2datetime('2018-6-4')
>>> dataset = nuri.data_lookup(path='./MagneticFieldData',tmin=tmin,tmax=tmax,station=4)
>>> print(len(dataset))
672
>>> print(dataset[0])
MagneticFieldData/2018/5/7/0/NURI-station-04/2018-5-7_0-xx.zip

             

Data continuity ¶

nuri. check_continuity ( path , dataset ) [source] ¶

Check if subsequent data file is found. This function is executed within the data_lookup() method and is used to ensure that while looping over consecutive hours, data files are indeed found and added to the list. If no files are found, an error message is prompted.

Parameters :

path str: Assumed path to next data file, the algorithm will check if file exists
dataset list: Most recent list of data file actually found so far and up to current searched date

Returns :

dataset list: Incremented list of data files found.

Data offset ¶

nuri. data_offset ( tmin , tmax , station , add = False ) [source] ¶

Determine UTC to local Local offset to be applied.

Parameters :

tmin datetime.datetime: Starting time
tmax datetime.datetime: Ending time
station int: Station number
output bool: Offset value requested

Returns :

offset float: Offset time to match time in targeted filename

Examples

              >>> tmin = nuri.str2datetime('2018-5-20')
>>> tmax = nuri.str2datetime('2018-5-30')
>>> offset = nuri.data_offset(tmin,tmax,station=4)
>>> print(offset)
14400.0

             

Biomed sensor data ¶

Extracting Biomed data ¶

nuri. data_extractor ( dataset , offset = 0 , rate = None , sensor = None , scalar = True ) [source] ¶

Retrieve data time series

Parameters :

rate int: Targeted sampling rate
scalar bool: Type of output value, default is the scalar magnitude among all directions.

Returns :

ts gwpy.timeseries.TimeSeries: Time series data, either scalar magnitude or for individual direction.

Biomed data conversion ¶

nuri. biomed_converter ( dataset , station , rate = 3960 , dest = None ) [source] ¶

Convert binary data into new binary files. This function was optimized to prepare the dataset for an entire month of data. Several downsample version of the data will be produced: 3960Hz, 1980Hz, 990Hz, 198Hz, 1Hz, and 1S/min.

Examples

              >>> nuri convert --path ~/Cloud/Google/MagneticFieldData/ \ 
>>>              --dest ~/Cloud/Dropbox/citymag/data/ --rate 1 \
>>>              --period 2016-5-15-18 2016-5-21 --station 4

             

Time Reading ¶

nuri. biomed_timing ( filename , rate , offset ) [source] ¶

Read timing binary file and construct full timestamp array.

Parameters :

filename str: Path to the timing binary file
offset datetime: Offset time to match time in targeted filename

Returns :

tgps numpy.array: Reconstructed full timing array

Sensor axes reading ¶

nuri. read_axis ( filename , rate ) [source] ¶

Read the binary file of magnetic field axis.

Parameters :

filename str: Path to data file

Returns :

data numpy.array: Magnetic field data.

Extract GPS coordinates ¶

nuri. get_coord ( filename ) [source] ¶

Extracted longitude and latitude from version 2 timing data files.

Parameters :

filename str: Name of the timing binary file.

Returns :

lat, lon float: Latitute and Longitude of the magnetic sensor.

Decimate bulk of data ¶

nuri. bulk_decimate ( input_dir , output_dir , station , tmin , tmax , rate = 1 ) [source] ¶

Decimate data in bulk. This method loops over files available within the target period and save resampled version of the data in the output data directory.

Parameters :

input_dir str: Path to input data directory.
output_dir str: Path to output directory where resampled data are saved.
station int: Index of station from which data are processed.
tmin str: Start timestamp of the target period.
tmax str: Final timestamp of the target period.
rate int: Target sampling rate

Examples

              >>> nuri.bulk_decimate(input_dir='/content/drive/MyDrive/CityMagData/3960Hz/',output_dir='1Hz/',station=2,tmin='2016-3-14-10',tmax='2016-3-14-11',rate=10)
> Extracting files from /content/drive/MyDrive/CityMagData/3960Hz//2016/3/14/10/NURI-station-02/2016-3-14_10-xx.zip
  > Process time: 2.2113912105560303s
  > Creating data with 10 sampling rate
  > Process time: 21.934987783432007s

             

Twinleaf sensor data ¶

nuri. get_vmr_data ( path , rate = None , offset = 0.0 , ufactor = 1000 , tmin = None , tmax = None , chunk_size = 60 ) [source] ¶

Extract data from individual VMR data log file

Parameters :

sample_rate int: Actual sampling rate of the recorded data
rate int: Targeted sampling rate after downsampling
ufactor float: Magnetic field factor to convert into microTesla

Returns :

data gwpy.timeseries.TimeSeries: Time series data of the scalar magnitude

GNOME data ¶

nuri. get_gnome ( station , t0 , t1 , local = False , rep = None ) [source] ¶

Glob all files withing user-defined period and extract data.

Parameters :

station str: Name of the station to be analysed
t0 str: Starting time in format YYYY-MM-DD-HH-MM
t1 str: Ending time in format YYYY-MM-DD-HH-MM
local bool: Whether we extract local (GPS) or universal (UTC) time.
rep str: Path in which the data are stored.

Returns :

data numpy.array: 2D magnetic field data array.