Reading ISMN data (ismn.interface.ISMN_Interface)

This example shows the basic functionality to read data downloaded from the International Soil Moisture Network (ISMN). The data can be selected and downloaded for free from http://ismn.earth after registration.

For this tutorial, data for ISMN networks ‘REMEDHUS’, ‘SMOSMANIA’, ‘FMI’, ‘WEGENERNET’, ‘GTK’, ‘VAS’ and ‘RSMN’ between 2009-08-04 and 2020-12-12 was downloaded from the ISMN website. Note that - depending on when you download the data - available stations and measurement values can vary slightly from those shown in this example.

06f55d3e468549d9b881224714e343fd

ISMN files are downloaded as a compressed .zip file after selecting the data from the website. You can extract it (with any zip software) locally into one (root) folder (in this case ‘Data_separate_files’). The will be organised like this:

Data_separate_files/
├── network/
│   ├── station/
│   │   ├── sensor.stm
│   │   ├── sensor.stm
│   │   ├── ...
│   │   ├── sensor.stm
│   │   ├── static_variables.csv
│   ├── station/
│   │   ├── ...
├── network/
│   ├── ...
├── ISMN_qualityflags_description.txt
├── Metadata.xml
├── Provider_qualityflags_description.txt
└── Readme.txt

You can either read from this extracted root folder, or read from the .zip file directly (the reader will then unzip files temporarily when needed). Reading from zip is therefore slower than reading the extracted files. Extracted files are (much) larger than compressed files.

The class for reading data from extracted or compressed files is ismn.interface.ISMN_Interface. It provides functions to access single networks, stations and sensors and the measured time series for each sensor as well as metadata for each station/sensor.

ISMN_Interface expects the path to the downloaded and locally stored ISMN data (zip-file or extracted root folder) as the only required argument.

[1]:
from ismn.interface import ISMN_Interface
import numpy as np
import matplotlib.pyplot as plt  # matplotlib is not installed automatically
%matplotlib inline

# Either a .zip file or one folder that contains all networks, here we read from .zip
data_path = "/tmp/ismn/Data_separate_files_20090804_20201212.zip"
ismn_data = ISMN_Interface(data_path)
Processing metadata for all ismn stations into folder /tmp/ismn/Data_separate_files_20090804_20201212.zip.
This may take a few minutes, but is only done once...
Hint: Use `parallel=True` to speed up metadata generation for large datasets
Files Processed: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 110/110 [00:20<00:00,  5.34it/s]
Metadata generation finished after 20 Seconds.
Metadata and Log stored in /tmp/ismn/python_metadata
Found existing ismn metadata in /tmp/ismn/python_metadata/Data_separate_files_20090804_20201212.csv.

The first time you initialize ISMN_Interface for a dataset, it will collect metadata for all sensors in your data collection from various files (this is always done for all available networks). The program will iterate through all files and collect information such as station names, sensor time coverage, measurement depths, landcover/climate classes, soil properties etc. for each sensor. Depending on the number of files and whether zipped/extracted files are used, this step can take a while. A log file is created in the displayed path. Parallel processing for this step can be activated manually by choosing ISMN_Interface(data_path, parallel=True) and will speed up metadata collection significantly. By default this step will create a folder called python_metadata (which contains the collected metadata as a .csv file) and place it inside the passed root directory. The next time the reader is created it will use python_metadata (if it is found) instead of generating it again (a different path to generate and search the metadata in can be passed via ISMN_Interface(data_path, meta_path='/custom/meta/path').

Note: When changing the data (e.g. if you add or remove folders from the collection that is passed to the reader) make sure to delete the ``python_metadata`` folder and its content to force re-generating it. Otherwise data and metadata won’t match anymore!!

To limit reading to a selection of networks from the beginning (e.g. to load one network from a large collection, which is faster than loading all networks and then selecting one), their names can be passed to the reader (e.g. ISMN_Interface(data_path, network=['REMEDHUS', 'SMOSMANIA']). Otherwise, all networks stored under the given path are loaded. Limiting the number of networks when calling the reader will result in faster initialization, because less files are loaded (this does not affect the one-time metadata collection, which is always done for all networks in data_path).

You can call ISMN_Interface(data_path, keep_loaded_data=True), which will cause that all sensor time series - once read - are kept in memory for faster subsequent calls. This can fill up your memory and is only recommended for small data collections, respectively if only a few networks are initialized.

Finally, you can define a custom temporary root folder, e.g. ISMN_Interface(data_path, temp_root='/tmp/my_tempdata'). By default this depends on your OS (e.g. /tmp for Linux), and is used by the reader to store some temporary files, e.g. when extracting while reading from .zip. The default temp_root is cleared automatically by your OS, so we recommend not to change this (unless necessary e.g. because of access restrictions).

ISMN Components (Overview)

Here we give a short overview over the components that build a ISMN data set. Each component provides various functions to work with its data and access sub-components. See the python module docs for more details. This is just a basic overview of how to pick data out of the collection. In practice, it is often required to loop over certain stations. This is described later in this tutorial. The components (hierarchically ordered) are NetworkCollection <- Network(s) <- Station(s) <- Sensor(s).

Collection

The ISMN_Interface object holds a ‘collection’ container for all loaded networks (e.g. ‘FMI’, ‘REMEDHUS’, …). Each Network contains multiple Stations (e.g. ‘SAA111’, ‘SA112’, … for the ‘FMI’ network). Each Station contains multiple Sensors (names not shown in this overview). You can access Networks directly from the reader, and subsequently access Stations and their Sensors.

[2]:
ismn_data.collection #  or just:  ismn_data  - overview over loaded networks and their stations
[2]:
FMI: ['SAA111', 'SAA112', 'SAA120', 'SOD011', 'SOD012', 'SOD013', 'SOD021', 'SOD022', 'SOD023', 'SOD031', 'SOD032', 'SOD033', 'SOD071', 'SOD072', 'SOD073', 'SOD081', 'SOD082', 'SOD083', 'SOD091', 'SOD092', 'SOD093', 'SOD101', 'SOD102', 'SOD103', 'SOD130', 'SOD140', 'SODAWS'],
GTK: ['IlomantsiII', 'Kuusamo', 'PoriII', 'Suomussalmi'],
REMEDHUS: ['Canizal', 'Carramedina', 'Carretoro', 'CasaPeriles', 'ConcejodelMonte', 'ElCoto', 'ElTomillar', 'Granja-g', 'Guarena', 'Guarrati', 'LaAtalaya', 'LaCruzdeElias', 'LasArenas', 'LasBodegas', 'LasBrozas', 'LasEritas', 'LasTresRayas', 'LasVacas', 'LasVictorias', 'LlanosdelaBoveda', 'Paredinas', 'Zamarron'],
RSMN: ['Adamclisi', 'Alexandria', 'Bacles', 'Banloc', 'Barlad', 'Calarasi', 'ChisineuCris', 'Corugea', 'Cotnari', 'Darabani', 'Dej', 'Dumbraveni', 'Iasi', 'Oradea', 'RosioriideVede', 'SannicolauMare', 'SatuMare', 'Slatina', 'Slobozia', 'Tecuci'],
SMOSMANIA: ['Barnas', 'Berzeme', 'CabrieresdAvignon', 'Condom', 'CreondArmagnac', 'LaGrandCombe', 'Lahas', 'LezignanCorbieres', 'Mazan-Abbaye', 'Mejannes-le-Clap', 'Montaut', 'Mouthoumet', 'Narbonne', 'PeyrusseGrande', 'Pezenas', 'Pezenas-old', 'Prades-le-Lez', 'Sabres', 'SaintFelixdeLauragais', 'Savenes', 'Urgons', 'Villevielle'],
VAS: ['MelbexI', 'MelbexII', 'ValenciaAnchorStation'],
WEGENERNET: ['15', '19', '27', '34', '50', '54', '6', '77', '78', '84', '85', '99']

Network

You can select a specific network from the collection above via its name. Here we pick the network ‘SMOSMANIA’ from our loaded set. Networks are sorted alphabetically, so you can also pass a number here, e.g. ismn_data[4] to get the fourth network from the list. This will (again) display all stations for that network. Note that we can call a network directly from ismn_data without using the collection.

[3]:
ismn_data['SMOSMANIA']  #  overview over stations in SMOSMANIA network
                        #  equivalent to:
                            #  - ismn_data.collection['SMOSMANIA']
                            #  - ismn_data[4]
[3]:
Stations in 'SMOSMANIA': ['Barnas', 'Berzeme', 'CabrieresdAvignon', 'Condom', 'CreondArmagnac', 'LaGrandCombe', 'Lahas', 'LezignanCorbieres', 'Mazan-Abbaye', 'Mejannes-le-Clap', 'Montaut', 'Mouthoumet', 'Narbonne', 'PeyrusseGrande', 'Pezenas', 'Pezenas-old', 'Prades-le-Lez', 'Sabres', 'SaintFelixdeLauragais', 'Savenes', 'Urgons', 'Villevielle']

Station

A network consists of multiple stations, multiple variables can be measured by different sensors at a station. You can select a specific station for a network via its name (stations are also sorted alphabetically and can be accessed by index). Here we access the Station ‘SaintFelixdeLauragais’ from the ‘SMOSMANIA’ network.

[4]:
ismn_data.collection['SMOSMANIA']['SaintFelixdeLauragais']  #  overview over sesnors at SaintFelixdeLauragais
                                                            #  equivalent to ismn_data[4][18]
[4]:
Sensors at 'SaintFelixdeLauragais': ['PT-100_soil_temperature_0.050000_0.050000', 'PT-100_soil_temperature_0.100000_0.100000', 'PT-100_soil_temperature_0.200000_0.200000', 'PT-100_soil_temperature_0.300000_0.300000', 'ThetaProbe-ML2X_soil_moisture_0.050000_0.050000', 'ThetaProbe-ML2X_soil_moisture_0.100000_0.100000', 'ThetaProbe-ML2X_soil_moisture_0.200000_0.200000', 'ThetaProbe-ML2X_soil_moisture_0.300000_0.300000', 'ThetaProbe-ML3_soil_moisture_0.200000_0.200000']

Each station has a metadata attribute. The station metadata contains all metadata variables from all sensors that measure at the station (such as the sensor type, soil properties etc. per sensor). Therefore, the station metadata can be different for different depths. You can call MetaData directly, or convert it to either a DataFrame (MetaData.to_pd()) or a dictionary (MetaData.to_dict()) of form:

{var_name: [(value, depth_from, depth_to),
            ...],
...}

In the example below, we read metadata without conversion. The first value in each Variable is the name of the metadata variable, the second the actual value for the variable. The third value is the depth range (depth_from, depth_to) that the value applies to - e.g. for soil properties (taken from the Harmonized World Soil Data Base) multiple layers are provided together with the ISMN data and during metadata generation the best matching depth for a sensor is selected. Some values apply to a specific depth (depth_from=depth_to) while others may apply to a depth range (usually depends on the network).

[5]:
ismn_data.collection['SMOSMANIA']['SaintFelixdeLauragais'].metadata  #  Get metadata for the station
[5]:
MetaData([
  MetaVar([clay_fraction, 22.8, Depth([0.05, 0.05])]),
  MetaVar([climate_KG, Cfb, None]),
  MetaVar([climate_insitu, unknown, None]),
  MetaVar([elevation, 337.0, None]),
  MetaVar([instrument, PT-100, Depth([0.05, 0.05])]),
  MetaVar([latitude, 43.4417, None]),
  MetaVar([lc_2000, 10, None]),
  MetaVar([lc_2005, 10, None]),
  MetaVar([lc_2010, 10, None]),
  MetaVar([lc_insitu, unknown, None]),
  MetaVar([longitude, 1.88, None]),
  MetaVar([network, SMOSMANIA, None]),
  MetaVar([organic_carbon, 1.15, Depth([0.05, 0.05])]),
  MetaVar([sand_fraction, 43.5, Depth([0.05, 0.05])]),
  MetaVar([saturation, 0.44, Depth([0.05, 0.05])]),
  MetaVar([silt_fraction, 33.7, Depth([0.05, 0.05])]),
  MetaVar([station, SaintFelixdeLauragais, None]),
  MetaVar([timerange_from, 2009-08-04 00:00:00, None]),
  MetaVar([timerange_to, 2020-01-01 00:00:00, None]),
  MetaVar([variable, soil_temperature, Depth([0.05, 0.05])]),
  MetaVar([clay_fraction, 22.4, Depth([0.1, 0.1])]),
  MetaVar([instrument, PT-100, Depth([0.1, 0.1])]),
  MetaVar([organic_carbon, 0.84, Depth([0.1, 0.1])]),
  MetaVar([sand_fraction, 40.3, Depth([0.1, 0.1])]),
  MetaVar([saturation, 0.43, Depth([0.1, 0.1])]),
  MetaVar([silt_fraction, 37.3, Depth([0.1, 0.1])]),
  MetaVar([variable, soil_temperature, Depth([0.1, 0.1])]),
  MetaVar([clay_fraction, 23.9, Depth([0.2, 0.2])]),
  MetaVar([instrument, PT-100, Depth([0.2, 0.2])]),
  MetaVar([organic_carbon, 0.97, Depth([0.2, 0.2])]),
  MetaVar([sand_fraction, 39.7, Depth([0.2, 0.2])]),
  MetaVar([saturation, 0.44, Depth([0.2, 0.2])]),
  MetaVar([silt_fraction, 36.4, Depth([0.2, 0.2])]),
  MetaVar([variable, soil_temperature, Depth([0.2, 0.2])]),
  MetaVar([clay_fraction, 29.4, Depth([0.3, 0.3])]),
  MetaVar([instrument, PT-100, Depth([0.3, 0.3])]),
  MetaVar([organic_carbon, 0.7, Depth([0.3, 0.3])]),
  MetaVar([sand_fraction, 32.0, Depth([0.3, 0.3])]),
  MetaVar([saturation, 0.44, Depth([0.3, 0.3])]),
  MetaVar([silt_fraction, 38.6, Depth([0.3, 0.3])]),
  MetaVar([variable, soil_temperature, Depth([0.3, 0.3])]),
  MetaVar([instrument, ThetaProbe-ML2X, Depth([0.05, 0.05])]),
  MetaVar([variable, soil_moisture, Depth([0.05, 0.05])]),
  MetaVar([instrument, ThetaProbe-ML2X, Depth([0.1, 0.1])]),
  MetaVar([variable, soil_moisture, Depth([0.1, 0.1])]),
  MetaVar([instrument, ThetaProbe-ML2X, Depth([0.2, 0.2])]),
  MetaVar([timerange_to, 2017-09-01 11:00:00, None]),
  MetaVar([variable, soil_moisture, Depth([0.2, 0.2])]),
  MetaVar([instrument, ThetaProbe-ML2X, Depth([0.3, 0.3])]),
  MetaVar([variable, soil_moisture, Depth([0.3, 0.3])]),
  MetaVar([instrument, ThetaProbe-ML3, Depth([0.2, 0.2])]),
  MetaVar([timerange_from, 2017-10-16 13:00:00, None])
])

Sensor

Accessing sensors at a station works similar to accessing stations in a network. By default, the name is created from the instrument type, the measured variable and the depth layer that the sensor operates in.

[6]:
ismn_data['SMOSMANIA']['SaintFelixdeLauragais']['ThetaProbe-ML2X_soil_moisture_0.050000_0.050000']
#  equivalent to ismn_data[4][18][4]
[6]:
ThetaProbe-ML2X_soil_moisture_0.050000_0.050000

Each sensor has a measurement time series (access via Sensor.read_data()) and sensor specific metadata (via Sensor.metadata) assigned. Here we convert metadata to a data frame.

[7]:
sensor = ismn_data['SMOSMANIA']['SaintFelixdeLauragais']['ThetaProbe-ML2X_soil_moisture_0.050000_0.050000']
print(sensor.metadata.to_pd())
ts = sensor.read_data()
ax = ts.plot(figsize=(12,4))
ax.set_xlabel("Time [year]")
ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
display(ts)
variable        key
clay_fraction   val                            22.8
                depth_from                     0.05
                depth_to                       0.05
climate_KG      val                             Cfb
climate_insitu  val                         unknown
elevation       val                           337.0
instrument      val                 ThetaProbe-ML2X
                depth_from                     0.05
                depth_to                       0.05
latitude        val                         43.4417
lc_2000         val                              10
lc_2005         val                              10
lc_2010         val                              10
lc_insitu       val                         unknown
longitude       val                            1.88
network         val                       SMOSMANIA
organic_carbon  val                            1.15
                depth_from                     0.05
                depth_to                       0.05
sand_fraction   val                            43.5
                depth_from                     0.05
                depth_to                       0.05
saturation      val                            0.44
                depth_from                     0.05
                depth_to                       0.05
silt_fraction   val                            33.7
                depth_from                     0.05
                depth_to                       0.05
station         val           SaintFelixdeLauragais
timerange_from  val             2009-08-04 00:00:00
timerange_to    val             2020-01-01 00:00:00
variable        val                   soil_moisture
                depth_from                     0.05
                depth_to                       0.05
Name: data, dtype: object
soil_moisture soil_moisture_flag soil_moisture_orig_flag
date_time
2009-08-04 00:00:00 0.1372 G M
2009-08-04 01:00:00 0.1368 G M
2009-08-04 02:00:00 0.1364 G M
2009-08-04 03:00:00 0.1361 G M
2009-08-04 04:00:00 0.1357 G M
... ... ... ...
2019-12-31 20:00:00 0.2500 G M
2019-12-31 21:00:00 0.2500 G M
2019-12-31 22:00:00 0.2500 G M
2019-12-31 23:00:00 0.2500 G M
2020-01-01 00:00:00 0.2500 G M

83939 rows × 3 columns

../_images/examples_interface_23_2.png

Some metadata is different for each sensor (e.g. time series range), some depends on the location of the station and is therefore shared by multiple sensors at one station (landcover and climate classes etc.). Sometime metadata is missing (if not provided, indicated by ‘unknown’, or NaN). Some meta data depends on the depth of a sensor (e.g. soil properties), during metadata collection (in the beginning) these values were collected and assigned.

[8]:
sensor.metadata.to_dict()
[8]:
{'clay_fraction': [(22.8, 0.05, 0.05)],
 'climate_KG': [('Cfb', None, None)],
 'climate_insitu': [('unknown', None, None)],
 'elevation': [(337.0, None, None)],
 'instrument': [('ThetaProbe-ML2X', 0.05, 0.05)],
 'latitude': [(43.4417, None, None)],
 'lc_2000': [(10, None, None)],
 'lc_2005': [(10, None, None)],
 'lc_2010': [(10, None, None)],
 'lc_insitu': [('unknown', None, None)],
 'longitude': [(1.88, None, None)],
 'network': [('SMOSMANIA', None, None)],
 'organic_carbon': [(1.15, 0.05, 0.05)],
 'sand_fraction': [(43.5, 0.05, 0.05)],
 'saturation': [(0.44, 0.05, 0.05)],
 'silt_fraction': [(33.7, 0.05, 0.05)],
 'station': [('SaintFelixdeLauragais', None, None)],
 'timerange_from': [(Timestamp('2009-08-04 00:00:00'), None, None)],
 'timerange_to': [(Timestamp('2020-01-01 00:00:00'), None, None)],
 'variable': [('soil_moisture', 0.05, 0.05)]}

Some other important functions

Each component (network, station, sensor) contains different functions to handle its data. ISMN_Interface provides general functions to filter and iterate over its components and visualize them.

Find nearest station

The data collection in ISMN_Interface contains a grid object that lists the locations of all stations in all active networks. For more details see https://github.com/TUW-GEO/pygeogrids

[9]:
import pandas as pd
grid = ismn_data.collection.grid
gpis, lons, lats = grid.get_grid_points()
pd.DataFrame(index=pd.Index(gpis, name='gpi'),
             data={'lon': lons, 'lat': lats}).T
[9]:
gpi 0 1 2 3 4 5 6 7 8 9 ... 100 101 102 103 104 105 106 107 108 109
lon 27.55062 27.55076 27.53543 26.63378 26.63378 26.63378 26.65176 26.65162 26.65196 26.65064 ... 15.81499 15.94361 15.96578 15.75960 15.85507 15.90710 15.92462 16.04056 15.78112 16.03337
lat 68.33019 68.33025 68.33881 67.36187 67.36179 67.36195 67.36691 67.36685 67.36691 67.36702 ... 46.97232 46.97125 46.95952 46.94327 46.99726 46.93296 46.93291 46.93427 46.91691 46.92135

2 rows × 110 columns

[10]:
# Using the GPI or coordinates, a station from **all** stations in **all** networks in the collection can be selected.
station, dist = ismn_data.collection.get_nearest_station(27.0, 68.0)
print(f'Station {station.name} is {int(dist)} metres away from the passed coordinates:')
assert ismn_data.collection.station4gpi(0) == station # same result when selecting with GPI
station
Station SAA111 is 43348 metres away from the passed coordinates:
[10]:
Sensors at 'SAA111': ['CS215_air_temperature_-2.000000_-2.000000', '5TE_soil_temperature_0.200000_0.200000', '5TE_soil_temperature_0.400000_0.400000', '5TE_soil_moisture_0.800000_0.800000', '5TE_soil_moisture_0.400000_0.400000', '5TE_soil_moisture_0.200000_0.200000', '5TE_soil_temperature_0.050000_0.050000', '5TE_soil_moisture_0.050000_0.050000', '5TE_soil_moisture_0.100000_0.100000', '5TE_soil_temperature_0.800000_0.800000', '5TE_soil_temperature_0.100000_0.100000']

Find network for a specific station

ISMN_Interface provides a function to find the network when only the name of a station is known. Here we simply read data for the first available sensor at the fist station in the found network.

[11]:
network = ismn_data.network_for_station('SAA111', name_only=False)
station = network[0]
sensor = station[0]
display(network, station, sensor)
sensor.read_data()
Stations in 'FMI': ['SAA111', 'SAA112', 'SAA120', 'SOD011', 'SOD012', 'SOD013', 'SOD021', 'SOD022', 'SOD023', 'SOD031', 'SOD032', 'SOD033', 'SOD071', 'SOD072', 'SOD073', 'SOD081', 'SOD082', 'SOD083', 'SOD091', 'SOD092', 'SOD093', 'SOD101', 'SOD102', 'SOD103', 'SOD130', 'SOD140', 'SODAWS']
Sensors at 'SAA111': ['CS215_air_temperature_-2.000000_-2.000000', '5TE_soil_temperature_0.200000_0.200000', '5TE_soil_temperature_0.400000_0.400000', '5TE_soil_moisture_0.800000_0.800000', '5TE_soil_moisture_0.400000_0.400000', '5TE_soil_moisture_0.200000_0.200000', '5TE_soil_temperature_0.050000_0.050000', '5TE_soil_moisture_0.050000_0.050000', '5TE_soil_moisture_0.100000_0.100000', '5TE_soil_temperature_0.800000_0.800000', '5TE_soil_temperature_0.100000_0.100000']
CS215_air_temperature_-2.000000_-2.000000
[11]:
air_temperature air_temperature_flag air_temperature_orig_flag
date_time
2012-09-19 11:00:00 0.1 G M
2012-09-19 12:00:00 0.1 G M
2012-09-19 13:00:00 0.2 G M
2012-09-19 14:00:00 0.2 G M
2012-09-19 15:00:00 0.1 G M
... ... ... ...
2020-12-12 19:00:00 -11.4 G M
2020-12-12 20:00:00 -11.2 G M
2020-12-12 21:00:00 -10.8 G M
2020-12-12 22:00:00 -10.3 G M
2020-12-12 23:00:00 -9.6 G M

61210 rows × 3 columns

Metadata overview

To get an overview over all metadata for the currently loaded networks, the attribute ISMN_Interface.metadata can be called. This will return a pandas DataFrame with all available metadata variables for different depths as (multi-index) columns and sensors as rows. The first column idx is the index of the sensor/filehandler, in line with what ISMN_Interface.get_dataset_ids() returns, and can therefore be used to read data via ISMN_Interface.read() as shown in the next section.

[12]:
ismn_data.metadata
[12]:
variable clay_fraction climate_KG climate_insitu elevation ... timerange_from timerange_to variable file_path file_type
key depth_from depth_to val depth_from depth_to val depth_from depth_to val depth_from ... depth_to val depth_from depth_to val depth_from depth_to val val val
0 NaN NaN NaN NaN NaN Dfc NaN NaN unknown NaN ... NaN 2012-09-19 11:00:00 NaN NaN 2020-12-12 23:00:00 -2.0 -2.0 air_temperature FMI/SAA111/FMI_FMI_SAA111_ta_-2.000000_-2.0000... ceop_sep
1 0.0 0.3 4.0 NaN NaN Dfc NaN NaN unknown NaN ... NaN 2012-10-14 00:00:00 NaN NaN 2020-12-12 23:00:00 0.2 0.2 soil_temperature FMI/SAA111/FMI_FMI_SAA111_ts_0.200000_0.200000... ceop_sep
2 0.3 1.0 4.0 NaN NaN Dfc NaN NaN unknown NaN ... NaN 2012-10-14 00:00:00 NaN NaN 2020-12-12 23:00:00 0.4 0.4 soil_temperature FMI/SAA111/FMI_FMI_SAA111_ts_0.400000_0.400000... ceop_sep
3 0.3 1.0 4.0 NaN NaN Dfc NaN NaN unknown NaN ... NaN 2012-10-14 00:00:00 NaN NaN 2020-12-12 23:00:00 0.8 0.8 soil_moisture FMI/SAA111/FMI_FMI_SAA111_sm_0.800000_0.800000... ceop_sep
4 0.3 1.0 4.0 NaN NaN Dfc NaN NaN unknown NaN ... NaN 2012-10-14 00:00:00 NaN NaN 2020-12-12 23:00:00 0.4 0.4 soil_moisture FMI/SAA111/FMI_FMI_SAA111_sm_0.400000_0.400000... ceop_sep
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
670 0.0 0.3 20.0 NaN NaN Dfb NaN NaN unknown NaN ... NaN 2009-08-04 00:00:00 NaN NaN 2013-06-18 16:00:00 0.3 0.3 soil_temperature WEGENERNET/99/WEGENERNET_WEGENERNET_99_ts_0.30... ceop_sep
671 NaN NaN NaN NaN NaN Dfb NaN NaN unknown NaN ... NaN 2009-08-04 00:00:00 NaN NaN 2020-12-12 23:00:00 -2.0 -2.0 air_temperature WEGENERNET/99/WEGENERNET_WEGENERNET_99_ta_-2.0... ceop_sep
672 0.0 0.3 20.0 NaN NaN Dfb NaN NaN unknown NaN ... NaN 2013-10-23 11:00:00 NaN NaN 2020-12-12 23:00:00 0.2 0.2 soil_moisture WEGENERNET/99/WEGENERNET_WEGENERNET_99_sm_0.20... ceop_sep
673 0.0 0.3 20.0 NaN NaN Dfb NaN NaN unknown NaN ... NaN 2013-10-23 11:00:00 NaN NaN 2020-12-12 23:00:00 0.2 0.2 soil_temperature WEGENERNET/99/WEGENERNET_WEGENERNET_99_ts_0.20... ceop_sep
674 NaN NaN NaN NaN NaN Dfb NaN NaN unknown NaN ... NaN 2009-08-04 00:00:00 NaN NaN 2020-12-12 15:00:00 -1.5 -1.5 precipitation WEGENERNET/99/WEGENERNET_WEGENERNET_99_p_-1.50... ceop_sep

675 rows × 62 columns

Selecting and reading specific sensors

We can filter the dataset a priori and get IDs of sensors that measure a specific variable. The ID can then be used to read the data directly. Here we extract the IDs of sensors in our data set that measure ‘soil_temperature’ in 0 to 1 meter depth and within a specific land cover and climate class (all conditions must be fulfilled).

[13]:
ids = ismn_data.get_dataset_ids(variable='soil_temperature',
                                max_depth=1,
                                filter_meta_dict={'lc_2005': 130, 'climate_KG': 'Csb'})
ids
[13]:
[464, 465, 466, 467]

Now we can use the so found IDs to read data from the according sensors.

[14]:
ts, meta = ismn_data.read(ids[1], return_meta=True)
meta
ax = ts.plot(figsize=(12,4), title=f'Time series for ID {ids[1]}')
ax.set_xlabel("Time [year]")
ax.set_ylabel("Soil Temp. [°C]")
[14]:
Text(0, 0.5, 'Soil Temp. [°C]')
../_images/examples_interface_38_1.png

Plot station locations

We can define a similar query to plot station locations for a specific variable on a map. If a min/max depth is passed, only stations with a sensor that measures within the passed range are included. Note that this kind of visualisation needs additional (optional) packages installed (use conda to install matplotlib and cartopy).

[15]:
import cartopy.crs as ccrs
#plot available station on a map
fig, axs = plt.subplots(1, 2, figsize=(16,10), subplot_kw={'projection': ccrs.Robinson()})
ismn_data.plot_station_locations('soil_moisture', min_depth=0., max_depth=0.1, ax=axs[0])
ismn_data.plot_station_locations('soil_temperature', min_depth=0.5, ax=axs[1])
plt.show()

../_images/examples_interface_40_0.png

Network data citations/references

When using ISMN data in a publication, please cite:

  • Dorigo, W. A., Wagner, W., Hohensinn, R., Hahn, S., Paulik, C., Xaver, A., Gruber, A., Drusch, M., Mecklenburg, S., van Oevelen, P., Robock, A., and Jackson, T. (2011). The International Soil Moisture Network: a data hosting facility for global in situ soil moisture measurements, Hydrol. Earth Syst. Sci., 15, 1675-1698, doi:10.5194/hess-15-1675-2011

  • Dorigo, W.A. , Xaver, A. Vreugdenhil, M. Gruber, A., Hegyiová, A. Sanchis-Dufau, A.D., Zamojski, D. , Cordes, C., Wagner, W., and Drusch, M. (2013). Global Automated Quality Control of In situ Soil Moisture data from the International Soil Moisture Network. Vadose Zone Journal, 12, 3, doi:10.2136/vzj2012.0097

In addition, it is required that you cite the networks you are using, information on how to cite a network can be found on https://ismn.geo.tuwien.ac.at/en/networks/ and in the downloaded README file. The ISMN package provides functions to export citations for a single network, and for all networks in a collection (e.g. in case you don’t use all the networks you downloaded before) as plain text.

[16]:
ismn_data['WEGENERNET'].get_citations()
[16]:
['Kirchengast, G., Kabas, T., Leuprecht, A., Bichler, C. & Truhetz, H. (2014), ‘Wegenernet: A pioneering high-resolution network for monitoring weather and climate’, Bulletin of the American Meteorological Society 95.',
 'Fuchsberger, J., Kirchengast, G. & Kabas, T. (2020), ‘Wegenernet high-resolution weather and climate data 2007 to 2019’, Earth System Science Data Discussions 2020, 1–49.']
[17]:
ismn_data.collection.export_citations(out_file='/tmp/ismn/citations_for_my_subset.txt')
[17]:
OrderedDict([('FMI',
              ['Ikonen, J., Smolander, T., Rautiainen, K., Cohen, J., Lemmetyinen, J., Salminen, M. & Pulliainen, J. (2018), ‘Spatially distributed evaluation of esa cci soil moisture products in a northern boreal forest environment’, Geosciences 8(2), 51.',
               'Ikonen, J., Vehvilaeinen, J., Rautiainen, K., Smolander, T., Lemmetyinen, J., Bircher, S. & Pulliainen, J. (2015), ‘The sodankylae in-situ soil moisture observation network: an example application to earth observation data product evaluation’, GID 5(2), 599–629.']),
             ('GTK',
              ['We acknowledge the work of Pekka Haenninen, Raimo Sutinen in support of the ISMN.']),
             ('REMEDHUS',
              ['Gonzalez-Zamora, A., Sanchez, N., Pablos, M. & Martinez-Fernandez, J. (2018), ‘Cci soil moisture assessment with smos soil moisture and in situ data under different environmental conditions and spatial scales in spain’, Remote Sensing of Environment 225.']),
             ('RSMN', ['No reference(s) for network RSMN available.']),
             ('SMOSMANIA',
              ['Calvet, J.-C., Fritz, N., Berne, C., Piguet, B., Maurel, W. & Meurey, C. (2016), ‘Deriving pedotransfer functions for soil quartz fraction in southern france from reverse modeling’, SOIL 2(4), 615–629.',
               'Calvet, J.-C., Fritz, N., Froissard, F., Suquia, D., Petitpa, A., and Piguet, B.: In situ soil moisture observations for the CAL/VAL of SMOS: the SMOSMANIA network, International Geoscience and Remote Sensing Symposium, IGARSS, Barcelona, Spain, 23-28 July 2007, 1196-1199, doi:10.1109/IGARSS.2007.4423019, 2007.',
               'Albergel, C., Ruediger, C., Pellarin, T., Calvet, J.-C., Fritz, N., Froissard, F., Suquia, D., Petitpa, A., Piguet, B., and Martin, E.: From near-surface to root-zone soil moisture using an exponential filter: an assessment of the method based on insitu observations and model simulations, Hydrol. Earth Syst. Sci., 12, 1323–1337, 2008.']),
             ('VAS',
              ['We acknowledge the work of Mike Schwank, Jean-Pierre Wigneron, Yann H. Kerr, Diego Intrigliolo, Jennifer Grant, Ernesto Lopez-Baeza (http://nimbus.uv.es/) in support of the ISMN.']),
             ('WEGENERNET',
              ['Kirchengast, G., Kabas, T., Leuprecht, A., Bichler, C. & Truhetz, H. (2014), ‘Wegenernet: A pioneering high-resolution network for monitoring weather and climate’, Bulletin of the American Meteorological Society 95.',
               'Fuchsberger, J., Kirchengast, G. & Kabas, T. (2020), ‘Wegenernet high-resolution weather and climate data 2007 to 2019’, Earth System Science Data Discussions 2020, 1–49.'])])

Selecting and iterating over sensors

It is often desired to iterate over all sensors that fulfill certain requirements (e.g. that measure soil moisture in a certain depth, and/or for a certain land cover class). For these cases the collection (and other components) provide iterators that take keywords and values for filtering the loaded networks/stations/sensor and iterating over the filtered data (of a whole collection, a network, or a station).

Select by variable and depth

In this example we iterate over all sensors in the previously loaded collection (i.e. over all activated networks) that measure ‘soil_moisture’ in any depth (range) between 0 and 0.05 meters.

[18]:
for network, station, sensor in ismn_data.collection.iter_sensors(variable='soil_moisture',
                                                                  depth=[0., 0.05]):
    display(network)
    display(station)
    display(sensor)

    data = sensor.read_data()

    print('\033[1m' + f'Metadata for sensor {sensor}:'+ '\033[0m')
    display(sensor.metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f'Time series for sensor {sensor.name}')
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
    break  # for this example we stop after the first sensor
Stations in 'FMI': ['SAA111', 'SAA112', 'SAA120', 'SOD011', 'SOD012', 'SOD013', 'SOD021', 'SOD022', 'SOD023', 'SOD031', 'SOD032', 'SOD033', 'SOD071', 'SOD072', 'SOD073', 'SOD081', 'SOD082', 'SOD083', 'SOD091', 'SOD092', 'SOD093', 'SOD101', 'SOD102', 'SOD103', 'SOD130', 'SOD140', 'SODAWS']
Sensors at 'SAA111': ['CS215_air_temperature_-2.000000_-2.000000', '5TE_soil_temperature_0.200000_0.200000', '5TE_soil_temperature_0.400000_0.400000', '5TE_soil_moisture_0.800000_0.800000', '5TE_soil_moisture_0.400000_0.400000', '5TE_soil_moisture_0.200000_0.200000', '5TE_soil_temperature_0.050000_0.050000', '5TE_soil_moisture_0.050000_0.050000', '5TE_soil_moisture_0.100000_0.100000', '5TE_soil_temperature_0.800000_0.800000', '5TE_soil_temperature_0.100000_0.100000']
5TE_soil_moisture_0.050000_0.050000
Metadata for sensor 5TE_soil_moisture_0.050000_0.050000:
variable        key
clay_fraction   val                           4.0
                depth_from                    0.0
                depth_to                      0.3
climate_KG      val                           Dfc
climate_insitu  val                       unknown
elevation       val                         441.0
instrument      val                           5TE
                depth_from                   0.05
                depth_to                     0.05
latitude        val                      68.33019
lc_2000         val                           110
lc_2005         val                           110
lc_2010         val                           110
lc_insitu       val                       unknown
longitude       val                      27.55062
network         val                           FMI
organic_carbon  val                          2.18
                depth_from                    0.0
                depth_to                      0.3
sand_fraction   val                          87.0
                depth_from                    0.0
                depth_to                      0.3
saturation      val                          0.49
                depth_from                    0.0
                depth_to                      0.3
silt_fraction   val                           9.0
                depth_from                    0.0
                depth_to                      0.3
station         val                        SAA111
timerange_from  val           2012-10-14 00:00:00
timerange_to    val           2020-12-12 23:00:00
variable        val                 soil_moisture
                depth_from                   0.05
                depth_to                     0.05
Name: data, dtype: object
../_images/examples_interface_47_5.png

Selecting by variable and other metadata (1)

In this example we iterate over all sensors for the network ‘RMSN’ and filter out those that measure precipitation within an ESA CCI Landcover pixel that is marked as ‘Cropland, rainfed’ (10) or ‘Grassland’ (130). Available land cover classes to choose are:

[19]:
ismn_data.print_landcover_dict()
CCI Landcover Classification
----------------------------
Cropland, rainfed: 10
Cropland, rainfed / Herbaceous cover: 11
Cropland, rainfed / Tree or shrub cover: 12
Cropland, irrigated or post-flooding: 20
Mosaic cropland (>50%) / natural vegetation (tree, shrub, herbaceous: 30
Mosaic natural vegetation (tree, shrub, herbaceous cover) (>50%) / cropland (<50%): 40
Tree cover, broadleaved, evergreen, Closed to open (>15%): 50
Tree cover, broadleaved, deciduous, Closed to open (>15%): 60
Tree cover, broadleaved, deciduous, Closed (>40%): 61
Tree cover, broadleaved, deciduous, Open (15-40%): 62
Tree cover, needleleaved, evergreen, closed to open (>15%): 70
Tree cover, needleleaved, evergreen, closed (>40%): 71
Tree cover, needleleaved, evergreen, open (15-40%): 72
Tree cover, needleleaved, deciduous, closed to open (>15%): 80
Tree cover, needleleaved, deciduous, closed (>40%): 81
Tree cover, needleleaved, deciduous, open (15-40%): 82
Tree cover, mixed leaf type (broadleaved and needleleaved): 90
Mosaic tree and shrub (>50%) / herbaceous cover (<50%): 100
Mosaic herbaceous cover (>50%) / tree and shrub (<50%): 110
Shrubland: 120
Shrubland / Evergreen Shrubland: 121
Shrubland / Deciduous Shrubland: 122
Grassland: 130
Lichens and mosses: 140
Sparse vegetation (tree, shrub, herbaceous cover) (<15%): 150
Sparse vegetation (tree, shrub, herbaceous cover) (<15%) / Sparse shrub (<15%): 152
Sparse vegetation (tree, shrub, herbaceous cover) (<15%) / Sparse herbaceous cover (<15%): 153
Tree cover, flooded, fresh or brakish water: 160
Tree cover, flooded, saline water: 170
Shrub or herbaceous cover, flooded, fresh/saline/brakish water: 180
Urban areas: 190
Bare areas: 200
Consolidated bare areas: 201
Unconsolidated bare areas: 202
Water: 210
Permanent snow and ice: 220
[20]:
for station, sensor in ismn_data['RSMN'].iter_sensors(variable='precipitation',
                                                      filter_meta_dict={'lc_2010': [10, 130]}):
    display(station)
    display(sensor)

    data = sensor.read_data()
    metadata = sensor.metadata

    print('\033[1m' + f'Metadata for sensor {sensor}:' + '\033[0m')
    display(metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f'Time series for sensor {sensor.name}')
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Precipitation [mm]")
    break # for this example we stop after the first sensor
Sensors at 'Barlad': ['HMP155-Vaisala_air_temperature_-2.000000_-2.000000', '5TM_soil_temperature_0.000000_0.050000', '5TM_soil_moisture_0.000000_0.050000', 'QMR102_precipitation_-1.400000_-1.400000', 'HMP155_air_temperature_-2.000000_-2.000000']
QMR102_precipitation_-1.400000_-1.400000
Metadata for sensor QMR102_precipitation_-1.400000_-1.400000:
variable        key
climate_KG      val                           Dfb
climate_insitu  val                       unknown
elevation       val                         172.0
instrument      val                        QMR102
                depth_from                   -1.4
                depth_to                     -1.4
latitude        val                      46.23313
lc_2000         val                            10
lc_2005         val                            10
lc_2010         val                            10
lc_insitu       val                       unknown
longitude       val                      27.64438
network         val                          RSMN
station         val                        Barlad
timerange_from  val           2014-06-06 07:00:00
timerange_to    val           2020-12-12 23:00:00
variable        val                 precipitation
                depth_from                   -1.4
                depth_to                     -1.4
Name: data, dtype: object
../_images/examples_interface_50_4.png

Selecting by variable, depth and metadata (2)

In this example we iterate over all sensors in the collection and filter those that measure ‘soil_moisture’ between 0 and 10 cm within an ESA CCI Landcover pixel that is marked as ‘Cropland, rainfed’ (10) or ‘Grassland’ (130), and has one of the following climate classes assigned: ‘Csc’, ‘Cfa’, ‘Dfc’. In addition, we set all those soil moisture values that are not flagged as ‘good’ (G) to ‘NaN’.

[21]:
ismn_data.print_climate_dict()
KOEPPEN GEIGER Climate Classification
-------------------------------------
Af  : Tropical Rainforest
Am  : Tropical Monsoon
As  : Tropical Savanna Dry
Aw  : Tropical Savanna Wet
BWk : Arid Desert Cold
BWh : Arid Desert Hot
BWn : Arid Desert With Frequent Fog
BSk : Arid Steppe Cold
BSh : Arid Steppe Hot
BSn : Arid Steppe With Frequent Fog
Csa : Temperate Dry Hot Summer
Csb : Temperate Dry Warm Summer
Csc : Temperate Dry Cold Summer
Cwa : Temperate Dry Winter, Hot Summer
Cwb : Temperate Dry Winter, Warm Summer
Cwc : Temperate Dry Winter, Cold Summer
Cfa : Temperate Without Dry Season, Hot Summer
Cfb : Temperate Without Dry Season, Warm Summer
Cfc : Temperate Without Dry Season, Cold Summer
Dsa : Cold Dry Summer, Hot Summer
Dsb : Cold Dry Summer, Warm Summer
Dsc : Cold Dry Summer, Cold Summer
Dsd : Cold Dry Summer, Very Cold Winter
Dwa : Cold Dry Winter, Hot Summer
Dwb : Cold Dry Winter, Warm Summer
Dwc : Cold Dry Winter, Cold Summer
Dwd : Cold Dry Winter, Very Cold Winter
Dfa : Cold Dry Without Dry Season, Hot Summer
Dfb : Cold Dry Without Dry Season, Warm Summer
Dfc : Cold Dry Without Dry Season, Cold Summer
Dfd : Cold Dry Without Dry Season, Very Cold Winter
ET  : Polar Tundra
EF  : Polar Eternal Winter
W   : Water
[22]:
from ismn.meta import Depth
for network, station, sensor in ismn_data.collection \
    .iter_sensors(variable='soil_moisture',
                  depth=Depth(0.,0.05),
                  filter_meta_dict={'lc_2010': [10, 130],
                                    'climate_KG':['Csc', 'Cfa', 'Dfc']}):

    display(network)
    display(station)
    display(sensor)

    data = sensor.read_data()
    data.loc[data['soil_moisture_flag'] != 'G', 'soil_moisture'] = np.nan
    metadata = sensor.metadata

    print('\033[1m' + f'Metadata for sensor {sensor}:'+ '\033[0m')
    display(metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f"G-flagged SM for '{sensor.name}' at station '{station.name}' in network '{network.name}''")
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
    break # for this example we stop after the first sensor
Stations in 'RSMN': ['Adamclisi', 'Alexandria', 'Bacles', 'Banloc', 'Barlad', 'Calarasi', 'ChisineuCris', 'Corugea', 'Cotnari', 'Darabani', 'Dej', 'Dumbraveni', 'Iasi', 'Oradea', 'RosioriideVede', 'SannicolauMare', 'SatuMare', 'Slatina', 'Slobozia', 'Tecuci']
Sensors at 'Dumbraveni': ['5TM_soil_moisture_0.000000_0.050000', '5TM_soil_temperature_0.000000_0.050000', 'QMR102_precipitation_-1.400000_-1.400000', 'HMP45DX_air_temperature_-2.000000_-2.000000']
5TM_soil_moisture_0.000000_0.050000
Metadata for sensor 5TM_soil_moisture_0.000000_0.050000:
variable        key
clay_fraction   val                          19.0
                depth_from                    0.0
                depth_to                      0.3
climate_KG      val                           Dfc
climate_insitu  val                       unknown
elevation       val                         318.0
instrument      val                           5TM
                depth_from                    0.0
                depth_to                     0.05
latitude        val                      46.22786
lc_2000         val                            10
lc_2005         val                            10
lc_2010         val                            10
lc_insitu       val                       unknown
longitude       val                      24.59161
network         val                          RSMN
organic_carbon  val                          0.99
                depth_from                    0.0
                depth_to                      0.3
sand_fraction   val                          37.0
                depth_from                    0.0
                depth_to                      0.3
saturation      val                          0.44
                depth_from                    0.0
                depth_to                      0.3
silt_fraction   val                          44.0
                depth_from                    0.0
                depth_to                      0.3
station         val                    Dumbraveni
timerange_from  val           2014-06-11 12:00:00
timerange_to    val           2020-12-12 23:00:00
variable        val                 soil_moisture
                depth_from                    0.0
                depth_to                     0.05
Name: data, dtype: object
../_images/examples_interface_53_5.png