ismn package
Submodules
ismn.base module
- class ismn.base.IsmnRoot(path)[source]
Bases:
object
Connection to the zip resp. extracted zip archive downloaded from the ismn website. This class only handles file access / requests made by the readers, lists files in path and can extract files to temp folders for safe reading.
- path
Data directory
- Type:
Path
- clean_subpath(subpath) Path | PurePosixPath [source]
Check if subpath is a valid path and adapt to archive format and os
- property cont
Get cont of object, or scan to create cont.
- find_files(subpath=None, fn_templ='*.csv')[source]
List files in archive or a subdirectory of the archive that match the passed filename pattern.
- Parameters:
- Returns:
files – Found files that match the passed template.
- Return type:
- scan(station_subdirs=True) OrderedDict [source]
Go through archive (zip or dir) and group station folders
- Parameters:
station_subdirs (bool, optional (default: True)) – Include the station dir as a subdir of network dir. If False is selected, station dir is included directly.
- Returns:
cont – Archive content, station dirs grouped by network dirs
- Return type:
OrderedDict
ismn.cli module
ismn.components module
- class ismn.components.Network(name, stations=None)[source]
Bases:
IsmnComponent
A network is described by a distinct name and can be composed of multiple ISMN stations.
- stations
Stations belonging to the network. If a string is passed, an empty Station is added.
- Type:
OrderedDict[name, Station]
- property coords: (<class 'list'>, <class 'list'>)
Get lists of lats and lons for all stations in the Network
- get_citations()[source]
Return reference(s) for this network. Users of ISMN should cite the networks they are using in a publication. This information can also be found on the ISMN website.
- Returns:
references – A list of references / citations / acknowledgements for this Network.
- Return type:
- property grid: CellGrid
Get grid for all Stations in Network
- iter_sensors(**filter_kwargs)[source]
Get all sensors in all stations in the network that comply with the passed filtering parameters.
- Parameters:
sensors (Keyword arguments are used to evaluate the) –
see –
:param
ismn.components.Sensor.eval()
:- Yields:
station (Station) – Station that contains Sensor.
sensor (Sensor) – Sensor at Station that matches to the passed filtering conditions.
- iter_stations(**filter_kwargs)[source]
Get all stations having at least one sensor observing a specific variable and/or sensing depth.
- Parameters:
stations (only) –
stations –
returned. (that have at least one matching sensor are) –
kwargs (For a description of possible filter) –
see –
:param
ismn.components.Sensor.eval()
:- Yields:
station (Station) – Stations that contain at least one sensor that matches to the passed conditions.
- to_xarray(**filter_kwargs) Dataset [source]
Collect all sensor data at this station into a xarray.DataSet object with a location and time dimension in a single chunk. The dataset will contain static (dimension sensor) and dynamic (dimensions sensor & time) variables.
- Parameters:
filter_kwargs (optional) – Filter sensors in the network to include in the dataset (variable, depth, etc.). For a description of possible filter kwargs, see
ismn.components.Sensor.eval()
- Returns:
dat – Sensor data as xarray Dataset. Sensor data are stored as Dask arrays!
- Return type:
xarray.DataSet
- class ismn.components.NetworkCollection(networks)[source]
Bases:
IsmnComponent
A NetworkCollection holds multiple networks and provides functionality to perform access to components from multiple networks. A grid is added that contains all stations to perform spatial searches.
- networks
Collection of network names and Networks
- Type:
OrderedDict
- grid
Grid that contains one point for each station in all networks.
- Type:
CellGrid
- export_citations(out_file=None)[source]
Returns the references for all networks in the collection. Optionally, they are also written to file. Information on how to correctly cite ISMN networks can be found on the ISMN website.
- Parameters:
out_file (str, optional (default: None)) – If a path is passed here, a new file will be generated with all references for the current collection.
- Returns:
references – Network names as keys and network references as values
- Return type:
OrderedDict
- export_geojson(path, network=True, station=True, sensor=False, depth=True, timerange=True, extra_props=None, filter_kwargs=None)[source]
Filter sensors in collection and create geojson file containing all features.
- Parameters:
path (str) – Path to geojson file
network (bool, optional (default: True)) – If True, network names are included in geojson file
station (bool, optional (default: True)) – If True, station names are included in geojson file
sensor (bool, optional (default: False)) – If True, sensor names are included in geojson file
depth (bool, optional (default: True)) – If True, depth_from and depth_to are included in geojson file
timerange (bool, optional (default: True)) – If True, timerange_from and timerange_to are included in geojson
extra_props (list[str], optional (default: None)) – List of extra properties from sensor metadata to include in geojson file By default only depth_from and depth_to are included e.g. [‘variable’, ‘frm_class’] etc.
filter_kwargs (dict, optional (default: None)) – Keyword arguments to filter sensors in collection before extracting metadata. see
ismn.components.Sensor.eval()
- get_nearest_station(lon, lat, max_dist=inf)[source]
Get nearest station for given longitude/latitude coordinates.
- Parameters:
- Returns:
station (Station or list[Station]) – The nearest Station(s) to the passed coordinates.
dist (float or list[float]) – Distance in meter between the passed coordinates and the actual location of the station.
- iter_sensors(**filter_kwargs) -> (<class 'ismn.components.Network'>, <class 'ismn.components.Station'>, <class 'ismn.components.Sensor'>)[source]
Iterate through Networks in the Collection and get (all/filtered) Stations and Sensors at each Station.
- class ismn.components.Sensor(instrument, variable, depth, name=None, filehandler=None, keep_loaded_data=False)[source]
Bases:
IsmnComponent
A Sensor with insitu observations.
- data
Container for data in memory (if it is being kept)
- Type:
- property data
- eval(variable=None, depth=None, filter_meta_dict=None, check_only_sensor_depth_from=False)[source]
Evaluate whether the sensor complies with the passed metadata requirements.
- Parameters:
variable (str or list[str], optional (default: None)) – Check if the variable name matches, e.g. soil_moisture. One or multiple of
ismn.const.VARIABLE_LUT
depth (Depth or list or tuple, optional (default: None)) – Check if the passed depth encloses the sensor depth. A list/tuple must contain 2 values where the first is the depth start and the second is the end. Start must be closer to 0 than end (or equal). A negative depth range is above the surface.
filter_meta_dict (dict, optional (default: None)) –
Additional metadata keys and values for which the file list is filtered e.g. {‘lc_2010’: [10, 130]} or
{‘climate_KG’: ‘Dwa’, ‘lc_2010’: [10, 130] }
to filter for a multiple landcover classes and a climate class.
check_only_sensor_depth_from (bool, optional (default: False)) – Ignores the sensors depth_to value and only checks if depth_from of the sensor is in the passed depth (e.g. for cosmic ray probes).
- Returns:
flag – Indicates whether metadata for this Sensor matches with the passed requirements.
- Return type:
- get_coverage(only_good=True, start=None, end=None, freq='1h')[source]
Estimate the temporal coverage of this sensor, i.e. the percentage of valid observations in the sensor time series.
- Returns:
only_good (bool, optional (default: True)) – Only consider values where the ISMN quality flag is ‘G’ as valid observations
start (str or datetime, optional (default: None)) – Beginning of the period in which measurements are expected. If None, the start of the time series is used.
end (str or datetime, optional (default: None)) – End of the period in which measurements are expected. If None, the start of the time series is used.
freq (str, optional (default: ‘1h’)) – Frequency at which the sensor is expected to take measurements. Most sensors in ISMN provide hourly measurements (default). If a different frequency is used, it must be on that
pd.date_range()
can interpret.
- Returns:
perc_coverage – Data coverage of the sensor at the chosen expected measurement frequency within the chosen period. 0=No data, 100=no data gaps
- Return type:
- read_data()[source]
Load data from filehandler for this Sensor by calling
ismn.filehandlers.DataFile.read_data()
.- Returns:
data – Insitu time series for this sensor, loaded from file or memory (if it was loaded and kept before).
- Return type:
- to_xarray() Dataset [source]
Convert the Sensors data to an xarray.DataSet object with a location and time dimension in a single chunk. The dataset will contain static (dimension sensor) and dynamic (dimensions sensor & time) variables.
- Returns:
dat – Sensor data as xarray dataset
- Return type:
xarray.DataSet
- class ismn.components.Station(name, lon, lat, elev)[source]
Bases:
IsmnComponent
A station is described by a distinct name and location. Multiple sensors at various depths can be part of a station.
- sensors
Collection of Sensors and their names.
- Type:
- add_sensor(instrument, variable, depth, filehandler=None, name=None, keep_loaded_data=False)[source]
Add a new Sensor to this Station.
- Parameters:
instrument (str) – Instrument name. e.g. ThetaProbe-ML2X
variable (str) – Observed variable. e.g. soil_moisture
depth (Depth) – Sensing depth. e.g. Depth(0, 0.1)
filehandler (DataFile, optional (default: None)) – File handler object that allows access to observation data and sensor metadata via its read_data() function (default: None).
name (str or int, optional (default: None)) – A name or id for the sensor. If None is passed, one is generated automatically from other properties.
keep_loaded_data (bool, optional (default: False)) – Keep data for the sensor in memory once it is loaded. This makes subsequent reading of the same data faster but can fill up memory if stations / networks are loaded.
- get_min_max_obs_timestamp(variable='soil moisture', min_depth=None, max_depth=None)[source]
Goes through the sensors associated with this station and checks the metadata to get and approximate time coverage of the station. This is just an overview. If holes have to be detected the complete file must be read.
- Parameters:
variable (str, optional (default: 'soil_moisture')) – name of the variable, only sensors measuring that variable are used.
min_depth (float, optional (default: None)) – depth_from of variable has to be >= min_depth in order to be included.
max_depth (float, optional (default: None)) – depth_to of variable has to be <= max_depth in order to be included.
- Returns:
start_date (datetime.datetime) – Earliest date observed by any sensor at the station after filtering for the passed requirements.
end_date (datetime.datetime) – Latest date observed by any sensor at the station after filtering for the passed requirements.
- get_sensors(variable, depth_from, depth_to)[source]
get the sensors at which the variable was measured at the given depth
- Parameters:
- Returns:
sensors – array of sensors found for the given combination of variable and depths
- Return type:
- get_variables()[source]
Get variables measured by all sensors at station.
- Returns:
variables – List of variables that are observed.
- Return type:
- iter_sensors(**filter_kwargs)[source]
Iterates over all sensors in this station and yields those that comply with the passed filter settings (or all).
- Parameters:
stations (Keyword arguments are used to check all sensors at all) –
:param : :param only stations that have at least one matching sensor are returned.: :param For a description of possible filter kwargs: :param see: :param
ismn.components.Sensor.eval()
:- Yields:
sensors (Sensor) – (Filtered) Sensors at the Station.
- to_xarray(**filter_kwargs) Dataset [source]
Collect all sensor data at this station into a xarray.DataSet object with a location and time dimension in a single chunk. The dataset will contain static (dimension sensor) and dynamic (dimensions sensor & time) variables.
- Parameters:
filter_kwargs (optional) – Filter sensors at the station to include in the dataset (variable, depth, etc.). For a description of possible filter kwargs, see
ismn.components.Sensor.eval()
- Returns:
dat – Sensor data as xarray dataset
- Return type:
xarray.DataSet
ismn.const module
- exception ismn.const.DepthError[source]
Bases:
ValueError
ismn.custom module
Module that handles custom, additional information that can be assigned to the ismn data by the user. Sometimes it is convenient to have additional information on at a sensor, station, or the surroundings, which is not directly provided by the ISMN, assigned to the ISMN metadata. This module contains a base class and implementations for certain metadata formats that the ISMN_Interface class can then use to add additional values to python_metadata during metadata collection.
- class ismn.custom.CustomMetaReader[source]
Bases:
object
Template class for a reader to assign additional metadata to ismn sensors. The read_metadata function must be implemented and return the metadata to add to a sensors either as MetaData object (which allows assigning depth information to metadata) or a dictionary (which will be converted later on to MetaData without depth information assigned)
Metadata readers take the existing metadata from a sensor, and based on the information there they can extract other metadata. Can return Metadata objects or dicts (which are converted in ISMN package to metadata)
Objects based on CustomMetaReaders can be passed to
ismn.interface.Ismn_Interface
- abstract read_metadata(meta) MetaData | dict [source]
Read metadata from additional sources (that are not provided directly by the ISMN). Uses available information for an ismn sensor for selecting the correct data (usually lat / lon of a sensor).
- Parameters:
meta (MetaData) –
Existing Metadata for a sensor, as collected from csv and .stm files. Contains for each sensor at least:
- Shared by all sensors at a station:
longitude, latitude, elevation, network, station, lc_2010, lc_insitu, climate_KG, climate_insitu
- Sensor specific:
instrument (with depth_from and depth_to) variable, clay_fraction, sand_fraction, organic_carbon, silt_fraction (and depths of dataset layer they were extracted from)
- Returns:
ancillary_meta – Metadata collected by this reader. Dict also works but will be converted to MetaData without depths assigned later on. Metadata is then assigned to the sensor
- Return type:
- class ismn.custom.CustomSensorMetadataCsv(station_meta_csv, fill_values=None, **kwargs)[source]
Bases:
CustomStationMetadataCsv
Allows passing metadata for ISMN sensors as a csv file. E.g. if the sensor specific variables provided by the ISMN are not enough. In this case that the metadata must be stored in a csv file with the following structure:
network;station;instrument;variable;depth_from;depth_to;<var1>;<var1>_depth_from;<var1>_depth_to;<var2> …
where <var1> etc. are the names of the custom metadata variables that are transferred into the python metadata where <var1>_depth_from etc. are the
- read_metadata(meta: MetaData)[source]
Match passed metadata entries to the csv file to find common sensors for which the csv metadata is then added.
- Parameters:
meta (MetaData) – Metadata that the csv values are added to for sensors where the network, station, instrument, and instrument depths match.
- Returns:
meta – Additional depth-dependent metadata at the location
- Return type:
Metadata
- class ismn.custom.CustomStationMetadataCsv(station_meta_csv, fill_values=None, **kwargs)[source]
Bases:
CustomMetaReader
Allows passing (static) metadata for ISMN stations as a csv file. E.g. if the station specific variables provided by the ISMN are not enough. In this case that the metadata must be stored in a csv file with the following structure:
network;station;<var1>;<var1>_depth_from;<var1>_depth_to;<var2>;…
where network and station refer to existing names in the metadata.
where <var1> etc. are the names of the custom metadata variables that are
transferred into the python metadata - where <var1>_depth_from and <var1>_depth_to etc are the depths that are assigned to the metadata (if columns exist)
ismn.filecollection module
- class ismn.filecollection.IsmnFileCollection(root, filelist, temp_root='/tmp')[source]
Bases:
object
The IsmnFileCollection class contains a list of file handlers to access data in the given data directory. The file list can be loaded from a previously stored csv file, or built by iterating over all files in the data root. This class also contains function to load filehandlers for certain networks only.
- filelist
A collection of filehandlers and network names
- Type:
- temp_root
Temporary root dir.
- Type:
Path
- classmethod build_from_scratch(data_root, parallel=True, log_path=None, temp_root='/tmp', custom_meta_readers=None)[source]
- Parameters:
data_root (IsmnRoot or str or Path) – Root path of ISMN files or path to metadata pkl file. i.e. path to the downloaded zip file or the extracted zip directory (faster) or a file list that contains these infos already.
parallel (bool, optional (default: True)) – Speed up metadata collecting with multiple processes.
log_path (str or Path, optional (default: None)) – Path where the log file is created. If None is set, no log file will be written.
temp_root (str or Path, (default: gettempdir())) – Temporary folder where extracted data is copied during reading from zip archive.
custom_meta_readers (tuple, optional (default: None)) – Custom metadata readers
- classmethod from_metadata_csv(data_root, meta_csv_file, network=None, temp_root='/tmp')[source]
Load a previously created and stored filelist from
ismn.filecollection.IsmnFileCollection.to_metadata_csv()
:param data_root: Path where the ismn data is stored, can also be a zip file :type data_root: IsmnRoot or str or Path :param meta_csv_file: Csv file where the metadata is stored. :type meta_csv_file: str or Path :param network: List of networks that are considered.Filehandlers for other networks are set to None.
- Parameters:
temp_root (str or Path, optional (default: gettempdir())) – Temporary folder where extracted data is copied during reading from zip archive.
- classmethod from_metadata_df(data_root, metadata_df, temp_root='/tmp')[source]
Load a previously created and stored filelist from
ismn.filecollection.IsmnFileCollection.to_metadata_csv()
:param data_root: Path where the ismn data is stored, can also be a zip file :type data_root: IsmnRoot or str or Path :param metadata_df: Metadata frame :type metadata_df: pd.DataFrame :param temp_root: Temporary folder where extracted data is copied during reading fromzip archive.
- get_filehandler(idx)[source]
Get the nth filehandler in a list of all filehandlers for all networks. e.g. if there are 2 networks, with 3 filehandlers/sensors each, idx=4 will return the first filehandler of the second network.
- iter_filehandlers(networks=None)[source]
Iterator over files for networks
- Parameters:
networks (list, optional (default: None)) – Name of networks to get files for, or None to use all networks.
- Yields:
file (DataFile) – Filehandler with metadata
- to_metadata_csv(meta_csv_file)[source]
Write filehandle metadata from filelist to metdata csv that contains ALL metadata / variables of the filehander. Can be read back in as filelist with filehandlers using
ismn.filecollection.IsmnFileCollection.from_metadata_csv()
.- Parameters:
meta_csv_file (Path or str, optional (default: None)) – Directory where the csv file with the correct name is crated
ismn.filehandlers module
- class ismn.filehandlers.DataFile(root, file_path, load_metadata=True, temp_root='/tmp', *args, **kwargs)[source]
Bases:
IsmnFile
IsmnFile class represents a single ISMN data file. This represents only .stm data files not metadata csv files.
- See :class:`ismn.filehandlers.IsmnFile`
- get_elements_from_file(delim='_', only_basename_elements=False)[source]
Read first line of file and split filename. Information is used to collect metadata information for all ISMN formats.
- Parameters:
- Returns:
headr (list[str] or None) – First line of file split into list, None if only_filename is True
secnd (list[str] or None) – Second line of file split into list, None if only_filename is True
last (list[str] or None) – Last non empty line elements, None if only_filename is True
file_basename_elements (list[str], None if only_filename is True) – File basename without path split by ‘delim’
- get_metadata_ceop_sep(elements=None)[source]
Get metadata in the file format called CEOP in separate files.
- Parameters:
elements (dict, optional (default: None)) – Previously loaded elements can be passed here to avoid reading the file again.
- Returns:
metadata (MetaData) – Metadata information.
depth (Depth) – Sensor Depth, generated from file name
- get_metadata_header_values(elements=None)[source]
Get metadata file in the format called Header Values.
- Parameters:
elements (dict, optional (default: None)) – Previously loaded elements can be passed here to avoid reading the file again.
- Returns:
metadata (MetaData) – Metadata information.
depth (Depth) – Sensor Depth, generated from file name
- read_data() DataFrame [source]
Read data in file. Load file if necessary.
- Returns:
data – File content.
- Return type:
pd.DataFrame
- read_metadata(best_meta_for_sensor=True) MetaData [source]
Read metadata from file name and first line of file.
- Parameters:
best_meta_for_sensor (bool, optional (default: True)) – Compare the sensor depth to metadata that is available in multiple depth layers (e.g. static metadata variables). Find the variable for which the depth matches best with the sensor depth.
- class ismn.filehandlers.IsmnFile(root, file_path, temp_root='/tmp', verify_filepath=True, verify_temp_root=True)[source]
Bases:
object
General base class for data and static metadata files (station csv file) in ismn archive.
- file_path
File subpath in root archive
- Type:
Path
- temp_root
Temporary directory
- Type:
Path
- check_metadata(variable=None, allowed_depth=None, filter_meta_dict=None, check_only_sensor_depth_from=False) bool [source]
Evaluate whether the file complies with the passed metadata requirements
- Parameters:
variable (str or list[str], optional (default: None)) – Name of the required variable(s) measured, e.g. soil_moisture
allowed_depth (Depth, optional (default: None)) – Depth range that is allowed, depth in metadata must be within this range.
filter_meta_dict (dict, optional (default: None)) – Additional metadata keys and values for which the file list is filtered e.g. {‘station’: ‘stationname’} to filter for a station name.
check_only_sensor_depth_from (bool, optional (default: False)) – Ignores the sensors depth_to value and only checks if depth_from of the sensor is in the passed depth (e.g. for cosmic ray probes).
- Returns:
valid – Whether the metadata complies with the passed conditions or not.
- Return type:
ismn.interface module
- class ismn.interface.ISMN_Interface(data_path, meta_path=None, network=None, parallel=False, keep_loaded_data=False, temp_root='/tmp', custom_meta_reader=None, force_metadata_collection=False)[source]
Bases:
object
Class provides interface to ISMN data downloaded from the ISMN website https://ismn.earth. Upon initialization it collects metadata from all files in path_to_data and saves metadata information in a csv file into the folder python_metadata in meta_path (or data_path if no meta_path is defined). First initialization can take some time if all ISMN data is present in data_path and will start multiple processes.
- Parameters:
data_path (str or Path) – Path to ISMN data to read, either to a zip archive or to the extracted directory that contains the network folders. Download data from https://ismn.earth after registration.
meta_path (str or Path) – Path where the metadata csv file(s) is / are stored. The actual filename is defined by the name of data_path and will be generated automatically if it does not yet exist.
network (str or list, optional (default: None)) – Name(s) of network(s) to load. Other data in the data_path will be ignored. By default or if None is passed, all networks are activated. If an empty list is passed no networks are activated.
parallel (bool, optional (default: False)) – Activate parallel processes to speed up metadata generation. All available CPUs will be used.
keep_loaded_data (bool, optional (default: False)) – Keep data for a file in memory once it is loaded. This makes subsequent calls of data faster (if e.g. a station is accessed multiple times) but can fill up memory if multiple networks are loaded.
custom_meta_reader (tuple, optional (default: None)) – Additional readers to collect station/sensor metadata from external sources e.g. csv files. See
ismn.custom.CustomMetaReader
.force_metadata_collection (bool, optional (default: False)) – If true, will run metadata collection and replace any existing metadata that would otherwise be re-used.
- Raises:
ISMNError – if given, network was not found
ISMN_Interface.data_path
- climate
All Climate classes and their descriptions.
- Type:
- collection
Contains all loaded networks with stations and sensors.
- Type:
- metadata
Metadata for active networks, with idx that could also be passed to
ismn.interface.read_metadata()
- Type:
- landcover
All Landcover classes and their descriptions.
- Type:
- Properties
- ----------
- networks
Access Networks container from collection directly.
- Type:
OrderedDict
- grid
Grid from collection that contains all station lats and lons
- Type:
pygeogrids.grid.BasicGrid
- activate_network(network=None, meta_path: str | None = None, temp_root: str = '/tmp')[source]
Load (file) collection for specific file ids.
- find_nearest_station(lon, lat, return_distance=False, max_dist=inf)[source]
Finds the nearest station to passed coordinates available in downloaded data.
- Parameters:
- Returns:
station (ismn.components.Station) – Nearest station object that was found in within the selected distance
distance (float, optional) – distance to station in meters, measured in cartesian coordinates and not on a great circle. Should be OK for small distances
- get_climate_types(variable: str = 'soil_moisture', min_depth: float = 0, max_depth: float = 10, climate: str = 'climate_KG') dict [source]
- get_dataset_ids(variable, min_depth=0, max_depth=0.1, filter_meta_dict=None, check_only_sensor_depth_from=False, groupby=None)[source]
Yield all sensors for a specific network and/or station and/or variable and/or depth. The id is defined by the position of the filehandler in the filelist.
- Parameters:
variable (str or list[str] or None) – Variable(s) to filer out, None to allow all variables.
min_depth (float, optional (default: 0)) – Min depth of sensors to search
max_depth (float, optional (default: 0.1)) – Max depth of sensors to search
filter_meta_dict (dict, optional (default: None)) – Additional metadata keys and values for which the file list is filtered e.g. {‘lc_2010’: 10} to filter for a landcover class. if there are multiple conditions, ALL have to be fulfilled. e.g. {‘lc_2010’: 10’, ‘climate_KG’: ‘Dfc’})
check_only_sensor_depth_from (bool, optional (default: False)) – Ignores the sensors depth_to value and only checks if depth_from of the sensor is in the passed depth (e.g. for cosmic ray probes).
groupby (str, optional (default: None)) – A metadata field name that is used to group sensors, e.g. network
- get_landcover_types(variable: str = 'soil_moisture', min_depth: float = 0, max_depth: float = 10, landcover: str = 'lc_2010') dict [source]
- get_min_max_obs_timestamps(variable='soil_moisture', min_depth=-inf, max_depth=inf, filter_meta_dict=None)[source]
Filter the active file list and return the min/max time stamp from ALL time series that match the passed criteria. This time period does NOT apply to all time series in the collection but is the OVERALL earliest and latest timestamp found.
- Parameters:
variable (str, optional (default: 'soil_moisture')) – One of those in
ismn.const.VARIABLE_LUT
or returned byismn.interface.ISMN_Interface.get_variables()
: ‘soil_moisture’, ‘soil_temperature’, ‘soil_suction’, ‘precipitation’, ‘air_temperature’, ‘field_capacity’, ‘permanent_wilting_point’, ‘plant_available_water’, ‘potential_plant_available_water’, ‘saturation’, ‘silt_fraction’, ‘snow_depth’, ‘sand_fraction’, ‘clay_fraction’, ‘organic_carbon’, ‘snow_water_equivalent’, ‘surface_temperature’, ‘surface_temperature_quality_flag_original’min_depth (float, optional (default: -np.inf)) – Only sensors in this depth are considered.
max_depth (float, optional (default: np.inf)) – Only sensors in this depth are considered.
filter_meta_dict (dict, optional (default: None)) – Additional metadata keys and values for which the file list is filtered e.g. {‘lc_2010’: 10} to filter for a landcover class. if there are multiple conditions, ALL have to be fulfilled. e.g. {‘lc_2010’: 10’, ‘climate_KG’: ‘Dfc’})
- Returns:
start_date (datetime.datetime) – Earliest time stamp found in all sensors that fulfill the passed requirements.
end_date (datetime.datetime) – Latest time stamp found in all sensors that fulfill the passed requirements.
- get_static_var_vals(variable='soil_moisture', min_depth=-inf, max_depth=inf, static_var_name='lc_2010') dict [source]
Get unique meta values for the selected static variable in the active networks.
- Parameters:
variable (str, optional (default: 'soil_moisture')) – One of those in
ismn.const.VARIABLE_LUT
or returned byismn.interface.ISMN_Interface.get_variables()
: ‘soil_moisture’, ‘soil_temperature’, ‘soil_suction’, ‘precipitation’, ‘air_temperature’, ‘field_capacity’, ‘permanent_wilting_point’, ‘plant_available_water’, ‘potential_plant_available_water’, ‘saturation’, ‘silt_fraction’, ‘snow_depth’, ‘sand_fraction’, ‘clay_fraction’, ‘organic_carbon’, ‘snow_water_equivalent’, ‘surface_temperature’, ‘surface_temperature_quality_flag_original’min_depth (float, optional (default: -np.inf)) – Only sensors in this depth are considered.
max_depth (float, optional (default: np.inf)) – Only sensors in this depth are considered.
static_var_name (str, optional (default: 'lc_2010')) – One of: ‘lc_2000’, ‘lc_2005’, ‘lc_2010’, ‘lc_insitu’, ‘climate_KG’, ‘climate_insitu’
- Returns:
vals – Unique values found in static meta and their meanings.
- Return type:
- property grid
- network_for_station(stationname, name_only=True)[source]
Find networks that contain a station of the passed name.
- Parameters:
- Returns:
network_names – Network that contains a station of that name, or None if no such network exists. Prints are warning and uses the FIRST found network name if there are multiple stations with the same name in different networks.
- Return type:
- property networks
- plot_station_locations(variable=None, min_depth=-inf, max_depth=inf, extent=None, stats_text=True, check_only_sensor_depth_from=False, markersize=12.5, markeroutline=True, borders=True, legend=True, text_scalefactor=1, dpi=300, filename=None, ax=None)[source]
Plots available stations on a world map in robinson projection.
- Parameters:
variable (str or list[str], optional (default: None)) – Show only stations that measure this/these variable(s), e.g. soil_moisture If None is passed, no filtering for variable is performed.
min_depth (float, optional (default: -np.inf)) – Minimum depth, only stations that have a valid sensor measuring the passed variable (if one is selected) in this depth range are included.
max_depth (float, optional (default: -np.inf)) – See description of min_depth. This is the bottom threshold for the allowed depth.
extent (list, optional (default: None)) – [lon min, lon max, lat min, lat max] Extent of the map that is plotted. If None is passed, a global map is plotted.
stats_text (bool, optianal (default: False)) – Include text of net/stat/sens counts in plot.
check_only_sensor_depth_from (bool, optional (default: False)) – Ignores the sensors depth_to value and only checks if depth_from of the sensor is in the passed depth_range (e.g. for cosmic ray probes).
markersize (int or float, optional (default: 12.5)) – Size of the marker, might depend on the amount of stations you plot.
markeroutline (bool, optional (default: True)) – If True, a black outline is drawn around the markers.
borders (bool, optional (default: True)) – If True, country borders are drawn.
legend (bool, optional (default: True)) – If True, a legend is drawn.
text_scalefactor (float, optional (default: 1)) – Scale factor that is applied to header and legend.
dpi (float, optional (default: 300)) – Only applies when figure is saved to file. Resolution of the output figure.
filename (str or Path, optional (default: None)) – Filename where image is stored. If None is passed, no file is created.
ax (plt.axes) – Axes object that can be used by cartopy (projection assigned).
- Returns:
fig (matplotlib.Figure) – created figure instance. If axes was given this will be None.
ax (matplitlib.Axes) – used axes instance, can be added to another figure for example.
count (dict) – Number of valid sensors and stations that contain at least one valid sensor and networks that contain at least one valid station.
- print_climate_dict() None [source]
print all classes provided by the Koeppen-Geiger climate Classification
- print_landcover_dict() None [source]
print all classes provided by the CCI Landcover Classification
- read_metadata(idx, format='pandas')[source]
Read only metadata by id as pd.DataFrame.
- Parameters:
idx (int or list) – id of sensor to read, best one of those returned from
ismn.interface.get_dataset_ids()
or one inISMN_Interface.metadata
.format (str, optional (default: 'pandas')) –
This only affects the return value when a SINGLE idx is passed. If multiple indices or None is passed, a DataFrame is returned.
pandas : return metadata as dataframe (Default)
dict : return metadata as dict (only for single idx)
obj : return metadata as MetaData object (only for single idx)
- Returns:
metadata – Metadata for the passed index.
- Return type:
- read_ts(idx, return_meta=False)[source]
Read a time series directly by the filehandler id.
- Parameters:
idx (int or list) – id(s) of filehandler to read, best one of those returned by
ismn.interface.ISMN_Interface.get_dataset_ids()
return_meta (bool, optional (default: False)) – Also return the metadata for this sensor (as a second return value)
- Returns:
timeseries (pd.DataFrame) – Observation time series, if multiple indices were passed, this contains a multiindex as columns with the idx in the first level and the variables for the idx in the second level.
metadata (pd.Series or pd.DataFrame, optional) – All available metadata for that sensor. Only returned when return_meta=False. If multiple indices were passed, this is a DataFrame with the index as columns, otherwise a Series.
- stations_that_measure(variable, **filter_kwargs)[source]
Goes through all stations and returns those that measure the specified variable
- Parameters:
variable (str) –
- variable name, one of:
soil_moisture
soil_temperature
soil_suction
precipitation
air_temperature
field_capacity
permanent_wilting_point
plant_available_water
potential_plant_available_water
saturation
silt_fraction
snow_depth
sand_fraction
clay_fraction
organic_carbon
snow_water_equivalent
surface_temperature
surface_temperature_quality_flag_original
filter_kwargs – Parameters are used to check all sensors at all stations, only stations that have at least one matching sensor are returned. For a description of possible filter kwargs, see
ismn.components.Sensor.eval()
- Yields:
ISMN_station (Station)
- subset_from_ids(ids) ISMN_Interface [source]
Create a new instance of an ISMN_Interface, but only built from ISMN data of the passed ids (from self.metadata, resp. from self.get_dataset_ids).
- Parameters:
ids (list) – List of ISMN Sensors IDs. Either from the index values of
ISMN_Interface.metadata_df
, or returned from functionISMN_Interface.get_dataset_ids()
- Returns:
subset – Another Interface, but only to the data of the selected ids
- Return type:
ismn.meta module
- class ismn.meta.Depth(start, end)[source]
Bases:
object
A class representing a depth range. For depth range start and end:
0: surface >0: Below surface <0: Above surface
- enclosed(other)[source]
Test if other Depth encloses this Depth. Reverse of
ismn.meta.Depth.encloses()
.
- encloses(other)[source]
Test if this Depth encloses other Depth. Reverse of
ismn.meta.Depth.enclosed()
.
- overlap(other, return_perc=False)[source]
Check if two depths overlap, (if the start of one depth is the same as the end of the other, they would also overlap), e.g. Depth(0, 0.1) and Depth(0.1, 0.2) do overlap.
- Parameters:
other (Depth) – Other Depth
return_perc (bool, optional (Default: False)) – Returns how much the depths overlap. See func:
ismn.meta.Depth.perc_overlap()
- Returns:
overlap (bool) – True if Depths overlap
perc_overlap (float, optional) – Normalised overlap.
- class ismn.meta.MetaData(vars: List[MetaVar] | None = None)[source]
Bases:
object
MetaData contains multiple MetaVars as a list (there can be multiple vars with the same name, e.g. for different depths)
- best_meta_for_depth(depth)[source]
For meta variables that have a depth assigned, find the ones that match best (see
ismn.depth.Depth.perc_overlap()
) to the passed depth.- Parameters:
depth (Depth) – Reference depth, e.g. the depth of a sensor.
- Returns:
best_vars – A dict of variable names and a single variable for each name that was found to match best to the passed depth. Any variables that have a depth assigned which does not overlap with the passed depth are excluded here!
- Return type:
- merge(other, inplace=False, exclude_empty=True)[source]
Merge two or more metadata sets, i.e. take all variables from other(s) that are not in this metadata, and add them.
- Parameters:
other (MetaData or list[MetaData]) – Other MetaData Collection or a list of MetaData, e.g. from multiple sensors.
inplace (bool, optional (default: False)) – Replace self.metadata with the merged meteadata, if False then the merged metadata is returned
exclude_empty (bool, optional (default: True)) – Variables where the value is NaN are ignored during merging.
- Returns:
merged_meta – The merged metadata (if inplace is False)
- Return type:
MetaData or None
- to_dict()[source]
Convert metadata to dictionary.
- Returns:
meta – Variable name as key, value and depth as values
- Return type: