Demonstration of the TWIGA API and TAHMO data¶

This demonstration call the TWIGA platform API in order to retrieve meta-data on TAHMO data and to download some TAHMO data results are displayed in the notebook in both tables and graphs

June 2022

In [ ]:
# load modules
import requests
import json
import datetime
import pandas as pd
import numpy as np
from io import StringIO
In [2]:
#############################
# TWIGA API settings        #
#############################

# The TWIGA Platform has a comprehensive API via which data can be retrieved
# this API is requires a valid bearer token
# this block defines the base end-point and token required

# The base API endpoint
api = 'https://hnapi.hydronet.com/api/'

# The bearer token for the TWIGA user
# this token is used to identify as a valid TWIGA user to the API
api_token = '##########'

# Using the token, generate a valid header for requests to the API
api_header = {'content-type': 'application/json', 'Authorization': 'bearer ' + api_token}
In [3]:
#############################
# Query meta-data           #
#############################


# define the selected data source of interest, i.e. the TAHMO stations
selected_datasource_code = "Tahmo.Stations.Data.Distribution.Hourly"

# json request to ask metadata of the selected datasource
request_metadata = {
     "DataSourceCodes": [selected_datasource_code]
}

# Send the request to the datasources endpoint of the API - which can be used to extract the start and end-date
metadata_datasource = requests.post(api + 'entity/datasources/get', headers=api_header, data=json.dumps(request_metadata))
metadata_datasource = metadata_datasource.json()

# Send the request to the variables endpoint of the API - to get all available variables
metadata_variables = requests.post(api + 'entity/variables/get', headers=api_header, data=json.dumps(request_metadata))
metadata_variables = metadata_variables.json()

# Send the request to the locations endpoint of the API - to get all locations
metadata_locations = requests.post(api + 'entity/locations/get', headers=api_header, data=json.dumps(request_metadata))
metadata_locations = metadata_locations.json()
In [4]:
# create data frame with all variables of the TAHMO stations
# each variable is identified with a unique Code
df_var = pd.DataFrame.from_dict(metadata_variables['Variables'])
df_var
Out[4]:
AZP Q P Radiation RH SoilMoisture SoilMoisture.10cm SoilMoisture.20cm SoilMoisture.30cm SoilMoisture.5cm SoilMoisture.50cm Temperature.Soil TMP VaporPressure Waterlevel WaterVelocity WindDirection WindGusts WindSpeed
Name Average Pressure Discharge Precipitation Radiation Relative Humidity Soil Moisture Soil Moisture Soil Moisture Soil Moisture Soil Moisture Soil Moisture Soil temperature Temperature Vapor Pressure Waterlevel Water velocity Wind Direction Wind gusts Wind Speed
Code AZP Q P Radiation RH SoilMoisture SoilMoisture.10cm SoilMoisture.20cm SoilMoisture.30cm SoilMoisture.5cm SoilMoisture.50cm Temperature.Soil TMP VaporPressure Waterlevel WaterVelocity WindDirection WindGusts WindSpeed
UnitCode None m3/s mm w/m2 % m3/m3 m3/m3 m3/m3 m3/m3 m3/m3 m3/m3 ⁰C ⁰C kPa m NAP m/s deg m/s m/s
State 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Description NaN NaN Precipitation NaN Relative Humidity 2m above ground NaN NaN NaN NaN NaN NaN NaN Temperature 2m above ground NaN NaN NaN Wind Direction (from which blowing) NaN Wind Speed at surface
In [5]:
# create data frame with all TAHMO station locations
# each location is identified with a unique Code
df_loc = pd.DataFrame.from_dict(metadata_locations['Locations'])
df_loc
Out[5]:
Tahmo.Stations#TA00001 Tahmo.Stations#TA00002 Tahmo.Stations#TA00003 Tahmo.Stations#TA00004 Tahmo.Stations#TA00005 Tahmo.Stations#TA00006 Tahmo.Stations#TA00007 Tahmo.Stations#TA00008 Tahmo.Stations#TA00009 Tahmo.Stations#TA00010 ... Tahmo.Stations#TA00755 Tahmo.Stations#TA00756 Tahmo.Stations#TA00757 Tahmo.Stations#TA00758 Tahmo.Stations#TA00759 Tahmo.Stations#TH00015 Tahmo.Stations#TA00760 Tahmo.Stations#TA00761 Tahmo.Stations#TA00762 Tahmo.Stations#TA00763
Identifier Tahmo.Stations#TA00001 Tahmo.Stations#TA00002 Tahmo.Stations#TA00003 Tahmo.Stations#TA00004 Tahmo.Stations#TA00005 Tahmo.Stations#TA00006 Tahmo.Stations#TA00007 Tahmo.Stations#TA00008 Tahmo.Stations#TA00009 Tahmo.Stations#TA00010 ... Tahmo.Stations#TA00755 Tahmo.Stations#TA00756 Tahmo.Stations#TA00757 Tahmo.Stations#TA00758 Tahmo.Stations#TA00759 Tahmo.Stations#TH00015 Tahmo.Stations#TA00760 Tahmo.Stations#TA00761 Tahmo.Stations#TA00762 Tahmo.Stations#TA00763
LocationId 23846 23847 23848 23849 23850 23851 23852 23853 23854 23855 ... 96072 96127 119873 121997 122148 122826 123419 123673 123676 183865
Name Lela Primary School Benso SHS Prestea SHTS Manso Amenfi NVTI Asankragwa SHS Nana Brentu SHTS Enchi Nana Yaa Kesse SHS Duayaw Nkwanta Sacred Heart SHS Nsoatre Adehyeman SHS Amasu Chiraa SHS ... Cascada Kwadaso Agric College Kumasi Institute of Tropical Agriculture Nhancuco Derunde Alimaua Bridge Mawenzi secondary school Agape Lutheran Junior seminary secondary school Machame girls secondary school Kazimzumbwi primary school
Code TA00001 TA00002 TA00003 TA00004 TA00005 TA00006 TA00007 TA00008 TA00009 TA00010 ... TA00755 TA00756 TA00757 TA00758 TA00759 TH00015 TA00760 TA00761 TA00762 TA00763
X 34.397992 -1.881544 -2.133929 -2.266748 -2.426395 -2.825592 -2.097477 -2.476063 -2.782209 -2.185991 ... 34.0502 -1.670523 -1.501921 34.050185 34.772193 39.242824 37.324712 37.552939 37.234377 39.001739
Y -1.123283 5.152567 5.446856 5.643226 5.807731 5.811308 7.188273 7.41298 7.224905 7.389595 ... -18.51171 6.671519 6.676117 -18.511691 -19.093018 -6.790046 -3.343375 -3.313086 -3.198653 -6.965531
Z 1393.0 89.752275 63.0 100.0 124.0464 119.3 343.4 332.0 290.5 338.7 ... 678.5 270.4 256.0 674.6 127.6 31.0 901.1 1217.1 1530.4 250.0
Tags [] [] [] [] [] [] [] [] [] [] ... [] [] [] [] [] [] [] [] [] []
ProjectionId 3 3 3 3 3 3 3 3 3 3 ... 3 3 3 3 3 3 3 3 3 3
DataSourceGroupCode Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations ... Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations Tahmo.Stations

10 rows × 674 columns

In [6]:
# Query the available time-series of Precipitation in January 2022
# this searches for all stations which have measured preciptitaion in January 2022
# start and end date are formatted as yyyyMMddHHmmss in universal time (UTC)
request_timeseriesinformations = {
     "DataSourceCodes": ["Tahmo.Stations.Data.Distribution.Hourly"],
     "VariableCodes": ["P"],
     "StartDate": "20220101000000",
     "EndDate": "20220201000000"
}
metadata_tsinformation = requests.post(api + 'entity/timeseriesinformations/get', headers=api_header, data=json.dumps(request_timeseriesinformations))
metadata_tsinformation = metadata_tsinformation.json()

# create data frame with all locations which match the search criteria
# all of these resulting locations might have data available over the given period
df_ts = pd.DataFrame.from_dict(metadata_tsinformation['TimeSeriesInformations'])
df_ts
Out[6]:
62912 62932 62958 62988 63018 63048 63078 63108 63138 63168 ... 196928 199814 199838 199958 199982 200043 212977 238161 238900 247118
TimeSeriesInformationId 62912 62932 62958 62988 63018 63048 63078 63108 63138 63168 ... 196928 199814 199838 199958 199982 200043 212977 238161 238900 247118
LocationIdentifier Tahmo.Stations#TA00001 Tahmo.Stations#TA00002 Tahmo.Stations#TA00003 Tahmo.Stations#TA00004 Tahmo.Stations#TA00005 Tahmo.Stations#TA00006 Tahmo.Stations#TA00007 Tahmo.Stations#TA00008 Tahmo.Stations#TA00009 Tahmo.Stations#TA00010 ... Tahmo.Stations#TA00749 Tahmo.Stations#TA00750 Tahmo.Stations#TA00751 Tahmo.Stations#TA00752 Tahmo.Stations#TA00753 Tahmo.Stations#TA00754 Tahmo.Stations#TA00756 Tahmo.Stations#TA00757 Tahmo.Stations#TA00758 Tahmo.Stations#TA00759
DataSourceVariableId 28858 28858 28858 28858 28858 28858 28858 28858 28858 28858 ... 28858 28858 28858 28858 28858 28858 28858 28858 28858 28858
StartDate 20170316000000 20190705000000 20180902000000 20190702000000 20200625000000 20190703000000 20190908000000 20210827000000 20190907000000 20190908000000 ... 20210507000000 20210218000000 20210526000000 20210527000000 20210529000000 20210701000000 20210526000000 20211008000000 20211018000000 20211030000000
EndDate 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 20220304071009 ... 20220304071010 20220304071010 20220304071010 20220223000000 20220304071010 20220304071010 20220304071010 20220304071010 20220304071010 20220304071010

5 rows × 576 columns

In [7]:
# Download data of 2 locations in a simple CSV format
# use the variable code and location codes
request_data_timeseries = {
    "Readers": [{
    "DataSourceCode": "Tahmo.Stations.Data.Distribution.Hourly",
        "Settings": {
            "StartDate": "20210101000000",
            "EndDate": "20210102000000",
            "VariableCodes": ["P"],
            "LocationCodes": ["TA00001", "TA00002"]
        }
    }],
    "Exporter": {
        "DataFormatCode": "hydronet.csv.simple"
    },
    "TimeZoneOffset": "+0000"
}
# send request to API
tahmo_data = requests.post(api + 'timeseries/get', headers=api_header, data=json.dumps(request_data_timeseries))

# parse the csv data into a pandas dataframe
tahmo_data_txt = StringIO(tahmo_data.text)
tahmo_df = pd.read_csv(tahmo_data_txt)

# Drop last row and column (as those are empty)
tahmo_df.drop(tahmo_df.tail(1).index, inplace = True)
tahmo_df = tahmo_df.iloc[:, :-1]
tahmo_df
Out[7]:
Date time Location code Variable code Value Quality Availability
0 2021-01-01 00:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
1 2021-01-01 01:00:00 TA00001 TAHMO.Hourly.Precipitation 0.085 1.0 1.0
2 2021-01-01 02:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
3 2021-01-01 03:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
4 2021-01-01 04:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
5 2021-01-01 05:00:00 TA00001 TAHMO.Hourly.Precipitation 0.153 1.0 1.0
6 2021-01-01 06:00:00 TA00001 TAHMO.Hourly.Precipitation 0.017 1.0 1.0
7 2021-01-01 07:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
8 2021-01-01 08:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
9 2021-01-01 09:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
10 2021-01-01 10:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
11 2021-01-01 11:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
12 2021-01-01 12:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
13 2021-01-01 13:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
14 2021-01-01 14:00:00 TA00001 TAHMO.Hourly.Precipitation 15.995 1.0 1.0
15 2021-01-01 15:00:00 TA00001 TAHMO.Hourly.Precipitation 0.102 1.0 1.0
16 2021-01-01 16:00:00 TA00001 TAHMO.Hourly.Precipitation 1.105 1.0 1.0
17 2021-01-01 17:00:00 TA00001 TAHMO.Hourly.Precipitation 0.901 1.0 1.0
18 2021-01-01 18:00:00 TA00001 TAHMO.Hourly.Precipitation 0.051 1.0 1.0
19 2021-01-01 19:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
20 2021-01-01 20:00:00 TA00001 TAHMO.Hourly.Precipitation 0.017 1.0 1.0
21 2021-01-01 21:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
22 2021-01-01 22:00:00 TA00001 TAHMO.Hourly.Precipitation 0.017 1.0 1.0
23 2021-01-01 23:00:00 TA00001 TAHMO.Hourly.Precipitation 0.425 1.0 1.0
24 2021-01-02 00:00:00 TA00001 TAHMO.Hourly.Precipitation 0.000 1.0 1.0
In [ ]: