87 actinia
Cloud-based geoprocessing with Actinia
# %pip install -U leafmap
The cloud based geoprocessing platform actinia is able to ingest and analyse large volumes of geodata in the cloud.
For the following actinia example we use the actinia-python-client (source code) to establish the connection to an actinia instance. First install the actinia-python-client (for latest version, see actinia-python-client releases).
import os
import leafmap
%pip install actinia_python_client
The results of actinia ephemeral processing are available via object storage as GeoTIFF/COG or GeoPackage files.
Add a helper function for "pretty printing" of actinia results:
from json import dumps as json_dumps
def print_dict(input_dict, text=None):
if text:
print(text)
if "region" in input_dict:
input_dict["region"] = input_dict["region"].__dict__
print(json_dumps(input_dict, sort_keys=True, indent=4))
def print_dict_keys(input_dict, text=None):
if text:
print(text)
print(", ".join(input_dict.keys()))
Connect to the default actinia server which is defined in the actinia-python-client, currently https://actinia.mundialis.de.
# connect to the actinia server
from actinia import Actinia
# connect to default actinia server (https://actinia.mundialis.de)
actinia_mundialis = Actinia()
# retrieve metadata about actinia server and related software versions
version = actinia_mundialis.get_version()
print_dict(version, "Version is:")
Set the authentication settings of the actinia demo user to gain access to the actinia server functionality.
actinia_user = "demouser"
actinia_password = "gu3st!pa55w0rd"
# we use the default actinia server
actinia_mundialis.set_authentication(actinia_user, actinia_password)
print("Connected to actinia server.")
Obtain the list of locations and retrieve the metadata of a selected location.
# obtain the list of projects (called "locations") which are accessible to current user
locations = actinia_mundialis.get_locations()
print_dict_keys(locations, "Locations: ")
Retrieve the metadata of a selected location (this shows the respective projection information, spatial extent, resolution, etc.) to get an idea how the output looks like.
print_dict(actinia_mundialis.locations["nc_spm_08"].get_info(), "Location info:")
At this point the connection to the selected actinia server is properly established.
Reading the online data resource into the actinia server
Next we demonstrate the data processing of a raster map available online in actinia, here a sample DEM GeoTIFF file.
# define raster elevation map name
raster_layer_name = "srtm90"
# cache file locally
out_dir = os.getcwd()
dem_file = os.path.join(out_dir, f"{raster_layer_name}.tif")
# dem_url = (
# "https://drive.google.com/file/d/1vRkAWQYsLWCi6vcTMk8vLxoXMFbdMFn8/view?usp=sharing"
# )
dem_url = f"https://github.com/giswqs/data/raw/main/raster/{raster_layer_name}.tif"
# leafmap.download_file(dem_url, dem_file, unzip=False, overwrite=True)
Prepare actinia location and mapset, i.e. generate a subproject for data processing.
# request list of all locations
locations = actinia_mundialis.get_locations()
print([loc for loc in locations])
# remove leftover location from previous run
# actinia_mundialis.locations["latlong_wgs84"].delete()
#
# remove leftover mapset from previous run
locations["latlong_wgs84"].delete_mapset("elevation")
# Create a new location for the data processing in actinia
new_location = actinia_mundialis.create_location("latlong_wgs84", 4326)
print(new_location.name)
print(new_location.region)
print([loc for loc in actinia_mundialis.locations])
# request list of mapsets in selected location
mapsets = actinia_mundialis.locations["latlong_wgs84"].get_mapsets()
print_dict_keys(mapsets, "Mapsets in latlong_wgs84:")
# Create a new mapset for the data processing in actinia
mapset_name = "elevation"
locations["latlong_wgs84"].create_mapset(mapset_name)
## Optional: Upload the sample DEM data set to actinia (indeed not needed since we use `vsicurl/` below
## to directly retrieve the online dataset).
# locations["latlong_wgs84"].mapsets[mapset_name].upload_raster(raster_layer_name, dem_file)
# print_dict_keys(locations["latlong_wgs84"].mapsets[mapset_name].raster_layers, "Raster maps in new mapset:")
Ephemeral Processing with actinia¶
Ephemeral processing is used to keep computed results, including user-generated data and temporary data, only for a limited period of time (e.g. 24 hours by default in the actinia demo server). This reduces cloud storage costs.
In contrast, persistent processing refers to the persistent retention of data without a scheduled deletion time, even in the event of a power outage, resulting in corresponding storage costs. In the geo/EO context, persistent storage is used to provide, for example, basic cartography, i.e. elevation models, road networks, building footprints, etc.
Hillshading example
Here an example for an ephemeral processing job: We download and import the remotely available GeoTIFF file. Then we use r.relief to generate a hillshading map and pre-define the resolution to 10 m. The computational region is set to the input elevation map. The resulting hillshade.tif raster map is then provided as a resource for download and visualization.
pc = {
"list": [
{
"id": "importer_0",
"comment": "Import of remote data source (here: COG)",
"module": "r.import",
"inputs": [
{"param": "input", "value": f"/vsicurl/{dem_url}"},
{"param": "memory", "value": "2000"},
{"param": "extent", "value": "input"},
],
"outputs": [{"param": "output", "value": f"{raster_layer_name}"}],
},
{
"id": "r.info_1",
"comment": "Print metadata of imported raster map",
"module": "r.info",
"inputs": [{"param": "map", "value": f"{raster_layer_name}"}],
},
{
"id": "computational_region_2",
"comment": "Set computational region to imported map, and print settings",
"module": "g.region",
"inputs": [{"param": "raster", "value": f"{raster_layer_name}"}],
"stdout": {"id": "region", "format": "kv", "delimiter": "="},
"flags": "g",
},
{
"id": "create_hillshading_3",
"comment": "Compute hillshading map",
"module": "r.relief",
"inputs": [{"param": "input", "value": f"{raster_layer_name}"}],
"outputs": [{"param": "output", "value": "hillshade"}],
},
{
"id": "exporter_4",
"comment": "Export hillshading map to COG file",
"module": "exporter",
"outputs": [
{
"export": {"type": "raster", "format": "COG"},
"param": "map",
"value": "hillshade",
}
],
},
],
"version": "1",
}
print(pc)
job = actinia_mundialis.locations["latlong_wgs84"].create_processing_export_job(
pc, "hillshading"
)
job.poll_until_finished()
print(job.status)
print(job.message)
exported_raster = job.urls["resources"][0]
print(exported_raster)
It will take a moment, then the communication by actinia is shown: "Status of hillshading job is accepted: Resource accepted" continued by further communication messages.
In case an error occurs, check the process log (use [x] with x being the step number in the process chain). Examples:
# check step 0 (r.import)
print_dict(job.process_log[0])
# check step 2 (g.region)
print_dict(job.process_log[2])
# check step 3 (r.relief)
print_dict(job.process_log[3])
Inject user:password@server into exported_raster URL (i.e., the actinia resource).
url = exported_raster.replace("//", f"//{actinia_user}:{actinia_password}@")
print(url)
Visualize the hillshade map in leafmap (colorbar inspired by this notebook):
m = leafmap.Map()
m.add_basemap("OpenTopoMap")
m.add_colormap(
cmap="terrain",
label="Elevation",
width=3,
height=0.3,
orientation="horizontal",
vmin=0,
vmax=4000,
)
m.add_cog_layer(
url,
name="SRTM90 hillshaded map",
attribution='<a href="https://e4ftl01.cr.usgs.gov/MEASURES/">https://e4ftl01.cr.usgs.gov/MEASURES/</a>',
)
# show map
m
Find further leafmap (styling) tools in the upper-right toolbox of leafmap.