IPPN 2024
Open Source Pipeline for UAS and satellite based High Throughput Phenotyping Applications - Part 1
This notebook is designed for workshop presented at the International Plant Phenotyping Network (IPPN) conference on October 7, 2024. Click the Open in Colab button above to run this notebook interactively in the cloud. For Part 2 of the workshop, please click here.
- Registration: https://www.plant-phenotyping.org/index.php?index=935
- Notebook: https://samgeo.gishub.org/workshops/IPPN_2024
- Earth Engine: https://earthengine.google.com
- Geemap: https://geemap.org
- Leafmap: https://leafmap.org
- Samgeo: https://samgeo.gishub.org
- Data to Science (D2S): https://ps2.d2s.org
- D2S Python API: https://py.d2s.org
Introduction¶
Recent advances in sensor technology have revolutionized the assessment of crop health by providing fine spatial and high temporal resolutions at affordable costs. As plant scientists gain access to increasingly larger volumes of Unmanned Aerial Systems (UAS) and satellite High Throughput Phenotyping (HTP) data, there is a growing need to extract biologically informative and quantitative phenotypic information from the vast amount of freely available geospatial data. However, the lack of specialized software packages tailored for processing such data makes it challenging to develop transdisciplinary research collaboration around these data. This workshop aims to bridge the gap between big data and agricultural research scientists by providing training on an open-source online platform for managing big UAS HTP data known as Data to Science. Additionally, attendees will be introduced to powerful Python packages, namely leafmap and Leafmap, designed for the seamless integration and analysis of UAS and satellite images in various agricultural applications. By participating in this workshop, attendees will acquire the skills necessary to efficiently search, visualize, and analyze geospatial data within a Jupyter environment, even with minimal coding experience. The workshop provides a hands-on learning experience through practical examples and interactive exercises, enabling participants to enhance their proficiency and gain valuable insights into leveraging geospatial data for agricultural research purposes.
Agenda¶
The main topics to be covered in this workshop include:
- Create interactive maps using leafmap
- Visualize drone imagery from D2S
- Segment drone imagery using samgeo
- Calculate zonal statistics from drone imagery
- Visualize Earth Engine data
- Create timelapse animations
Environment setup¶
Change Colab dark theme¶
Currently, ipywidgets does not work well with Colab dark theme. Some of the leafmap widgets may not display properly in Colab dark theme.It is recommended that you change Colab to the light theme.
Change runtime type to GPU¶
To speed up the processing, you can change the Colab runtime type to GPU. Go to the "Runtime" menu, select "Change runtime type", and choose "T4 GPU" from the "Hardware accelerator" dropdown menu.
Install packages¶
Uncomment the following code to install the required packages.
# %pip install -U "leafmap[raster]" segment-geospatial d2spy
# %pip install numpy==1.26.4
Import libraries¶
Import the necessary libraries for this workshop.
import leafmap
Creating interactive maps¶
Let's create an interactive map using the ipyleaflet plotting backend. The leafmap.Map class inherits the ipyleaflet.Map class. Therefore, you can use the same syntax to create an interactive map as you would with ipyleaflet.Map.
m = leafmap.Map()
To display it in a Jupyter notebook, simply ask for the object representation:
m
To customize the map, you can specify various keyword arguments, such as center ([lat, lon]), zoom, width, and height. The default width is 100%, which takes up the entire cell width of the Jupyter notebook. The height argument accepts a number or a string. If a number is provided, it represents the height of the map in pixels. If a string is provided, the string must be in the format of a number followed by px, e.g., 600px.
m = leafmap.Map(center=[40, -100], zoom=4, height="600px")
m
Adding basemaps¶
There are several ways to add basemaps to a map. You can specify the basemap to use in the basemap keyword argument when creating the map. Alternatively, you can add basemap layers to the map using the add_basemap method. leafmap has hundreds of built-in basemaps available that can be easily added to the map with only one line of code.
Create a map by specifying the basemap to use as follows. For example, the Esri.WorldImagery basemap represents the Esri world imagery basemap.
m = leafmap.Map(basemap="Esri.WorldImagery")
m
You can add as many basemaps as you like to the map. For example, the following code adds the OpenTopoMap basemap to the map above:
m.add_basemap("OpenTopoMap")
You can also add an XYZ tile layer to the map.
basemap_url = "https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}"
m.add_tile_layer(basemap_url, name="Hybrid", attribution="Google")
You can also change basemaps interactively using the basemap GUI.
m = leafmap.Map()
m.add_basemap_gui()
m
Visualizing Drone Imagery from D2S¶
The Data to Science (D2S) platform (https://ps2.d2s.org) hosts a large collection of drone imagery that can be accessed through the D2S API (https://py.d2s.org). To visualize drone imagery from D2S, you need to sign up for a free account on the D2S platform and obtain an API key.
Login to D2S¶
Login and connect to your D2S workspace in one go using the d2spy.
from d2spy.workspace import Workspace
# Replace with URL to a D2S instance
d2s_url = "https://ps2.d2s.org"
# Login and connect to workspace with your email address
workspace = Workspace.connect(d2s_url, "workshop@d2s.org")
# Check for API key
api_key = workspace.api_key
if not api_key:
print(
"No API key. Please request one from the D2S profile page and re-run this cell."
)
import os
from datetime import date
os.environ["D2S_API_KEY"] = api_key
os.environ["TITILER_ENDPOINT"] = "https://tt.d2s.org"
Choose a project to work with¶
The Workspace get_projects method will retrieve a collection of the projects your account can currently access on the D2S instance.
# Get list of all your projects
projects = workspace.get_projects()
for project in projects:
print(project)
The projects variable is a ProjectCollection. The collection can be filtered by either the project descriptions or titles using the methods filter_by_title or filter_by_name.
# Example of creating new collection of only projects with the keyword "Citrus Orchard" in the title
filtered_projects = projects.filter_by_title("Citrus Orchard")
print(filtered_projects)
Now you can choose a specific project to work with. In this case, the filtered projects returned only one project, so we will use that project.
project = filtered_projects[0]
Get the project boundary¶
get_project_boundary method of the Project class will retrieve a GeoJSON object of the project boundary.
# Get project boundary as Python dictionary in GeoJSON structure
project_boundary = project.get_project_boundary()
project_boundary
Get project flights¶
The Project get_flights method will retrieve a list of flights associated with the project.
# Get list of all flights for a project
flights = project.get_flights()
# Print first flight object (if one exists)
for flight in flights:
print(flight)
Filter flights by date¶
The flights variable is a FlightCollection. The collection can be filtered by the acquisition date using the method filter_by_date. This method will return all flights with an acquisition date between the provided start and end dates.
# Example of creating new collection of only flights from June 2022
filtered_flights = flights.filter_by_date(
start_date=date(2022, 6, 1), end_date=date(2022, 7, 1)
)
for flight in filtered_flights:
print(flight)
Now, we can choose a flight from the filtered flight. Let's choose the flight on June 9, 2022.
flight = filtered_flights[0]
flight
Get data products¶
The Flight get_data_products method will retrieve a list of data products associated with the flight.
# Get list of data products from a flight
data_products = flight.get_data_products()
for data_product in data_products:
print(data_product)
The data_products variable is a DataProductCollection. The collection can be filtered by data type using the method filter_by_data_type. This method will return all data products that match the requested data type.
# Example of creating new collection of data products with the "ortho" data type
ortho_data_products = data_products.filter_by_data_type("ortho")
print(ortho_data_products)
Visualize ortho imagery¶
Now we can grab the ortho URL to display it using leafmap.
m = leafmap.Map()
m.add_basemap("HYBRID", show=False)
ortho_data = ortho_data_products[0]
ortho_url_202206 = ortho_data.url
ortho_url_202206 = leafmap.d2s_tile(ortho_url_202206)
m.add_cog_layer(ortho_url_202206, name="Ortho Imagery 202206")
m
Visualize DSM¶
Similarly, you can visualize the Digital Surface Model (DSM) from D2S using the code below.
# Example of creating new collection of data products with the "dsm" data type
dsm_data_products = data_products.filter_by_data_type("dsm")
print(dsm_data_products)
dsm_data = dsm_data_products[0]
dsm_url_202206 = dsm_data.url
dsm_url_202206 = leafmap.d2s_tile(dsm_url_202206)
m.add_cog_layer(dsm_url_202206, colormap_name="terrain", name="DSM 202206")
leafmap.cog_stats(dsm_url_202206)
Add a colorbar to the map.
m.add_colormap(cmap="terrain", vmin=3, vmax=33, label="Elevation (m)")
m
Visualize CHM¶
Similarly, you can visualize the Canopy Height Model (CHM) from D2S using the code below.
# Example of creating new collection of data products with the "chm" data type
chm_data_products = data_products.filter_by_data_type("chm")
print(chm_data_products)
chm_data = chm_data_products[0]
chm_url_202206 = chm_data.url
chm_url_202206 = leafmap.d2s_tile(chm_url_202206)
m.add_cog_layer(chm_url_202206, colormap_name="jet", name="CHM 202206")
leafmap.cog_stats(chm_url_202206)
m.add_colormap(cmap="jet", vmin=0, vmax=13, label="Elevation (m)")
m
Add the project boundary to the map.
m.add_geojson(project_boundary, layer_name="Project Boundary")
Add tree boundaries to the map.
map_layers = project.get_map_layers()
tree_boundaries = map_layers[0]
m.add_geojson(tree_boundaries, layer_name="Tree Boundaries")
Get another flight¶
Retrieve the Ortho data product for the December 2022 flight.
filtered_flights = flights.filter_by_date(
start_date=date(2022, 12, 1), end_date=date(2022, 12, 31)
)
for flight in filtered_flights:
print(flight)
flight_202212 = filtered_flights[0]
data_products = flight_202212.get_data_products()
ortho_data_products = data_products.filter_by_data_type("ortho")
ortho_data = ortho_data_products[0]
ortho_url_202212 = ortho_data.url
ortho_url_202212 = leafmap.d2s_tile(ortho_url_202212)
Compare two ortho images¶
Create a split map for comparing the 2022 and 2024 ortho images.
from ipyleaflet import TileLayer
m = leafmap.Map()
left_layer = TileLayer(
url=leafmap.cog_tile(ortho_url_202206), max_zoom=30, name="2022-06 Ortho"
)
right_layer = TileLayer(
url=leafmap.cog_tile(ortho_url_202212), max_zoom=30, name="2022-12 Ortho"
)
m.split_map(left_layer, right_layer, left_label="2022-06", right_label="2022-12")
m.set_center(-97.955281, 26.165595, 18)
m
Download data from D2S¶
Read the ortho image from D2S as a DataArray.
import rioxarray as rxr
data = rxr.open_rasterio(ortho_url_202206)
data
m = leafmap.Map()
m.add_cog_layer(ortho_url_202206, name="Ortho Imagery 202206")
m
Draw an area of interest (AOI) on the map. If an AOI is not provided, a default AOI will be used.
if m.user_roi is not None:
bbox = m.user_roi_bounds()
else:
bbox = [-97.956252, 26.165315, -97.954992, 26.165883]
geojson = leafmap.bbox_to_geojson(bbox)
gdf = leafmap.geojson_to_gdf(geojson)
m.add_gdf(gdf, layer_name="AOI", info_mode=None)
crs = data.rio.crs.to_string()
print(crs)
gdf = gdf.to_crs(crs)
print(gdf.crs)
Resample the ortho imagery from 1 cm to 10 cm resolution.
resampled_data = data.rio.reproject(crs, resolution=(0.1, 0.1))
resampled_data.shape
Clip the ortho image to the AOI.
clipped_data = resampled_data.rio.clip(gdf.geometry, gdf.crs)
clipped_data.shape
Save the clipped ortho image to a GeoTIFF file.
image = "ortho_image_202206.tif"
clipped_data.sel(band=[1, 2, 3]).rio.to_raster(image)
Read the CHM dataset from D2S as a DataArray.
chm_data = rxr.open_rasterio(chm_url_202206)
chm_data
resampled_chm_data = chm_data.rio.reproject_match(resampled_data)
resampled_chm_data.shape
clipped_chm_data = resampled_chm_data.rio.clip(gdf.geometry, gdf.crs)
clipped_chm_data.shape
chm_image = "chm_202206.tif"
clipped_chm_data.sel(band=[1]).rio.to_raster(chm_image)
Visualize the clipped ortho image.
m = leafmap.Map()
m.add_raster(image, layer_name="Ortho Image 202206")
m.add_geojson(tree_boundaries, layer_name="Tree Boundaries")
m
from samgeo import SamGeo, SamGeo2
sam2 = SamGeo2(model_id="sam2-hiera-large", automatic=True)
Automatic mask generation¶
sam2.generate(image)
sam2.save_masks(output="masks.tif")
sam2.show_masks(cmap="binary_r")
sam2.show_masks(cmap="jet")
Show the object annotations (objects with random color) on the map.
sam2.show_anns(axis="off", alpha=0.7, output="annotations.tif")
Compare images with a slider.
leafmap.image_comparison(
image,
"annotations.tif",
label1="Drone Imagery",
label2="Image Segmentation",
)
Add segmentation result to the map.
m.add_raster("masks.tif", colormap="jet", layer_name="Masks", nodata=0, opacity=0.7)
m
Convert the object masks to vector format, such as GeoPackage, Shapefile, or GeoJSON.
sam2.raster_to_vector("masks.tif", "masks.gpkg")
m.add_vector("masks.gpkg", layer_name="Objects")
Automatic mask generation options¶
There are several tunable parameters in automatic mask generation that control how densely points are sampled and what the thresholds are for removing low quality or duplicate masks. Additionally, generation can be automatically run on crops of the image to get improved performance on smaller objects, and post-processing can remove stray pixels and holes. Here is an example configuration that samples more masks:
sam2 = SamGeo2(
model_id="sam2-hiera-large",
apply_postprocessing=False,
points_per_side=64,
points_per_batch=128,
pred_iou_thresh=0.7,
stability_score_thresh=0.92,
stability_score_offset=0.7,
crop_n_layers=1,
box_nms_thresh=0.7,
crop_n_points_downscale_factor=2,
min_mask_region_area=25,
use_m2m=True,
)
sam2.generate(image, output="masks2.tif")
sam2.show_masks(cmap="jet")
sam2.show_anns(axis="off", alpha=0.7, output="annotations2.tif")
leafmap.image_comparison(
image,
"annotations2.tif",
label1="Image",
label2="Image Segmentation",
)
Remove small objects.
da, gdf = sam2.region_groups(
"masks2.tif",
connectivity=1,
min_size=10,
max_size=2000,
intensity_image="chm_202206.tif",
out_image="objects.tif",
out_csv="objects.csv",
out_vector="objects.gpkg",
)
Using box prompts¶
gdf = leafmap.geojson_to_gdf(tree_boundaries)
gdf.head()
geojson = "tree_boundaries.geojson"
gdf.to_file(geojson)
m = leafmap.Map()
m.add_raster(image, layer_name="image")
style = {
"color": "#ffff00",
"weight": 2,
"fillColor": "#7c4185",
"fillOpacity": 0,
}
m.add_vector(geojson, style=style, zoom_to_layer=True, layer_name="Bounding boxes")
m
sam = SamGeo(
model_type="vit_h",
automatic=False,
sam_kwargs=None,
)
sam.set_image(image)
sam.predict(
boxes=geojson, point_crs="EPSG:4326", output="tree_masks.tif", dtype="uint16"
)
m.add_raster(
"tree_masks.tif", cmap="jet", nodata=0, opacity=0.5, layer_name="Tree masks"
)
m