AI for Good Workshop 2025¶
Join us for the AI for Good Workshop 2025, part of the UN's AI for Good workshop series! This workshop will take place online on February 5, 2025, from 9:00 AM to 10:30 AM EST. It is free and open to the public. Please register using this link: Mastering Remote Sensing Image Segmentation with AI: A Hands-On Workshop with the Segment Anything Model.
Overview¶
Built upon Meta’s Segment Anything Model (SAM), the SAMGeo Python package brings advanced segmentation capabilities to geospatial data. This hands-on workshop is tailored for geospatial enthusiasts, researchers, and professionals eager to unlock the potential of GeoAI in their projects.
Participants will explore how to leverage SAMGeo for accurate and efficient image segmentation of satellite and aerial imagery. The workshop includes step-by-step demonstrations and practical exercises covering:
- Introduction to SAM and SAMGeo: Learn the architecture and functionality of SAM and its transformative applications in geospatial analysis.
- Data Preparation: Prepare geospatial datasets with multi-spectral channels for segmentation tasks.
- Hands-On with SAMGeo: Leverage SAMGeo to segment geospatial features (e.g., buildings, trees, water bodies) using prompts such as point coordinates, bounding boxes, and text.
- Postprocessing Techniques: Calculate geometric properties of segmented features, filter results, and extract meaningful insights.
- Data Visualization: Visualize object masks and segmented features in standard geospatial formats for analysis and reporting.
By the end of the workshop, participants will gain practical experience applying SAMGeo to real-world geospatial challenges and leave equipped with new tools to elevate their geospatial data workflows.
Target audience¶
This workshop is ideal for geospatial data scientists, remote sensing analysts, researchers, and anyone interested in applying AI to geospatial data.
Prerequisites¶
- A Google Colab account
- Basic understanding of Python programming and geospatial data concepts is recommended
Recording¶
The recording of the workshop is available on YouTube: https://www.youtube.com/watch?v=pTlIIr-ZS4s
Introduction to SAM and SAMGeo¶
The Segment Anything Model (SAM), introduced by Meta AI in April 2023, represents a significant advancement in computer vision, particularly in the field of image segmentation. Designed as a promptable segmentation model, SAM is capable of generating accurate segmentation masks based on various prompts, such as points, bounding boxes, or textual inputs. A notable feature of SAM is its zero-shot transfer ability, allowing it to adapt to new image distributions and tasks without additional training. This adaptability is largely attributed to its training on the extensive SA-1B dataset, which comprises over 1 billion segmentation masks across 11 million images.
Building upon the foundation laid by SAM, Meta AI released Segment Anything Model 2 (SAM 2) in August 2024. SAM 2 extends the capabilities of its predecessor by introducing real-time, promptable object segmentation in both images and videos. This unified model achieves state-of-the-art performance, enabling fast and precise selection of any object in any visual context. Key enhancements in SAM 2 include improved accuracy and processing speed, advanced prompting techniques, and the ability to handle video segmentation tasks seamlessly.
Building on the success of SAM and SAM 2, the SAMGeo Python package extends these capabilities to geospatial data. SAMGeo empowers users to perform advanced image segmentation tasks on satellite and aerial imagery, enabling the extraction of valuable insights from geospatial datasets. By leveraging the power of SAMGeo, geospatial professionals can streamline their workflows, enhance data analysis, and unlock new possibilities in remote sensing applications.
For more information on SAM and SAMGeo, please check out the slides from here: https://bit.ly/aiforgood-samgeo.
Environment setup¶
Install the required packages locally¶
If you are running this notebook locally, you can install the required packages using the following commands:
conda create -n sam python=3.12
conda activate sam
conda install -c conda-forge mamba
mamba install -c conda-forge segment-geospatial groundingdino-py gdal
Use Google Colab¶
If you are using Google Colab, make sure you use GPU runtime for this notebook. Go to Runtime -> Change runtime type and select GPU as the hardware accelerator. Then you can run the following cell to install the required packages.
%pip install segment-geospatial groundingdino-py
import leafmap
from samgeo import SamGeo2, regularize
Create an interactive map¶
Create an interactive map using leafmap.
m = leafmap.Map(center=[47.653287, -117.588070], zoom=16, height="800px")
m.add_basemap("Satellite")
m
Download a sample image¶
Pan and zoom the map to select the area of interest. Use the draw tools to draw a polygon or rectangle on the map. If no geometry is drawn, the default bounding box will be used.
if m.user_roi is not None:
bbox = m.user_roi_bounds()
else:
bbox = [-117.6029, 47.65, -117.5936, 47.6563]
Download the image within the selected region using map_tiles_to_geotiff() function.
image = "satellite.tif"
leafmap.map_tiles_to_geotiff(
output=image, bbox=bbox, zoom=18, source="Satellite", overwrite=True
)
Important note: The code is provided for educational purposes only. By using the information and code provided, users acknowledge that they are using the APIs and models at their own risk and agree to comply with any applicable laws and regulations. Users who intend to download a large number of image tiles from any basemap are advised to contact the basemap provider to obtain permission before doing so. Unauthorized use of the basemap or any of its components may be a violation of copyright laws or other applicable laws and regulations.
Alternatively, you can also use your own image. Uncomment and run the following cell to use your own image.
# image = '/path/to/your/own/image.tif'
Display the downloaded image on the map.
m.layers[-1].visible = False
m.add_raster(image, layer_name="Image")
m
Initialize SAM class¶
Set automatic=False to enable the SAM2ImagePredictor.
sam = SamGeo2(
model_id="sam2-hiera-large",
automatic=False,
)
Specify the image to segment.
sam.set_image(image)
Segment the image¶
Use the predict_by_points() method to segment the image with specified point coordinates. You can use the draw tools to add place markers on the map. If no point is added, the default sample points will be used.
if m.user_rois is not None:
point_coords_batch = m.user_rois
else:
point_coords_batch = [
[-117.599896, 47.655345],
[-117.59992, 47.655167],
[-117.599928, 47.654974],
[-117.599518, 47.655337],
]
Segment the objects using the point prompts and save the output masks.
sam.predict_by_points(
point_coords_batch=point_coords_batch,
point_crs="EPSG:4326",
output="mask.tif",
dtype="uint8",
)
Display the result¶
Add the segmented image to the map.
m.add_raster("mask.tif", cmap="viridis", nodata=0, opacity=0.7, layer_name="Mask")
m
Use an existing vector dataset as point prompts¶
Alternatively, you can specify a file path or HTTP URL to a vector dataset containing point geometries.
geojson = "https://github.com/opengeos/datasets/releases/download/places/wa_building_centroids.geojson"
Display the vector dataawr on the map.
m = leafmap.Map()
m.add_raster(image, layer_name="Image")
m.add_circle_markers_from_xy(
geojson, radius=3, color="red", fill_color="yellow", fill_opacity=0.8
)
m
Segment image with a vector dataset¶
Segment the image using the specified file path to the vector dataset.
output_masks = "building_masks.tif"
sam.predict_by_points(
point_coords_batch=geojson,
point_crs="EPSG:4326",
output=output_masks,
dtype="uint8",
multimask_output=False,
)
Display the segmented masks on the map.
m.add_raster(
output_masks, cmap="jet", nodata=0, opacity=0.7, layer_name="Building masks"
)
m
Clean up the result¶
Remove small objects from the segmented masks, fill holes, and compute geometric properties.
out_vector = "building_vector.geojson"
out_image = "buildings.tif"
array, gdf = sam.region_groups(
output_masks, min_size=200, out_vector=out_vector, out_image=out_image
)
gdf.head()
Regularize building footprints¶
Regularize the building footprints using the regularize() method.
output_regularized = "building_regularized.geojson"
regularize(out_vector, output_regularized)
Display the regularized building footprints on the map.
m = leafmap.Map()
m.add_raster(image, layer_name="Image")
style = {
"color": "#ffff00",
"weight": 2,
"fillColor": "#7c4185",
"fillOpacity": 0,
}
m.add_raster(out_image, cmap="tab20", opacity=0.7, nodata=0, layer_name="Buildings")
m.add_vector(
output_regularized, style=style, layer_name="Building regularized", info_mode=None
)
m
Interactive segmentation¶
Place markers on the map to segment the objects interactively.
sam.show_map()
import leafmap
from samgeo import SamGeo2, raster_to_vector, regularize
Create an interactive map¶
Create an interactive map using leafmap.
m = leafmap.Map(center=[47.653287, -117.588070], zoom=16, height="800px")
m.add_basemap("Satellite")
m
Download a sample image¶
Pan and zoom the map to select the area of interest. Use the draw tools to draw a polygon or rectangle on the map. If no geometry is drawn, the default bounding box will be used.
if m.user_roi is not None:
bbox = m.user_roi_bounds()
else:
bbox = [-117.6029, 47.65, -117.5936, 47.6563]
Download the image within the selected region using map_tiles_to_geotiff() function.
image = "satellite.tif"
leafmap.map_tiles_to_geotiff(
output=image, bbox=bbox, zoom=18, source="Satellite", overwrite=True
)
You can also use your own image. Uncomment and run the following cell to use your own image.
# image = '/path/to/your/own/image.tif'
Display the downloaded image on the map.
m.layers[-1].visible = False
m.add_raster(image, layer_name="Image")
m
Initialize SAM class¶
Set automatic=False to enable the SAM2ImagePredictor.
sam = SamGeo2(
model_id="sam2-hiera-large",
automatic=False,
)
Specify the image to segment.
sam.set_image(image)
Display the map. Use the drawing tools to draw some rectangles around the features you want to extract, such as trees, buildings.
m
Create bounding boxes¶
If no rectangles are drawn, the default bounding boxes will be used as follows:
if m.user_rois is not None:
boxes = m.user_rois
else:
boxes = [
[-117.5995, 47.6518, -117.5988, 47.652],
[-117.5987, 47.6518, -117.5979, 47.652],
]
Segment the image¶
Use the predict() method to segment the image with specified bounding boxes. The boxes parameter accepts a list of bounding box coordinates in the format of [[left, bottom, right, top], [left, bottom, right, top], ...], a GeoJSON dictionary, or a file path to a GeoJSON file.
sam.predict(boxes=boxes, point_crs="EPSG:4326", output="mask.tif", dtype="uint8")
Display the result¶
Add the segmented image to the map.
m.add_raster("mask.tif", cmap="viridis", nodata=0, layer_name="Mask")
m
Use an existing vector dataset as box prompts¶
Alternatively, you can specify a file path to a vector dataset. Let's download a sample vector dataset from GitHub.
url = "https://github.com/opengeos/datasets/releases/download/samgeo/building_bboxes.geojson"
geojson = "building_bboxes.geojson"
leafmap.download_file(url, geojson)
Display the vector dataset on the map.
m = leafmap.Map()
m.add_raster(image, layer_name="Image")
style = {
"color": "#ffff00",
"weight": 2,
"fillColor": "#7c4185",
"fillOpacity": 0,
}
m.add_vector(geojson, style=style, zoom_to_layer=True, layer_name="Bboxes")
m
Segment image with box prompts¶
Segment the image using the specified file path to the vector mask.
output_masks = "building_masks.tif"
sam.predict(
boxes=geojson,
point_crs="EPSG:4326",
output=output_masks,
dtype="uint8",
multimask_output=False,
)
Display the segmented masks on the map.
m.add_raster(
output_masks, cmap="jet", nodata=0, opacity=0.5, layer_name="Building masks"
)
m
Convert raster to vector¶
Convert the segmented masks to a vector format.
output_vector = "building_vector.geojson"
raster_to_vector(output_masks, output_vector)
Regularize building footprints¶
Regularize the building footprints using the regularize() method.
output_regularized = "building_regularized.geojson"
regularize(output_vector, output_regularized)
Display the regularized building footprints on the map.
m.add_vector(
output_regularized, style=style, layer_name="Building regularized", info_mode=None
)
import leafmap
from samgeo.text_sam import LangSAM
Create an interactive map¶
Create an interactive map using leafmap.
m = leafmap.Map(center=[-22.17615, -51.253043], zoom=18, height="800px")
m.add_basemap("SATELLITE")
m
Download a sample image¶
Pan and zoom the map to select the area of interest. Use the draw tools to draw a polygon or rectangle on the map.
bbox = m.user_roi_bounds()
if bbox is None:
bbox = [-51.2565, -22.1777, -51.2512, -22.175]
Download the image within the selected region using map_tiles_to_geotiff() function.
image = "Image.tif"
leafmap.map_tiles_to_geotiff(
output=image, bbox=bbox, zoom=19, source="Satellite", overwrite=True
)
You can also use your own image. Uncomment and run the following cell to use your own image.
# image = '/path/to/your/own/image.tif'
Display the downloaded image on the map.
m.layers[-1].visible = False
m.add_raster(image, layer_name="Image")
m
Initialize LangSAM class¶
The initialization of the LangSAM class might take a few minutes. The initialization downloads the model weights and sets up the model for inference.
sam = LangSAM(model_type="sam2-hiera-large")
Specify text prompts¶
Specify the text prompt to segment the objects in the image. The text prompt can be a single word or a phrase that describes the object you want to segment.
text_prompt = "tree"
Segment the image¶
Part of the model prediction includes setting appropriate thresholds for object detection and text association with the detected objects. These threshold values range from 0 to 1 and are set while calling the predict method of the LangSAM class.
box_threshold: This value is used for object detection in the image. A higher value makes the model more selective, identifying only the most confident object instances, leading to fewer overall detections. A lower value, conversely, makes the model more tolerant, leading to increased detections, including potentially less confident ones.
text_threshold: This value is used to associate the detected objects with the provided text prompt. A higher value requires a stronger association between the object and the text prompt, leading to more precise but potentially fewer associations. A lower value allows for looser associations, which could increase the number of associations but also introduce less precise matches.
Remember to test different threshold values on your specific data. The optimal threshold can vary depending on the quality and nature of your images, as well as the specificity of your text prompts. Make sure to choose a balance that suits your requirements, whether that's precision or recall.
sam.predict(image, text_prompt, box_threshold=0.24, text_threshold=0.24)
Visualize the results¶
Show the result with bounding boxes on the map.
sam.show_anns(
cmap="Greens",
box_color="red",
title="Automatic Segmentation of Trees",
blend=True,
)
Show the result without bounding boxes on the map.
sam.show_anns(
cmap="Greens",
add_boxes=False,
alpha=0.5,
title="Automatic Segmentation of Trees",
)
Show the result as a grayscale image.
sam.show_anns(
cmap="Greys_r",
add_boxes=False,
alpha=1,
title="Automatic Segmentation of Trees",
blend=False,
output="trees.tif",
)
Convert the result to a vector format.
da, gdf = sam.region_groups(
image="trees.tif",
min_size=100,
out_csv="objects.csv",
out_image="objects.tif",
out_vector="objects.gpkg",
)
Show the results on the interactive map.
m.add_raster("objects.tif", layer_name="Trees", palette="Greens", opacity=0.5, nodata=0)
style = {
"color": "#3388ff",
"weight": 2,
"fillColor": "#7c4185",
"fillOpacity": 0.5,
}
m.add_vector("objects.gpkg", layer_name="Vector", style=style)
m
Interactive segmentation¶
sam.show_map()
import leafmap
from samgeo import SamGeo2
Download sample data¶
For now, SamGeo2 supports remote sensing data in the form of RGB images, 8-bit integer. Make sure all images are in the same width and height. Let's download a sample timeseries dataset from GitHub.
url = "https://github.com/opengeos/datasets/releases/download/raster/landsat_ts.zip"
leafmap.download_file(url)
Initialize the model¶
Initialize the SamGeo2 class with the model ID and set the video parameter to True.
predictor = SamGeo2(
model_id="sam2-hiera-large",
video=True,
)
Specify the input data¶
Point to the directory containing the images or the video file.
video_path = "landsat_ts"
predictor.set_video(video_path)
Specify the input prompts¶
The prompts can be points and boxes. The points are represented as a list of tuples, where each tuple contains the x and y coordinates of the point. The boxes are represented as a list of tuples, where each tuple contains the x, y, width, and height of the box.
predictor.show_images()
prompts = {
1: {
"points": [[1582, 933], [1287, 905], [1473, 998]],
"labels": [1, 1, 1],
"frame_idx": 0,
},
}
predictor.show_prompts(prompts, frame_idx=0)
Althernatively, prompts can be provided in lon/lat coordinates. The model will automatically convert the lon/lat coordinates to pixel coordinates when the point_crs parameter is set to the coordinate reference system of the lon/lat coordinates.
prompts = {
1: {
"points": [[-74.3713, -8.5218], [-74.2973, -8.5306], [-74.3230, -8.5495]],
"labels": [1, 1, 1],
"frame_idx": 0,
},
}
predictor.show_prompts(prompts, frame_idx=0, point_crs="EPSG:4326")
Segment objects¶
Segment the objects from the video or timeseries images.
predictor.predict_video()
Save results¶
To save the results as gray-scale GeoTIFFs with the same georeference as the input images:
predictor.save_video_segments("segments")
To save the results as blended images and MP4 video:
predictor.save_video_segments_blended(
"blended", fps=5, output_video="segments_blended.mp4"
)
Preview the video.
from IPython.display import Video
Video("segments_blended.mp4", embed=True, width=600, height=400)
import leafmap
from samgeo import SamGeo2
Initialize the model¶
predictor = SamGeo2(
model_id="sam2-hiera-large",
video=True,
)
Specify the input data¶
url = "https://github.com/opengeos/datasets/releases/download/videos/cars.mp4"
video_path = url
predictor.set_video(video_path)
Specify the input prompts¶
predictor.show_images()
prompts = {
1: {
"points": [[335, 203]],
"labels": [1],
"frame_idx": 0,
},
2: {
"points": [[420, 201]],
"labels": [1],
"frame_idx": 0,
},
}
predictor.show_prompts(prompts, frame_idx=0)
Segment objects¶
predictor.predict_video(prompts)
Save results¶
predictor.save_video_segments_blended("cars", output_video="cars_blended.mp4", fps=25)
Preview the video.
from IPython.display import Video
Video("cars_blended.mp4", embed=True, width=600, height=400)