Segmenting objects from timeseries images with SAM 2¶

This notebook shows how to segment objects from timeseries with the Segment Anything Model 2 (SAM 2).

Make sure you use GPU runtime for this notebook. For Google Colab, go to Runtime -> Change runtime type and select GPU as the hardware accelerator.

Install dependencies¶

Uncomment and run the following cell to install the required dependencies.

In [ ]:

# %pip install -U segment-geospatial

# %pip install -U segment-geospatial

Import libraries¶

In [ ]:

import leafmap
from samgeo import SamGeo2

import leafmap from samgeo import SamGeo2

Download sample data¶

For now, SamGeo2 supports remote sensing data in the form of RGB images, 8-bit integer. Make sure all images are in the same width and height.

In [ ]:

url = "https://github.com/opengeos/datasets/releases/download/raster/landsat_ts.zip"

url = "https://github.com/opengeos/datasets/releases/download/raster/landsat_ts.zip"

In [ ]:

leafmap.download_file(url)

leafmap.download_file(url)

Initialize the model¶

In [ ]:

predictor = SamGeo2(
    model_id="sam2-hiera-large",
    video=True,
)

predictor = SamGeo2( model_id="sam2-hiera-large", video=True, )

Specify the input data¶

Point to the directory containing the images or the video file.

In [ ]:

video_path = "landsat_ts"
predictor.set_video(video_path)

video_path = "landsat_ts" predictor.set_video(video_path)

Specify the input prompts¶

The prompts can be points and boxes. The points are represented as a list of tuples, where each tuple contains the x and y coordinates of the point. The boxes are represented as a list of tuples, where each tuple contains the x, y, width, and height of the box.

In [ ]:

predictor.show_images()

predictor.show_images()

In [ ]:

prompts = {
    1: {
        "points": [[1582, 933], [1287, 905], [1473, 998]],
        "labels": [1, 1, 1],
        "frame_idx": 0,
    },
}

prompts = { 1: { "points": [[1582, 933], [1287, 905], [1473, 998]], "labels": [1, 1, 1], "frame_idx": 0, }, }

In [ ]:

predictor.show_prompts(prompts, frame_idx=0)

predictor.show_prompts(prompts, frame_idx=0)

Althernatively, prompts can be provided in lon/lat coordinates. The model will automatically convert the lon/lat coordinates to pixel coordinates when the point_crs parameter is set to the coordinate reference system of the lon/lat coordinates.

In [ ]:

prompts = {
    1: {
        "points": [[-74.3713, -8.5218], [-74.2973, -8.5306], [-74.3230, -8.5495]],
        "labels": [1, 1, 1],
        "frame_idx": 0,
    },
}
predictor.show_prompts(prompts, frame_idx=0, point_crs="EPSG:4326")

prompts = { 1: { "points": [[-74.3713, -8.5218], [-74.2973, -8.5306], [-74.3230, -8.5495]], "labels": [1, 1, 1], "frame_idx": 0, }, } predictor.show_prompts(prompts, frame_idx=0, point_crs="EPSG:4326")

Segment the objects¶

In [ ]:

predictor.predict_video()

predictor.predict_video()

Save results¶

To save the results as gray-scale GeoTIFFs with the same georeference as the input images:

In [ ]:

predictor.save_video_segments("segments")

predictor.save_video_segments("segments")

To save the results as blended images and MP4 video:

In [ ]:

predictor.save_video_segments_blended(
    "blended", fps=5, output_video="segments_blended.mp4"
)

predictor.save_video_segments_blended( "blended", fps=5, output_video="segments_blended.mp4" )

Segment the objects from a video¶

In [ ]:

predictor = SamGeo2(
    model_id="sam2-hiera-large",
    video=True,
)

predictor = SamGeo2( model_id="sam2-hiera-large", video=True, )

In [ ]:

url = "https://github.com/opengeos/datasets/releases/download/videos/cars.mp4"

url = "https://github.com/opengeos/datasets/releases/download/videos/cars.mp4"

In [ ]:

video_path = url
predictor.set_video(video_path)

video_path = url predictor.set_video(video_path)

In [ ]:

predictor.show_images()

predictor.show_images()

In [ ]:

prompts = {
    1: {
        "points": [[335, 203]],
        "labels": [1],
        "frame_idx": 0,
    },
    2: {
        "points": [[420, 201]],
        "labels": [1],
        "frame_idx": 0,
    },
}

prompts = { 1: { "points": [[335, 203]], "labels": [1], "frame_idx": 0, }, 2: { "points": [[420, 201]], "labels": [1], "frame_idx": 0, }, }

In [ ]:

predictor.show_prompts(prompts, frame_idx=0)

predictor.show_prompts(prompts, frame_idx=0)

In [ ]:

predictor.predict_video(prompts)

predictor.predict_video(prompts)

In [ ]:

predictor.save_video_segments_blended("cars", output_video="cars_blended.mp4", fps=25)

predictor.save_video_segments_blended("cars", output_video="cars_blended.mp4", fps=25)