Video Segmentation with SAM 3¶

This notebook demonstrates how to use SAM 3 for video segmentation and tracking. SAM 3 provides:

• Text prompts: Segment objects using natural language (e.g., "person", "car")
• Point prompts: Add clicks to segment and refine objects
• Object tracking: Track segmented objects across all video frames
• Time series support: Process GeoTIFF time series with georeferencing

Installation¶

SAM 3 requires CUDA-capable GPU. Install with:

In [ ]:

# %pip install "segment-geospatial[samgeo3]"

# %pip install "segment-geospatial[samgeo3]"

Import Libraries¶

In [ ]:

import os
from samgeo import SamGeo3Video, download_file

import os from samgeo import SamGeo3Video, download_file

Initialize Video Predictor¶

The SamGeo3Video class provides a simplified API for video segmentation. It automatically uses all available GPUs.

In [ ]:

sam = SamGeo3Video()

sam = SamGeo3Video()

Load a Video¶

You can load from different sources:

• MP4 video file
• Directory of JPEG frames
• Directory of GeoTIFFs (for remote sensing time series)

In [ ]:

url = "https://github.com/opengeos/datasets/releases/download/videos/cars.mp4"
video_path = download_file(url)

url = "https://github.com/opengeos/datasets/releases/download/videos/cars.mp4" video_path = download_file(url)

In [ ]:

sam.set_video(video_path)

sam.set_video(video_path)

In [ ]:

sam.show_video(video_path)

sam.show_video(video_path)

Text-Prompted Segmentation¶

Use natural language to describe objects. SAM 3 finds all instances and tracks them.

In [ ]:

# Segment all car in the video
sam.generate_masks("car")

# Segment all car in the video sam.generate_masks("car")

Visualize Results¶

In [ ]:

# Show the first frame with masks
sam.show_frame(0, axis="on")

# Show the first frame with masks sam.show_frame(0, axis="on")

In [ ]:

# Show multiple frames in a grid
sam.show_frames(frame_stride=20, ncols=3)

# Show multiple frames in a grid sam.show_frames(frame_stride=20, ncols=3)

Remove Objects¶

Remove specific objects by ID and re-propagate.

In [ ]:

# Remove object 2 and re-propagate
sam.remove_object(2)
sam.propagate()
sam.show_frame(0)

# Remove object 2 and re-propagate sam.remove_object(2) sam.propagate() sam.show_frame(0)

Point Prompts¶

Add objects back or refine segmentation using point prompts.

In [ ]:

# Add back object 2 with a positive point click
sam.add_point_prompts(
    points=[[335, 203]],  # [x, y] coordinates
    labels=[1],  # 1=positive, 0=negative
    obj_id=2,
    frame_idx=0,
)
sam.propagate()
sam.show_frame(0)

# Add back object 2 with a positive point click sam.add_point_prompts( points=[[335, 203]], # [x, y] coordinates labels=[1], # 1=positive, 0=negative obj_id=2, frame_idx=0, ) sam.propagate() sam.show_frame(0)

Refine with Multiple Points¶

Use positive and negative points to refine the mask.

In [ ]:

# Refine to segment only the shirt (not pants)
sam.add_point_prompts(
    points=[[335, 195], [335, 220]],  # detect windshield, not the car
    labels=[1, 0],  # positive, negative
    obj_id=2,
    frame_idx=0,
)
sam.propagate()
sam.show_frames(frame_stride=20, ncols=3)

# Refine to segment only the shirt (not pants) sam.add_point_prompts( points=[[335, 195], [335, 220]], # detect windshield, not the car labels=[1, 0], # positive, negative obj_id=2, frame_idx=0, ) sam.propagate() sam.show_frames(frame_stride=20, ncols=3)

Save Results¶

Save masks as images or create an output video.

In [ ]:

os.makedirs("output", exist_ok=True)

# Save mask images
sam.save_masks("output/masks")

os.makedirs("output", exist_ok=True) # Save mask images sam.save_masks("output/masks")

In [ ]:

# Save video with blended masks
sam.save_video("output/segmented.mp4", fps=25)

# Save video with blended masks sam.save_video("output/segmented.mp4", fps=25)

Close Session¶

Close the session to free GPU resources.

In [ ]:

sam.close()

sam.close()

To completely shutdown and free all resources:

In [ ]:

sam.shutdown()

sam.shutdown()