Instance Segmentation with Point Prompts and SAM 3¶

This notebook demonstrates how to use the Segment Anything Model 3 (SAM3) for interactive instance segmentation using point and box prompts.

Installation¶

In [ ]:

# %pip install "segment-geospatial[samgeo3]"

# %pip install "segment-geospatial[samgeo3]"

Import Libraries¶

In [ ]:

from samgeo import SamGeo3, download_file, show_image

from samgeo import SamGeo3, download_file, show_image

Download Sample Data¶

In [ ]:

url = "https://raw.githubusercontent.com/facebookresearch/sam3/refs/heads/main/assets/images/truck.jpg"
image_path = download_file(url)

url = "https://raw.githubusercontent.com/facebookresearch/sam3/refs/heads/main/assets/images/truck.jpg" image_path = download_file(url)

In [ ]:

show_image(image_path, axis="on")

show_image(image_path, axis="on")

Initialize SAM3¶

To use point and box prompts (SAM1-style interactive segmentation), initialize SAM3 with enable_inst_interactivity=True.

In [ ]:

sam = SamGeo3(backend="meta", enable_inst_interactivity=True)

sam = SamGeo3(backend="meta", enable_inst_interactivity=True)

In [ ]:

sam.set_image(image_path)

sam.set_image(image_path)

Generate Masks by Point Prompts¶

Select an object by clicking a point on it. Points are input in (x, y) format with labels:

• 1 = foreground point (include this region)
• 0 = background point (exclude this region)

In [ ]:

# Single foreground point - input as Python list
sam.generate_masks_by_points([[520, 375]])

# Single foreground point - input as Python list sam.generate_masks_by_points([[520, 375]])

In [ ]:

print(sam.masks)

print(sam.masks)

In [ ]:

print(sam.scores)

print(sam.scores)

Visualize Point Prompts¶

In [ ]:

sam.show_points([[520, 375]], [1])

sam.show_points([[520, 375]], [1])

Show the Results¶

In [ ]:

sam.show_anns()

sam.show_anns()

In [ ]:

sam.show_masks()

sam.show_masks()

Save Masks¶

In [ ]:

sam.save_masks("truck_mask.png", unique=True)

sam.save_masks("truck_mask.png", unique=True)

Multiple Points with Labels¶

Use multiple points to refine selection. Use label=0 for background points to exclude regions.

In [ ]:

# Two foreground points on the truck
sam.generate_masks_by_points([[500, 375], [1125, 625]], point_labels=[1, 1])

# Two foreground points on the truck sam.generate_masks_by_points([[500, 375], [1125, 625]], point_labels=[1, 1])

In [ ]:

sam.show_points([[500, 375], [1125, 625]], [1, 1])

sam.show_points([[500, 375], [1125, 625]], [1, 1])

In [ ]:

sam.show_anns()

sam.show_anns()

In [ ]:

sam.show_masks()

sam.show_masks()

Using Background Points¶

Add a background point (label=0) to exclude a region from the mask.

In [ ]:

# One foreground point on window, one background point on car body
sam.generate_masks_by_points(
    [[500, 375], [1125, 625]], point_labels=[1, 0]  # foreground, background
)

# One foreground point on window, one background point on car body sam.generate_masks_by_points( [[500, 375], [1125, 625]], point_labels=[1, 0] # foreground, background )

In [ ]:

sam.show_points([[500, 375], [1125, 625]], [1, 0])

sam.show_points([[500, 375], [1125, 625]], [1, 0])

In [ ]:

sam.show_anns()

sam.show_anns()

Box Prompts¶

Use a bounding box in XYXY format (xmin, ymin, xmax, ymax) to select an object.

In [ ]:

# Box around the front wheel
sam.generate_masks_by_boxes_inst([[425, 600, 700, 875]])

# Box around the front wheel sam.generate_masks_by_boxes_inst([[425, 600, 700, 875]])

In [ ]:

sam.show_boxes([[425, 600, 700, 875]])

sam.show_boxes([[425, 600, 700, 875]])

In [ ]:

sam.show_anns()

sam.show_anns()

Multiple Box Prompts¶

Process multiple boxes at once for efficient batch segmentation.

In [ ]:

boxes = [
    [75, 275, 1725, 850],  # Whole truck
    [425, 600, 700, 875],  # Front wheel
    [1375, 550, 1650, 800],  # Rear wheel
    [1240, 675, 1400, 750],  # License plate area
]

sam.generate_masks_by_boxes_inst(boxes)

boxes = [ [75, 275, 1725, 850], # Whole truck [425, 600, 700, 875], # Front wheel [1375, 550, 1650, 800], # Rear wheel [1240, 675, 1400, 750], # License plate area ] sam.generate_masks_by_boxes_inst(boxes)

In [ ]:

sam.show_boxes(boxes)

sam.show_boxes(boxes)

In [ ]:

sam.show_anns()

sam.show_anns()

In [ ]:

sam.save_masks("truck_boxes_mask.png", unique=True)

sam.save_masks("truck_boxes_mask.png", unique=True)

Low-Level API: predict_inst¶

For more control, you can use the lower-level predict_inst() method which returns masks, scores, and logits directly. Input points and boxes can be provided as Python lists.

In [ ]:

# Using Python lists for input
masks, scores, logits = sam.predict_inst(
    point_coords=[[520, 375]],
    point_labels=[1],
    multimask_output=True,
)

print(f"Generated {len(masks)} masks")
print(f"Scores: {scores}")

# Using Python lists for input masks, scores, logits = sam.predict_inst( point_coords=[[520, 375]], point_labels=[1], multimask_output=True, ) print(f"Generated {len(masks)} masks") print(f"Scores: {scores}")

In [ ]:

# Show all masks with point overlays
sam.show_inst_masks(masks, scores, point_coords=[[520, 375]], point_labels=[1])

# Show all masks with point overlays sam.show_inst_masks(masks, scores, point_coords=[[520, 375]], point_labels=[1])

In [ ]:

# Box prompt with Python list
masks, scores, logits = sam.predict_inst(
    box=[425, 600, 700, 875],
    multimask_output=False,
)

sam.show_inst_masks(masks, scores, box_coords=[425, 600, 700, 875])

# Box prompt with Python list masks, scores, logits = sam.predict_inst( box=[425, 600, 700, 875], multimask_output=False, ) sam.show_inst_masks(masks, scores, box_coords=[425, 600, 700, 875])

Summary¶

This notebook demonstrated SAM3's interactive instance segmentation capabilities:

High-level API (recommended):

• generate_masks_by_points() - Generate masks from point prompts
• generate_masks_by_boxes_inst() - Generate masks from box prompts
• show_points() / show_boxes() - Visualize prompts
• show_anns() / show_masks() - Visualize results
• save_masks() - Save masks to file

Low-level API:

• predict_inst() - Direct access to masks, scores, and logits
• show_inst_masks() - Display masks with overlays

Input formats:

• Points and boxes can be provided as Python lists or numpy arrays
• Point labels: 1 = foreground, 0 = background