Automated Segmentation of Remote Sensing Imagery with SAM 3¶

In this notebook, we demonstrate the automated segmentation of remote sensing imagery using SAM 3. The process begins with image captioning, which automatically identifies key features within the image. These features can then be utilized as text prompts for SAM 3, enabling precise segmentation.

Installation¶

First, make sure you have the required dependencies installed:

In [ ]:

# %pip install "segment-geospatial[samgeo3]"

# %pip install "segment-geospatial[samgeo3]"

Import Libraries¶

In [ ]:

import leafmap
from samgeo import SamGeo3, download_file
from samgeo.caption import ImageCaptioner

import leafmap from samgeo import SamGeo3, download_file from samgeo.caption import ImageCaptioner

Download Sample Data¶

Let's download a sample satellite image covering the University of California, Berkeley, for testing:

In [ ]:

url = "https://huggingface.co/datasets/giswqs/geospatial/resolve/main/uc_berkeley.tif"
image_path = download_file(url)

url = "https://huggingface.co/datasets/giswqs/geospatial/resolve/main/uc_berkeley.tif" image_path = download_file(url)

In [ ]:

m = leafmap.Map()
m.add_raster(image_path, layer_name="Satellite image")
m

m = leafmap.Map() m.add_raster(image_path, layer_name="Satellite image") m

Image captioning¶

In [ ]:

captioner = ImageCaptioner(
    blip_model_name="Salesforce/blip-image-captioning-base",
    spacy_model_name="en_core_web_sm",
)

captioner = ImageCaptioner( blip_model_name="Salesforce/blip-image-captioning-base", spacy_model_name="en_core_web_sm", )

In [ ]:

caption, features = captioner.analyze(image_path)
print(f"Caption: {caption}")
print(f"Features: {features}")

caption, features = captioner.analyze(image_path) print(f"Caption: {caption}") print(f"Features: {features}")

Request access to SAM3¶

To use SAM3, you need to request access by filling out this form on Hugging Face: https://huggingface.co/facebook/sam3

Once you have access, uncomment the following code block and run it.

In [ ]:

# from huggingface_hub import login
# login()

# from huggingface_hub import login # login()

Initialize SAM3¶

When initializing SAM3, you can choose the backend from "meta", or "transformers".

In [ ]:

sam3 = SamGeo3(backend="meta", device=None, checkpoint_path=None, load_from_HF=True)

sam3 = SamGeo3(backend="meta", device=None, checkpoint_path=None, load_from_HF=True)

Set the image¶

You can set the image by either passing the image path or the image URL.

In [ ]:

sam3.set_image(image_path)

sam3.set_image(image_path)

Generate masks with text prompt¶

In [ ]:

feature = features[0]
sam3.generate_masks(prompt=feature)

feature = features[0] sam3.generate_masks(prompt=feature)

Show the results¶

In [ ]:

sam3.show_anns()

sam3.show_anns()

In [ ]:

sam3.show_masks()

sam3.show_masks()

Save Masks¶

Save the generated masks to a file. If the input is a GeoTIFF, the output will be a GeoTIFF with the same georeferencing. Otherwise, it will be saved as PNG.

In [ ]:

# Save masks with unique values for each object
# Since uc_berkeley.tif is a GeoTIFF, the output will also be a GeoTIFF
sam3.save_masks(output=f"{feature}_masks.tif", unique=True)

# Save masks with unique values for each object # Since uc_berkeley.tif is a GeoTIFF, the output will also be a GeoTIFF sam3.save_masks(output=f"{feature}_masks.tif", unique=True)

In [ ]:

# Save as binary mask (all foreground pixels are 255)
sam3.save_masks(output=f"{feature}_masks_binary.tif", unique=False)

# Save as binary mask (all foreground pixels are 255) sam3.save_masks(output=f"{feature}_masks_binary.tif", unique=False)

Save Masks with Confidence Scores¶

You can also save the confidence scores for each mask. The scores indicate the model's confidence for each predicted mask.

In [ ]:

# Save masks and confidence scores
# Each pixel in the scores image will have the confidence value of its mask
sam3.save_masks(
    output=f"{feature}_masks_with_scores.tif",
    save_scores=f"{feature}_scores.tif",
    unique=True,
)

# Save masks and confidence scores # Each pixel in the scores image will have the confidence value of its mask sam3.save_masks( output=f"{feature}_masks_with_scores.tif", save_scores=f"{feature}_scores.tif", unique=True, )

In [ ]:

sam3.show_masks(cmap="coolwarm")

sam3.show_masks(cmap="coolwarm")

Visualize Confidence Scores¶

Let's visualize the confidence scores to see which masks have higher confidence:

In [ ]:

m.add_raster(f"{feature}_masks.tif", layer_name=f"{feature} masks", visible=False)
m.add_raster(
    f"{feature}_scores.tif",
    layer_name=f"{feature} scores",
    cmap="coolwarm",
    opacity=0.8,
    nodata=0,
)
m

m.add_raster(f"{feature}_masks.tif", layer_name=f"{feature} masks", visible=False) m.add_raster( f"{feature}_scores.tif", layer_name=f"{feature} scores", cmap="coolwarm", opacity=0.8, nodata=0, ) m