Image segmentation

What it is

A machine learning model assigns a category label to every pixel in a source image, outputting a bitmask rather than a single classification tag. These models typically use an encoder-decoder architecture where the network first captures spatial features and then reconstructs a pixel-level map of the same resolution as the input. Inference costs are significantly higher than classification because the model must compute a high-resolution tensor output, often resulting in larger payloads and increased GPU memory requirements compared to simple label tagging.

Why it matters

You need this when the position, boundary, or geometry of an object matters for your downstream logic, such as blurring a user's background or calculating the exact area of a physical object. If you only need to know that an image contains a car, use standard classification to save on latency and cost. Ignoring the difference between tagging and segmentation leads to deploying models that are orders of magnitude slower than necessary for the required feature set.

In practice

You will typically interact with this via APIs or libraries that return a separate mask layer or a JSON array of coordinates representing the polygon of each detected object. Watch for the output resolution parameter in your inference request, as requesting a mask at the full input dimensions will drastically increase your latency and memory overhead per request.

The tradeoff

The main tradeoff is precision versus latency, as segmenting at high resolution provides accurate boundaries but drastically increases the time per request and the computational resources required.

Where it appears

Research summaries that use Image segmentation, each linked to its source paper.

• Segment Anything Model for Image Segmentation
  Segment Anything

  The paper introduces a foundation model and data engine for promptable image segmentation that enables zero shot transfer to downstream tasks.

• U-Net for Biomedical Image Segmentation
  U-Net: Convolutional Networks for Biomedical Image Segmentation

  The paper introduces U-Net, a convolutional network architecture designed for precise biomedical image segmentation using very few training images.